ftp timeout is failing my job

Hello, can someone please suggest to me how I can keep my ftp connection from
timing out after a large file download?

I am using SSIS and I have tried multiple ways of ftp'ing the files in my
directory to my local hard drive, but the tasks keep failing after a large
file is downloaded. The error I'm getting is:

An error occurred in the requested FTP operation. Detailed error
description: Receiving file "usfile.zip".
The operation timed out

The file is about 120MB.

I tried using the ftp task, as well as writing (reverse engineering) an ftp
script task using FtpWebRequest.Create and both methods work as long as the
files are small.

It looks like the ftp timeout property has to be set between 1 and 300, so
even though I am explicitly assigning a timeout property of 0 in my
connection via script, it is still timing out.

Any suggestions are much appreciated. I've been struggling with this issue
for about a week now.

Thanks!

Jason
http://www.lonestarfinancing.com

On 27 Ago, 15:03, Jason wrote:
> Hello, can someone please suggest to me how I can keep my ftp connection from
> timing out after a large file download? *
>
> I am using SSIS and I have tried multiple ways of ftp'ing the files in my
> directory to my local hard drive, but the tasks keep failing after a large
> file is downloaded. *The error I'm getting is:
>
> An error occurred in the requested FTP operation. Detailed error
> description: Receiving file "usfile.zip".
> The operation timed out
>
> The file is about 120MB.
>
> I tried using the ftp task, as well as writing (reverse engineering) an ftp
> script task using FtpWebRequest.Create and both methods work as long as the
> files are small. *
>
> It looks like the ftp timeout property has to be set between 1 and 300, so
> even though I am explicitly assigning a timeout property of 0 in my
> connection via script, it is still timing out. *
>
> Any suggestions are much appreciated. *I've been struggling with this issue
> for about a week now.
>
> Thanks!
>
> Jasonhttp://www.lonestarfinancing.com

Jason, as you may know an FTP client establishes two different
connections (using 2 different ports, ie 20 and 21):
- 1 control connection (authentication, directory listing...)
- 1 data connection (effective file transfer)
So said i believe that the ftp task times out (and fails) because the
control connection times out. This should be normal: try to ftp a
large file with a ftp client like filezilla, after the timeout has
expired, the control conn times out and thus you're no more able to
see the listing, but the transfer keeps on going.

Assuming that the link between the two machines is stable (and it
should be if you say that you got it worked with other methods), I
would guess that in SSIS's FTP task, the task fails when the control
connection times out even if the transfer is going on.

HTH,
Matteo

Hello Matteo,

After all my research on the issue, I believe you got it correct. My
control connection is timing out. The question is, how do I fix it?

I set keepalive = true, and I set the timeout property to -1 after reading
this:

http://msdn.microsoft.com/en-us/libr...t.timeout.aspx

It says that you can set the timeout to -1, and this will make the timeout
infinite. However, this is not the case.

I also tried downloading the one file that is giving me issues, and I still
get the timeout error. The file is 120MB. Therefore, even if I create a
new connection for each file rather than stuffing all files in
ftp.recievefiles(), I will still get the error.

The solutions I've read are:
send a Hash subcommand (not sure how to do)
modify the Quote subcommand (not sure how to do)
turn off firewall (I did this)
set keepalive = true
set timeout = -1 (i tried 0 too and it didn't work). I also tried timeout =
3600, and this did not work.


I am wondering if I should be using the ftpwebrequest for recieving files,
and that this might explain why the timeout = -1 is not working?

I am new to programming, so I'm not exactly sure what I'm doing .

Any assistance is greatly appreciated.

Here is my code:


Public Sub Main()

Dim ftp As FtpClientConnection

Dim retrieveFiles As Boolean = False

Dim ftpWeb As FtpWebRequest

Try

ftpWeb = CType(FtpWebRequest.Create("ftp://" +
Dts.Variables("FTPURL").Value.ToString + _

"/" + Dts.Variables("FTPDirectory").Value.ToString), FtpWebRequest)

ftpWeb.Credentials = New
NetworkCredential(Dts.Variables("FTPUser").Value.ToString,
Dts.Variables("FTPPassword").Value.ToString())

ftpWeb.Method = WebRequestMethods.Ftp.ListDirectoryDetails

ftpWeb.KeepAlive = True

'Using fileReader As StreamReader = File.OpenText(baseFileName)

Dim srResponse As New StreamReader(ftpWeb.GetResponse().GetResponseStrea m())

Dim readLine As String

Dim stringComponents() As String

Dim sbDownloadFiles As New StringBuilder()

Dim fileEnding As String = Dts.Variables("FtpFileEnding").Value.ToString

Dim ignoreDate As Boolean =
Convert.ToBoolean(Dts.Variables("IgnoreDate").Value.ToString)

While Not srResponse.EndOfStream()

readLine = srResponse.ReadLine()

If readLine.Trim().EndsWith(fileEnding) Then

stringComponents = readLine.Split(" ".ToCharArray())

'Since the file is updated at least once a month, we just check the day:

If ignoreDate Or Math.Abs(DateTime.Now.Day -
Int32.Parse(stringComponents(17))) < 2 Then

'This file has been modified within the last day or so...

retrieveFiles = True

sbDownloadFiles.Append(stringComponents(19) + "|")

End If

End If

End While

srResponse.Close()

srResponse = Nothing

ftpWeb = Nothing

If retrieveFiles Then

'Now we need to build our FTP connection:

Dim fiRemoteFile As FileInfo

Dim ftpConnectionManager As ConnectionManager

ftpConnectionManager = Dts.Connections.Add("FTP")

ftpConnectionManager.Properties("ServerName").SetValue(ftpConnectionManager,
Dts.Variables("FTPURL").Value)

ftpConnectionManager.Properties("ServerPort").SetValue(ftpConnectionManager,
Dts.Variables("FTPPort").Value)

ftpConnectionManager.Properties("ServerUserName").SetValue(ftpConnectionManager, Dts.Variables("FTPUser").Value)

ftpConnectionManager.Properties("ServerPassword").SetValue(ftpConnectionManager, Dts.Variables("FTPPassword").Value)

ftpConnectionManager.Properties("Timeout").SetValue(ftpConnectionManager,
Dts.Variables("FTPTimeout").Value)

ftpConnectionManager.Properties("ChunkSize").SetValue(ftpConnectionManager,
Dts.Variables("FtpChunkSize").Value)

ftpConnectionManager.Properties("Retries").SetValue(ftpConnectionManager,
Dts.Variables("FTPRetries").Value)

ftp = New FtpClientConnection(ftpConnectionManager.AcquireCo nnection(Nothing))

'Next we connect to the ftp server

ftp.Connect()

ftp.SetWorkingDirectory(Dts.Variables("FTPDirectory").Value.ToString)

'And we obtain our file(s):

ftp.ReceiveFiles(sbDownloadFiles.ToString().Substr ing(0, _

sbDownloadFiles.ToString.Length - 1).Split("|".ToCharArray()), _

Dts.Variables("FtpDestination").Value.ToString, True, False)

End If

'If we did not download anything new there is nothing to process...

Dts.Variables("ContinueProcessing").Value = retrieveFiles

Dts.TaskResult = Dts.Results.Success

Catch exError As Exception

Throw exError

Finally

If Not ftpWeb Is Nothing Then

ftpWeb = Nothing

End If

If Not ftp Is Nothing Then

ftp.Close()

End If

End Try

End Sub









"matteus" wrote:

> On 27 Ago, 15:03, Jason wrote:
> > Hello, can someone please suggest to me how I can keep my ftp connection from
> > timing out after a large file download?
> >
> > I am using SSIS and I have tried multiple ways of ftp'ing the files in my
> > directory to my local hard drive, but the tasks keep failing after a large
> > file is downloaded. The error I'm getting is:
> >
> > An error occurred in the requested FTP operation. Detailed error
> > description: Receiving file "usfile.zip".
> > The operation timed out
> >
> > The file is about 120MB.
> >
> > I tried using the ftp task, as well as writing (reverse engineering) an ftp
> > script task using FtpWebRequest.Create and both methods work as long as the
> > files are small.
> >
> > It looks like the ftp timeout property has to be set between 1 and 300, so
> > even though I am explicitly assigning a timeout property of 0 in my
> > connection via script, it is still timing out.
> >
> > Any suggestions are much appreciated. I've been struggling with this issue
> > for about a week now.
> >
> > Thanks!
> >
> > Jasonhttp://www.lonestarfinancing.com
>
> Jason, as you may know an FTP client establishes two different
> connections (using 2 different ports, ie 20 and 21):
> - 1 control connection (authentication, directory listing...)
> - 1 data connection (effective file transfer)
> So said i believe that the ftp task times out (and fails) because the
> control connection times out. This should be normal: try to ftp a
> large file with a ftp client like filezilla, after the timeout has
> expired, the control conn times out and thus you're no more able to
> see the listing, but the transfer keeps on going.
>
> Assuming that the link between the two machines is stable (and it
> should be if you say that you got it worked with other methods), I
> would guess that in SSIS's FTP task, the task fails when the control
> connection times out even if the transfer is going on.
>
> HTH,
> Matteo
>
>