Outbound FTP Dropping

Question

Hi Folks,
We're experiencing an issue in Azure that has only appeared in the past month. We have a number of reasonably sized FTP uploads that previously completed without error and now fail. I've enabled Remote Desktop on one of our VMs and installed Filezilla to
do a little bit of testing and have found...
Transfer < 60s does not fail
Transfer > 60s fails consistently
From what I can tell Azure seems to be dropping the control connection. I understood that this was the case with inbound connections to the load balancer, but this behaviour now seems to be replicated for outbound connections too. This was not the case a
month ago! Any insight as to what or why this is no longer working would be greatly appreciated.
Under the hood we're using FtpWebRequest and have little influence over the control connection it uses (so we can't, for example, send keep alive or NOOP requests over it). This'll also play havoc with any connection pooling that might be going on under
the hood. It should be noted that outside of Azure we can transfer the files just fine.
Thanks in advance for any advice.
Dean 

Answer 1

Hi Ming,
We've had to re-deploy the service numerous times in order to develop a bugfix that works in the Azure environment and we can repeatedly reproduce this issue. In addition this was working reliably for 6 months without any code changes at all - it's only
broken in the past month. I can't be more exact because we only noticed the issue 2 weeks ago and the previous upload was 2 weeks before that.
Our server-side code effectively does this:
Uri uri = new Uri("ftp://server/uploads/test.pdf");
FtpWebRequest webRequest = (FtpWebRequest)WebRequest.Create(uri);
webRequest.ConnectionGroupName = uri.DnsSafeHost;
webRequest.KeepAlive = true;
webRequest.Method = WebRequestMethods.Ftp.UploadFile;
webRequest.Credentials = new NetworkCredential("username", "password");
using (Stream destinationStream = webRequest.GetRequestStream())
sourceStream.CopyTo(destinationStream, 0x100000);
using (webRequest.GetResponse())
The code throws a WebException after the end brace for the request stream using block but only after hanging waiting for the control connection to time out (usually 60 seconds). The WebException simply states that the connection was closed. When this occurs
the file has been successfully uploaded. Right now our workaround is to re-establish a connection and check the file size - if it is what we expect we assume the upload succeeded. Flaky code at best!
I can reproduce this using Filezilla repeatedly by performing the following steps:
1) Generate a file that takes < 60 secs to upload to the destination server. We use...
fsutil file createnew D:\Test.pdf <size in bytes>
 Obviously the size will vary depending on the bandwidth you have out to the destination server.
2) Upload the file. Filezilla will not timeout.
Repeat this with a file that takes just over 60 secs and Filezilla hangs and then asks to overwrite the file again  - it never got confirmation that the upload completed because the control connection dropped. In Filezilla this can be mitigated by
sending keep alive messages (NOOP) over the control connection. Sadly this isn't easily doable using FtpWebRequest...
Thanks & Regards,
Dean