Download XMl file from SFTP: performance issue

Question

Hi All,
I am downloading an XML file of size almost 15-20 MB from an SFTP using winSCP.
A C# code is written in SSIS script component which loops the remote directory to find the recent file then it downloads it. This process is taking almost 20 minutes to run in SSIS.
Could anyone please suggest me the optimized solution.
few keynotes: 1> on SFTP, there will be always more number of files, say 50-60.
                     2> file size may grow with time
                     3> comparing filename to find latest file because filename is suffixed with timestamp e.g. filename_YYYYMMDD.xml
below is my C# code used in script component
public void Main()
string hostName = (string)Dts.Variables["HostName"].Value;
string userName = (string)Dts.Variables["UserName"].Value;
string password = (string)Dts.Variables["Password"].Value;
string sshHostKeyFingerprint = (string)Dts.Variables["SshHostKeyFingerprint"].Value;
string winscpexecutablePath = (string)Dts.Variables["winscpExecutablePath"].Value;
string localOutPath = (string)Dts.Variables["User::localOutPath"].Value;
string remoteDirectory = (string)Dts.Variables["User::ftpRemoteDirectory"].Value;
string latestFileName = null;
List<DateTime> fileDate = new List<DateTime>();
Dictionary<DateTime, string> dicfiledate = new Dictionary<DateTime, string>();
// Setup session options
SessionOptions sessionOptions = new SessionOptions
Protocol = Protocol.Sftp,
HostName = hostName,
UserName = userName,
Password = password,
SshHostKeyFingerprint = sshHostKeyFingerprint
try
using (Session session = new Session())
session.ExecutablePath = winscpexecutablePath;
session.Open(sessionOptions);
RemoteDirectoryInfo directory = session.ListDirectory(remoteDirectory);
RemoteFileInfo[] fileInfo = new RemoteFileInfo[directory.Files.Count];
if (fileInfo.Length <= 0)
Dts.Variables["User::isFileExist"].Value = false;
else
Dts.Variables["User::isFileExist"].Value = true;
List<string> lstFileNames = new List<string>();
for (int i = 0; i < directory.Files.Count; i++)
lstFileNames.Add(directory.Files[i].Name);
Dictionary<DateTime, int> dictFinal = new Dictionary<DateTime, int>();
for (int i = 0; i < lstFileNames.Count; i++)
if (lstFileNames[i].StartsWith("metrics_"))
int year = Convert.ToInt32(lstFileNames[i].Substring(8, 4));
int month = Convert.ToInt32(lstFileNames[i].Substring(12, 2));
int date = Convert.ToInt32(lstFileNames[i].Substring(14, 2));
dictFinal.Add(new DateTime(year, month, date), i);
var sortedDateTime = dictFinal.Keys.OrderByDescending(x => x);
int latestFileIndex = dictFinal[Convert.ToDateTime(sortedDateTime.First())];
latestFileName = lstFileNames[latestFileIndex];
// files transfer mode
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
TransferOperationResult transferResult;
transferResult = session.GetFiles(remoteDirectory + latestFileName, @"" + localOutPath + @"\", false, transferOptions);
transferResult.Check();
// Print results
bool fireAgain = false;
foreach (TransferEventArgs transfer in transferResult.Transfers)
Dts.Events.FireInformation(0, null,
string.Format("Download of {0} succeeded", transfer.FileName),
null, 0, ref fireAgain);
Dts.TaskResult = (int)DTSExecResult.Success;
catch (Exception e)
Dts.Events.FireError(0, null,
string.Format("Error when using WinSCP to Download files: {0}", e),
null, 0);
Dts.TaskResult = (int)DTSExecResult.Failure;

Answer 1

Hi Rahul,
Is it possible for you to get the latest file by comparing the CreationTime of these files other than comparing the file names? If so, you can try the code in Reza’s blog:
http://www.rad.pasfu.com/index.php?/archives/30-Find-Last-Created-File-in-Special-Directory-SSIS.html 
Alternatively, maybe you can try a free third party SFTP Task available on the CodePlex:
http://ssissftp.codeplex.com/ 
Regards,
Mike Yin
TechNet Community Support