SSIS: Performance Issue while calling Web Service

Question

Hi
I am trying to load data from an API calling web service in SSIS. I wrote a small piece of C# code, which connects the webservice and pulls data for a given time interval.
My code will first connect to the webservice and then pulls data for a given interval of time. The returning data will be first stored in an ssis variable in XML format and then from that variable, data will be loaded to sql server table.
WebService --> XML Variable --> SQL server table
Here all is fine till we have small amount of data say 1000 - 2000 rows returning for given time period. If data volume is more say 10000, it
fails !!! error: "timeout"
Could anyone please suggest me some best/optimized solution for this.
C# Code used:
public void Main()
string URL = Dts.Variables["User::URL"].Value.ToString();
string LD_JOB_NR = Dts.Variables["User::V_LD_JOB_NR"].Value.ToString();
Dts.TaskResult = (int)ScriptResults.Success;
string DATA1 = HttpGet(URL).ToString();
Dts.Variables["User::xmlDoc"].Value = DATA1;
writefiles(DATA1, "OrderInfo", "xml", "E:\\myDirectory\\storage\\XMLfiles\\", LD_JOB_NR);
Dts.TaskResult = (int)ScriptResults.Success;
static string HttpGet(string url)
HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
req.Timeout = 600000;
string result = null;
using (HttpWebResponse resp = req.GetResponse() as HttpWebResponse)
StreamReader reader =
new StreamReader(resp.GetResponseStream());
result = reader.ReadToEnd();
result = result.Replace("&lt;", "<");
result = result.Replace("&gt;", ">");
result = result.Replace("&#xD;", "");
result = result.Replace("&amp;", "&");
result = result.Replace("&apos;", "'");
result = result.Replace("&quot;", "\"");
return result;
private void writefiles(string delistring, string tgtfilename, string fileextention, string filepath, string LD_JOB_NR)
DateTime dt = DateTime.Now;
string s = dt.ToString("yyyyMMddHHmmss");
StreamWriter writer = null;
string fileName = filepath + tgtfilename + "_" + s + "_" + LD_JOB_NR + "." + fileextention;
try
writer = new System.IO.StreamWriter(fileName, true);
writer.WriteLine(delistring);
catch (Exception ex)
throw ex;
finally
writer.Close();

Answer 1

Hi Arthur,
Thanks for your reply !
I tried to figure it out on API end but no success, actually it is out of our network(organization).
If any other approach/way... please guide me
meanwhile I 'll try to get information regarding timeout
at API end.
Please Mark as Answer if my post solved your problem or Vote As Helpful if this helps. Blogs: www.sqlserver2005forum.blogspot.com