CLR Procedure and returning large number of rows

Question

I have a CLR stored procedure coded in C# that retrieves data from a web service, and returns that data using SendResultsStart/SendResultsRow/SendResultsEnd. This all works fine, except when the data from the web service is tens or even thousands of records.
The code itself takes about 3 minutes on average to do all it's work with around 50000-60000 records, but the procedure does not return in SSMS for about another 10-15 minutes, during which time the CPU and memory usage go up significantly.
To rule out any of the CLR code as the culprit, I created a very simple CLR procedure that just loops to return 100000 records with int and nvarchar(256) fields with the current count, and "ABC" followed by the count.  Here is the code:
[Microsoft.SqlServer.Server.SqlProcedure]
public static void ABC()
System.Diagnostics.Stopwatch ExecuteTimer = System.Diagnostics.Stopwatch.StartNew();
SqlMetaData[] ResultMetaData = new SqlMetaData[2];
ResultMetaData[0] = new SqlMetaData("count", SqlDbType.Int);
ResultMetaData[1] = new SqlMetaData("text", SqlDbType.NVarChar, 256);
SqlContext.Pipe.SendResultsStart(new SqlDataRecord(ResultMetaData));
for (int x = 0; x < 100000; x++)
SqlDataRecord ResultItem = new SqlDataRecord(ResultMetaData);
ResultItem.SetValue(0, x);
ResultItem.SetValue(1, "ABC" + x.ToString());
SqlContext.Pipe.SendResultsRow(ResultItem);
SqlContext.Pipe.SendResultsEnd();
TimeSpan ExecTime = ExecuteTimer.Elapsed;
SqlContext.Pipe.Send("Elapsed Time: " + ExecTime.Minutes.ToString() + ":" + ExecTime.Seconds.ToString() + "." + ExecTime.Milliseconds.ToString());
I then executed procedure ABC in SSMS, and it took 21 minutes to return.  All of the data rows were visible after just a couple of seconds, but the query continued to run as the CPU and memory went up again.
Is this really how long it should take to return 100000 rows, or am I missing something?  Is there a better approach than using SendResultsStart/SendResultsRow/SendResultsEnd?
I've googled this to death and haven't found anything that helped or even explained why this is.
I would greatly appreciate any suggestions or alternate methods to achieve this faster.
Thanks!
Alex

Answer 1

When you create a new object, space on the garbage-collected heap is allocated for that object, and the address will be stored in a reference. Some time later, there will no longer be any references that hold the address of the allocated object. It doesn't
matter whether the reference count went to 0 because the reference was set to null or because the reference was on the stack and is no longer in lexical scope, the end result is the same: the garbage collector will, at some point in time, have to perform the
book-keeping operations necessary to identify that the space allocated for that now-unreferenced object can be re-used. When, on the other hand, you only create a single SqlDataRecord object and hold onto the reference, all of the book-keeping operations
associated with creating 100,000 objects are eliminated. This is why the documentation for the SqlDataReader class advises that:
When writing common language runtime (CLR) applications, you should re-use existing
SqlDataRecord objects instead of creating new ones every time. Creating many new
SqlDataRecord objects could severely deplete memory and adversely affect performance.