Best practices for batch processing without SSIS

Question

Hi,
The gist of this question is in general how should a C# client insert/update batches records using stored procedures. The ideas I can think of are:
1) create 1 SP with a parameter of type XML, and pass say 100 records at a time, on 1 thread.  The SP reads the XML as a table and does a single INSERT.
2) create 1 SP with many parameters, that inserts 1 records.  I can either build a big block of EXEC statements for say 100 records at a time, or call the SP 1 and a time, on 1 thread.  Obviously this seems the slowest.
3) Parallel processing version of either of the above: Pass 100 records at a time via XML parameter, big block of EXEC statements, or 1 at a time, and use PLINQ to make multiple connections to the database.
The records will be fairly wide, substantial records.
Which scenario is likely to be fastest and avoid lock contention?
(We are doing batch processing and there is not a current SSIS infrastructure, so it's manual: fetch data, call web services, update batches.  I need a batch strategy that doesn't involve SSIS - yet).
Thanks.

Answer 1

The "streaming" option you mention in your linked thread sounds interesting, is that a way to input millions of rows at once?  Are they not staged into the tempdb?
The entire TVP is stored in tempdb before the query/proc is executed.  The advantage of the streaming method is that it eliminates the need to load the entire TVP into memory on either the client or server.  The rowset is streamed to the server
and SQL Server uses the insert bulk method is to store it in tempdb.  Below is an example C# console app that streams 10M rows as a TVP.
using System;
using System.Data;
using System.Data.SqlClient;
using System.Collections;
using System.Collections.Generic;
using Microsoft.SqlServer.Server;
namespace ConsoleApplication1
class Program
static string connectionString = @"Data Source=.;Initial Catalog=MyDatabase;Integrated Security=SSPI;";
static void Main(string[] args)
using(var connection = new SqlConnection(connectionString))
using(var command = new SqlCommand("dbo.usp_tvp_test", connection))
command.Parameters.Add("@tvp", SqlDbType.Structured).Value = new Class1();
command.CommandType = CommandType.StoredProcedure;
connection.Open();
command.ExecuteNonQuery();
connection.Close();
class Class1 : IEnumerable<SqlDataRecord>
private SqlMetaData[] metaData = new SqlMetaData[1] { new SqlMetaData("col1", System.Data.SqlDbType.Int) };
public IEnumerator<SqlDataRecord> GetEnumerator()
for (int i = 0; i < 10000000; ++i)
var record = new SqlDataRecord(metaData);
record.SetInt32(0, i);
yield return record;
IEnumerator IEnumerable.GetEnumerator()
throw new NotImplementedException();
Dan Guzman, SQL Server MVP, http://www.dbdelta.com