Service design for handling large datasets

Question

As an overnight process we need to invoke 2 services against every record in our database (over 1 million records). Specifically, the process flow should be as follows:
- For each record in the database invoke service A.
- For each record use the return value from service A as a parameter to invoke service B.
If we were to process each record one at a time in a synchronous fashion, the time needed for processing all records would be too great. I was wondering if there is a better way to implement this? I have considered batching and making asynchronous calls
using a duplex but am unclear about which option would be superior.

Answer 1

Datasets with datatables, the salad bowl,  are two slow for Service Oriented Achitecture.
http://www.hanselman.com/blog/ReturningDataSetsFromWebServicesIsTheSpawnOfSatanAndRepresentsAllThatIsTrulyEvilInTheWorld.aspx
Datatables use boxing and unboxing, which makes it slow..
http://www.csharphelp.com/2010/02/c-best-practices-to-write-high-performance-code/
You should be using DTO(s) and a List of DTO(s)
http://lauteikkehn.blogspot.com/2012/03/datatable-vs-list.html
http://en.wikipedia.org/wiki/Data_transfer_object
http://www.mindscapehq.com/documentation/lightspeed/Building-Distributed-Applications-/Building-WCF-Services-using-Data-Transfer-Objects
On the other hand and if using SQL Server,  you may want to look into MS SQL Server Service Broker too.
https://technet.microsoft.com/en-us/library/ms166104(v=sql.105).aspx