Processing Multiple Files for more than 100 Receive Location - File Size - 25 MB each file, file type DML

Question

Hi Everybody
Please suggest.
For one of our BizTalk interface, we have around 120 receive locations.  We are just moving (*.dml) files from the source to destination without doing any processing.  
We receive lots of files in different receive locations and in few cases the file size will be around 25 MB and so while moving these large files, the CPU usage is varying between 10% to 90+% and the time consuming for this single huge file is around
10 to 20 minutes.  This solution was already in place and was designed by the previous vendor for moving the files to their clients.  Each client has 2 receive locations and they have around 60 clients.  Is there any best solution for implementing
this with in BizTalk or outside BizTalk? Please suggest.
I am also looking for how to control the number of files which gets picked from the BizTalk receive location.  For example, If we have say 1000 files in receive location and we want to pick at a time only 50 files only (batch of 50) then is it possible?
because currently it is picking all the files available in source location, and one of the process is dropping thousands of files in to the source location, so we want to control  the number of files getting picked (or even if we can control to pick the
number of KBs).  Please guide us on how we can control the number of files.

Answer 1

Hi Rajeev,
25 MB per file, 1000 files. Certainly you got to revisit the reason for choosing BizTalk.
“the time consuming for this single huge file is around 10 to 20 minutes”
 - This is a problem.
You could consider other file transfer options like XCopy or RobotCopy etc if you want to transfer to another local/shared drive. Or you can consider using SSIS
which does comes with many adapters to send to destination system depending on their destination transfer protocol.
But in your case, you have some of the advantages that you get with BizTalk. For your scenario, you have more source systems (more Receive locations), with BizTalk
it’s always easier to manage these configurations, you can easily enable and disable them when a need arise. You can easily configure tracking; configure host instances based on load etc. So you can consider following design for your requirement. This design
would suit you well since you’re not processing the message and just pass it through from source to destination:
Use a custom pipeline component in the Receive Locations which receives the large file.
Stores the received file into disk and creates a small XML metadata message that contains the information about where the large file is stored.
The small XML message is then published into the
message box db
instead of the large file. Let the metadata file also contain the same context properties as the received file.
In the send port, use another custom pipeline component that process the metadata xml file, retrieve the location of the disk where the file is stored, access the file and send it to destination.
Read the following article on this design..
http://www.codeproject.com/Articles/180333/Transfer-Large-Files-using-BizTalk-Send-Side
This way you don’t need to publish the whole message into message box DB which would considerably reduce the processing time and utilises host instance to process
more files. This way you can still get the advantages of BizTalk and still process large files.
And regarding your question of restricting the Receive location to handles the number of files from receives location. No it’s not possible.
Regards,
M.R.Ashwin Prabhu
If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.