Import server Process consuming maximum time at Parsing and Importing steps

Question

We are using Import server to Import files and most of the time looks like is spent on Impoting and Parsing, the whole load took little more than an hour and these two steps (see the log below) took almost 50 mins.
1543 2008/09/29 17:23:07.041   [MDS=usil0d32 Repos=ZCustomer11_Q ClientSystem=R3_ERP Port=IMPORT_MAIN]: xImporter: Thread started.  
772 2008/09/29 18:12:20.231   [MDS=usil0d32 Repos=ZCustomer11_Q ClientSystem=R3_ERP Port=IMPORT_MAIN]: xSaxParser: Thread finished; Start -> End: 2612.191168000 seconds.
So I wonder how can we reduce the time taken for these two steps?
Same thing on Import Manager takes 10 mins max to open up and I assume Import manager opening up an XML is equivalent to Importer & Parser thread of Import server ?

Answer 1

It is Queued parallel Processing, you will not see a true parallel processing here, just like a pipelined CPU architecture.
Say Chunk 1 is at stage 1 preparing import records
Then Nothing else happens parallelly
Now is Chunk 1 moved to stage 2 Filter/Merge
Then Chunk 2 can move to stage 1 preparing import records
Now Chunk 1 moves to stage 3 MDS
Then Chunk 2 moves to stage 2 Filter/Merge
And Chunk 3 moves to stage 1 preparing import records
so on ......and so forth......
So if you prepare a Gantt chart with all these tasks in swim lanes then over a period of time you will observe that actually 'n' number of tasks will be running on various chunks and this 'n' is the what you set in No.of Chunks parameter.
try observing, when the stage 1 task started on Chunk 2 , see if it kinda overlaps stage 2 of chunk 1. Try posting all the logs , very interesting observation !
Size of Chunk is like preparing the packet, in any communication protocol, the more records you have in chunk , the less overhead/record but more wait time. The less the no.of records you have in your chunk the more the overhead/record but less the wait time. So yeah this can be tweaked according to your memory.
-Sudhir.