Sudden slow down in data transfer

Question

I'm reading data from the hard disk on a Target PXI System (8106 RT) and sending it via Ethernet to a Host PC and writing it to its hard disk. I read a chuck of data from the Target, send it using TCP Write function. The Host uses the TCP Read function get the data and write it to its hard disk. Data is being transfer over a Gbit Ethernet. A total of 9 files ranging in size from 700 MB - 2 GB are being transferred. The process moves along at an adequate speed until it suddenly slows down to slower than a snail's pace in the middle of a file.
By using probes (see enclosures) I've been able to determine it take approximately 17 seconds between when the data is read on the PXI to when it is written on the Host. However, an iteration of the loop on the Target has increased to over 3 minutes. The CPU usage on the Target and Host are minimal, less than 1%. Memory usage on the Host is also small. Any suggestions on how to diagnose what has caused this sudden slow down?
Attachments:
Slide1.PNG ‏104 KB
Slide2.PNG ‏124 KB
Slide3.PNG ‏39 KB

Answer 1

I would recommend using smaller chunk sizes for your data transfers. I would recommend something inthe 10K to 50K range. I have done quite a bit of profiling for sending large amounts of data and I have found that it is better to send lots of smaller chunks rather than fewer large chunks. Also make sure that you are not automatically building up large buffers in your application if you are using shift registers. If you are reading from a file read the file in chunks and send the chuncks. Don't read the whole file in in one read. One experiment I ran resulted in a single read of a large file (2 MB or so) took something like 10 minutes to read it in one read. However it only took a couple of 100 ms to read the whole thing in 10K chunks. Same applies to wrinting very large chunks using the TCP write. You can easily get a timeout if you write 1 MB as a single write because the receiver has not consumed all of the data within the timeout period. However, multiple small writes will reset the timeout for each chunk written.
Mark Yedinak
"Does anyone know where the love of God goes when the waves turn the minutes to hours?"
Wreck of the Edmund Fitzgerald - Gordon Lightfoot