Splitting and type casting huge string into arrays

Question

Hello,
I'm developing an application which is supposed to read measurement files. Files contain I16 and SGL data which is type casted into string. I16 data is data from analog input and SGL is from CAN-bus data in channel form. CAN and analog data is recorded using same scan rate.
For example, if we have 6 analog channels and 2 CAN channels string will be (A represents analog and C represents CAN):
A1 A2 A3 A4 A5 A6 C1 C2 A1 A2 A3 A4 A5 A6 C1 C2 A1 A2 .... and so on
Anyway, I have problems reading this data fast enough into arrays. Most obvious solution to me was to use shift registers and split string in for loop. I created a for loop with two inner for loops. Number of scans to read from string is wired to N terminal of the outermost loop. Shift register is initialized with string read from file.
First of the inner loops reads analog input data. Number of analog channels is wired to its N terminal. It's using split string to read 2 bytes at a time and then type casts data to I16. Rest of the string is wired to shift register. When every I16 channel from scan is read, rest of the string is passed to shift register of the second for loop.
Second loop is for reading CAN channels. It's similar to first loop except data is read 4 bytes at a time and type casted to SGL. When every CAN channel from scan is read, rest of the string is passed to shift register of the outermost loop. Outputs of type cast functions are tunneled out of loops to produce 2D arrays.
This way reading about 500 KB of data can take for example tens of seconds depending on PC and number of channels. That's way too long as we want to read several megabytes at a time.
I created also an example with one inner loop and all data is type casted to I16. That is extremely fast when compared to two inner loops!
Then I also made a test with two inner loops where I replaced shift register and split string with string subset. That's also faster than two inner loops + shift register, but still not fast enough. Any improvement ideas are highly appreciated. Shift register example attached (LV 7.1)
Thanks in advance,
Jakke Palonen
Attachments:
String to I16 and SGL arrays.vi ‏39 KB

Answer 1

OK, there is clearly room for improvement. I did some timing and my two above suggestions are already about 100x faster than yours. A few teeaks led to a version that is now over 500x faster than the original code.
A few timings on my rather slow computer (1GHz PIII Laptop) are shown on the front panel. For example with 10000 scans (~160kB data as 6+2) my new fastest version (Reshape II) takes 14 ms versus the original 7200ms! It can do 100000 scans (1.6MB data) in under 200 ms and 1000000 scans (15MB data) in under 2 seconds.
I am sure the code could be further improved. I recommend the Reshape II algoritm. It is fastest and can deal with variable channel counts. Modify as needed.
Attached is a LabVIEW 7.1 version of the benchmarking code, containing all algorithms. I have verified that all algorithms produce the same result (with the limitation that the cluster version only works for 6*I16+2*SGL data, of course). Remember that the reshape function is extremely efficient, because it does not move the data in memory. I have some ideas for further improvements, but this should get you going.
Message Edited by altenbach on 08-05-2005 03:06 PM
LabVIEW Champion . Do more with less code and in less time .
Attachments:
StringI16SGLCastingTimer.png ‏48 KB
StringtoI16andSGLArraysMODTimer.vi ‏120 KB