Efficiently accumulating arrays

Question

Hello,
I am attempting to acquire a reasonable amount of data (between 100000
and 1000000 floats) that I would like to accumulate into an array of
single-precision floats.
Efficiency is important due to time requirements of the device that I
am acquiring from.
I have observed that using a large array size - such as those
mentioned above - results in my program running considerably slower
throughout. The array is pre-allocated to the correct size using
"Initialize Array" and then elements are replaced one-by-one using
"Replace array Subset". This I believe to be the most efficient way
of doing it from what I have read, for example in:
http://zone.ni.com/devzone/conceptd.nsf/2d17d611efb58b22862567a9006ffe76/979f737147fd5c01862569f3
00638674?OpenDocument
Figure 2 on that page is quite similar to what I have (although much
simplified). For each iteration of the loop I have my array wired to
"Replace Array Subset", with the output wired to a local variable of
the input array.
Is there any reason why Labview should perform so poorly with large
array sizes? My suspicion is that a copy of the entire array or a
size check is being performed somewhere, but neither should be
necessary in this case.
If anyone could offer advice as to how to accumulate data into an
array more efficiently I would be most grateful.
I am using Labview 6.1
Regards,
Stephen

Answer 1

> being received in blocks of 20 bytes which I then cast to a U32 array
> so that I can "Byte-Swap" and "Word-Swap" before casting to
> single-precision floats to extract the 5 floats which are the real
> data. Each float is then appended to an array (since each of the 5
> floats represents a different value) and then the 5 arrays are passed
> to the next iteration using shift-registers as suggested by Mads.
>
> I am aware that casting is inefficient in Labview as it makes a copy
> of the data(?), but it seems unavoidable in this case as I must
> perform byte-swapping and Labview will only byte-swap integers, even
> though I really want a byte-swapped float! I could move the
> byte-swapping out of the acquisition phase, but this should be a
> linear problem and so would not cause my program to slow down as it
> does.
It is probably just terminology, but here are some comments. After
casting your 5 U32s to 5 floats, you say you extract the data and append
it to an array. If you are truly using build array, then you are
resizing the big array each time through the loop.
Here is what you should try instead. Before starting the loop use
Initialize Array to build a suitably sized buffer of zeros or other
initialized value and wire that to your shift register. Inside the loop
use Replace Array Subset to put all five floats from your array into the
large array at the correct offset, or in older LVs, use five index nodes
and five replaces, or use a for loop that autoindexes your small array,
running five iterations, and does an Element Replace on the large array
at the correct location.
If you don't know how much data will arrive before entering the loop,
then take a guess and inside the loop extend the array using Reshape
Array once in awhile to add another couple thousand elements. You will
also need to shrink the array after the loop, again using reshape.
The key here is that resizing a big array gets more and more expensive
as it gets larger. If you resize the array each iteration of the loop,
this is far more expensive than the occasional resize, say each 1000
elements, which is still more expensive that one resize before the loop
begins.
If this doesn't help, it might help if you were to post the VI or a
picture of the VI diagram.
Greg Mckaskle