Best practices to manage large array inside an event loop?

Question

My program is a multi-event structure, where each event corresponds to a boolean value change by the user. The program begins by reading a .csv file, typically, 50000 rows and 100 columns, of string values. I add some columns, predict the final array size and write to two array controls - input and output. The input array always shows the original data as read from the file. The output array shows the current state of data. The user then presses button that triggers events, in each event I read the output array value, manipulate data, and write back to output array. If an event has to manipulate only the 7th column (for example), then I index that column, manipulate it, and replace that column in the control. I am using 'Value' property node to read and write data to the output array. I NEVER use build array function. I have to update the control at the end of each event to allow the user to see effect of the event.
Typically, my program runs fast. I can go through all events and even repeat them at a good speed. However, when I read .csv file second time, the program slows down, third time even slower. When I read the file, I predict the final size of input and output array and refresh the controls with new values from the .csv file. What am I doing wrong?
1. Should I use shift register in my event structure to read and update output array instead of using property node? Or, should I read and write using a Reference to the control?
2. When I read a control using property node, am I creating a second copy of the data?
3. I believe each time I read or write a control or indicator, LabVIEW switches to threads. This may be causing some delay but I doubt is the reason for my troubles. Right or wrong?
4. Is there a way to deallocate all input and output array control memory? Would it help to write a null array before I refill the controls with a new .csv file?
I can add screenshots of my code. I cannot provide the complete program due to security issues.
Using LV 8.5 Professional on Windows 7 64-bit.
Thanks,
Gurdas
Gurdas Singh
PhD. Candidate | Civil Engineering | NCSU.edu

Answer 1

Crossulz, I updated my code and replaced all property nodes of the output array indicator with shift register. I see some speed/memory improvements (more about that later).
1. Sorry about the wrong terminology. My output array is an indicator and not a control. I was using the word since I use this array to control the saved output. I am not sure I understood what you meant by writing right before the event structure. Did you mean having the terminal between the outer while loop (which has the shift register) and the event structure? Wouldn't that have it update the indicator each time the while loop runs? For the lack of a better solution, I created an event to update the indicator using shift register value. See attached screenshot, "Gurdas_3*.jpg"
2. Okay.
2a. Wires won't work because control and indicators are being manipulated in different events. Shift registers could work, but that means having many shift register, one for each control or indicator. I've read about Feedback nodes, not sure if they exist in LV 8.5.1. If they do, is that an option?
4. See attached screenshot "Gurdas_4*.jpg". I intiliaze the shift register with an empty array and then update it with the data when file is read. Is this what you meant? I still have an input data indicator that I am writing to. But this is one time; this indicator is not read or updated anywhere else.
4a. Notice the 3 branchout data wires just after the file read? Is that creating 3 copies by any chance?
5. See attached screenshot "Gurdas_5*.jpg". This is from the most computation intensive event that uses current data from the shift register. I am indexing many columns out. Is that creating extra copies of the columns?
About the improvements from using shift register. LV's memory usage as reported by Task Manager:
A. Before file read 70 MB
B. After file read 750 MB (includes writing one time to the input data indicator; if I do not write, memory usage is 450 MB)
C. After completing all steps done but never writing to output iindicator array 1.1 GB
D. After completing all steps done and writing to output iindicator array 1.5 GB (this used to be 2 GB previously)
The file I am reading is 47000 rows and 78 columns. I read the file as string. The ondisk size of the file is less than 20 MB.
6. Comparing steps A and B above, why does memory use increase so much more than the actual file size being read?
Thanks for helping me here!
Gurdas Singh
PhD. Candidate | Civil Engineering | NCSU.edu
Attachments:
Gurdas_03_UpdateOutputArray.JPG ‏41 KB
Gurdas_04_New-ReadCSV.JPG ‏125 KB
Gurdas_05_IndexArray.JPG ‏77 KB