Out of memory error while opening 1GB TDMS file, no splitting possible

Question

Hi all,
the file in question contains about 1,4 Million channels, each has 295 integer points. The channel name is the time stamp where its measurement has been taken. 
Diadem cannot load them completely and only displays the first 65.000 channels.
Labview (TDMS read) cannot even read small chunks of data. I've tried separating the file using Split TDMS File Memory Independent LV2009.vi (https://decibel.ni.com/content/docs/DOC-20054) but to no avail - the error I get is Code 2 (Memory is full).
I use Win7 and Labview 2012 SP1 32Bit.
Thanks in advance for any help.
Best regards, Markus
Solved!
Go to Solution.

Answer 1

Markus,
the term "channel" refers commonly to data coming from a single transducer. So if you have e.g. 50 transducers in your system, you should have 50 channels.
Acquiring data continuously generates chunks of data per time unit (e.g. 1kHz acquisition rate will generate 1000 samples per second). The chunks should be saved in the file to its appropriate channel. TDMS Write gives you the option to save data repeatedly to the same channels (again, group/channel name the important parameter!).
Nevertheless, the overall amount of data will not be less, even if you distribute them over channels in a better way. Still, loading ALL data at once will not be possible in a 32bit process.
BUT: If channels have all data as recommended, loading a singel channel at once will give you all data from the transducer for the complete measurement time. Then you can display/modify the data as requested, maybe store back to another file (or the same as new channel).
Having less channels could also improve loading times, but maybe there is no difference at all.... i don't have specific benchmarks on this.
Depending on the algorithms you are using, data reduction can improve overall times.
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.