Large Binary file to array

Hi,
I'm relatively new to Labview and am having trouble with large arrays. I'm using Win32 IO functions to read a 8MB file into Labview and this products a large U8 array. I want to produce a binary array containing the second bit of each byte in this array. For example:
File:00000000 00000010 00000000 00000010 00000010
Binary array: 01011
At the moment I'm using a For Loop and clearly its very inefficient. Is there any way to do the above in a relatively short period of time?
I used a subvi containing a For Loop and enabled Reentrant execution but this isnt very fast either.
Any help would be greatly appreciated,
John
Attachments:
image_bin.png ‏9 KB

Hi John,
You can use boolean operators on integers. No need to convert to boolean array. To extract a bit, just AND with a constant with the bit in question set. Also, you don't need the loop.
See attached example.
The quotient & remainder is one way to scale the resulting array to 1. Without this you'd get an array of 0's and 2's.
Hope this helps,
Daniel
EDIT: Sorry for repeating 10degree, I was distracted while writing the message
Message Edited by dan_u on 07-07-2009 03:52 PM

Similar Messages

Convert Large Binary File to Large ASCI File

Hello, I need some suggestions on how to convert a large binary file > 200MB to an ASCI File. I have a program that streams data to disk in binary format and now I would like to either add to my application or create a new app if necessary. Here is what I want to do:
Open the Binary File
Read a portion of the File into an array
Convert the array to ASCI
Save the array to a file
Go back and read more binary data
Convert the array to ASCI
Append the array to the ASCI file
Keep converting until the end of the binary file.
I should say that the binary data is 32-bits and I do need to parse the data; bits 0-11, bits 12-23, and bits 31-28, but I can figure that out later. The problem I see is that the file will be very large, perhaps even greater than 1GB and I don't have a clue how to read a portion of the file and come back and read another portion and then stop at the end of the file. I hope to save the data in a spreadsheet. If anyone has some experience with a similiar situation I'd appreciate any input or example code.
Thanks,
joe

sle,
In the future, please create a new thread for unrelated questions. To answer your question, you can use "Split Number" from the Data Manipulation palette.
Message Edited by jasonhill on 03-14-2006 03:46 PM
Attachments:
split number.PNG ‏2 KB

Read large binary file

How would you read a large binary file into memory please? I had a thought about creating a Byte Array on the fly, however you cannot createa byte array with a long so what happens when you reach the maximum value an integer can store?

a) You can map the file, instead of reading it physically.
b) Let's suppose that you are running Sun JVM in Windows 2000/XP/2003 and you have 4-GB of RAM in your machine (memory is so cheap nowadays...)
- Windows can not use the full 4GB (it reserves 2GB for itself, and if you buy a special server version, it reserves only 1GB for itself.)
- Java can't use more than 1.6 GB in Windows due to some arcane reasons. (Someone posted the Bug Database explanation in this forum.)
So you have an upper limit of 1.6 GB.
1.6GB is a value smaller than the size of an int, so you could have an array as big (using the class java.nio.ByteBuffer and the like). Try and see.

Any info on CRC, checksum, or other file integity VIs for large binary files?

Working on send rather large binary files (U16 stream to file) via internet. Would like to check for file integity via CRC or comparable checksum. Would appreciate any comments/suggestions

Hi Brian,
You said;
"Would appreciate any comments/suggestions".
You did not mention what transport mechanism you plan on using.
As I understand ALL of the standard mechanism use CRC of some form to ensure the validity of the packet BEFORE it is ever passed up the OSI 7-Layer model.
TCP/IP based protocols will see to it that all of the segments of a transfer are completed and in order.
UDP on the other hand is a broadcast type protocol and does not ensure any packets are recieved.
So,
At the very worst you should be able to handle your "sanity checks" by simply using a sequence value that is included in your out-going message. The reciever should just have to check if the current seq value is equal to the previous +1.
I co-developed an app that ut
ilized this technique to transfer status messages from a RT platform to a Windows machine. The status messages in this app where concidered FYI, so the sequence counter served as a way of determining if anything was missed.
I am insterested in others thoughts on this subject.
Ben
Ben Rayner
I am currently active on.. MainStream Preppers
Rayner's Ridge is under construction

Large binary file reading

Hi,
I'm currently using a java.io.BufferedInputStream to read a large binary file.
I recently discovered there is a chunk of data that shows up near the end of each file. (these files are binary and are XX to XXX mb in size)
Loading it all to memory first would kill my performance so I'd like to be able to come up with an alternate method.
Does Java or the class above offer a way I can
1) get the length of the file
2) seek to a point say 2000 bytes from the end so I can start reading the binary data?
Ideally I'd like to do a backwards read as that would be quickest. Is there a way to change the operation so that a 'read' would be reading backwards (from end to beginning)?
For me speed is the #1 thing i have to worry about. So to be able to seek forward several hundred thousand bytes at a time would help tremendously.

how does the 'skip' method work? Probably by using OS specific calls to read to a point in the file.
maybe I could
'skip' length - 20k from the start or something like
that.
Yep.

Reading large binary files into an array for parsing

I have a large binary log file, consisting of binary data separted by header flags scattered nonuniformly thorughout the data. The file size is about 50M Byte. When I read the file into an array, I get the Labview Memory full error. The design of this is to read the file in and then parse it fro the flags to determine where to separate the data blocks in the byte stream.
There are a few examples that I have read on this site but none seem to give a straight answer for such a simple matter. Does anyone have an example of how I should approach this?

I agree with Gerd. If you are working with binaries, why not use U8 instead of doubles.
If the file is indeed 50MB, then the array should be expecting 52428800 elements, not 50000000. So if you read the file in a loop and populate an element at a time, you could run out of memory fast because any additional element insertion above 50000000 may require additional memory allocation of the size above 50000000 (potentially for each iteration). This is just speculation since I don't see the portion of your code that populates the array.
Question: Why do you need an array? What do you do with the data after you read it? I agree with Altenbach, 50MB is not that big, so working with a file of such a size should not be a problem.

Binary File to Array

I need to choose a file with a file chooser and then convert it to an array. The file is a binary file and I just want every byte out of it as an element in a very large byte array. Im not quite sure how to go about this, can someone who has done this or knows how guide me in the right direction?
When I run the enclosed vi it acts as if the file is not 512 bytes big and says that an EOF has been reached. Im sure Im just doing it plain wrong tho .
The zip contains two of the files I wish to read in as well as my poor attempt to read the file into a byte array. The vi has two main problems:
1) It doesn't work
2) It doesn't take into account variably sized files
Thanks
Cason Clagg
SwRI
LabView 7.1, Windows XP
Attachments:
FileOpen.zip ‏12 KB

(1) You wire a byte stream type of I32, meaning you read 4x as many bytes as intended.
--> solution: right click in the type constant and select Representation:U8.
(2) You can get the actual file size e.g. with the eof function.
See attached modification (LabVIEW7.1).
... and don't forget to close the file when done
Message Edited by altenbach on 07-21-2005 06:16 PM
LabVIEW Champion . Do more with less code and in less time .
Attachments:
OpenFileMOD.vi ‏24 KB

UTL_FILE write_error when writing large binary files to unix os

I am trying to write large files to a folder in unix from a table containing a BLOB object. The procedure below is called by another procedure I have written to do this. It works in windows environment fine with files up to 360MB. When I run this exact same procedure in UNIX I get an initialization error. When I change the WB in the fopen call to W it works. I can store all the files I want up to 130MB in size. The next size larger file I have is 240MB and it fails after writing the first 1KB passing the utl_file.write_error message. If someone can help me to diagnose the problem, I would really appreciate it. i have been trying everything I can think of to get this to work.
Specifics are, the windows version is 10GR2, on unix we are running on Sun Solaris 9 using 9iR2
PROCEDURE writebin(pi_file_name IN VARCHAR2, pi_file_url IN VARCHAR2, pi_file_data IN BLOB)
IS
v_file_ref utl_file.file_type;
v_lob_size NUMBER;
v_raw_max_size constant NUMBER := 32767;
v_buffer raw(32767);
v_buffer_offset NUMBER := 1;
-- Position in stream
v_buffer_length NUMBER;
BEGIN
-- WB used in windows environment. W used in unix
v_lob_size := dbms_lob.getlength(pi_file_data);
v_file_ref := utl_file.fopen(pi_file_url, pi_file_name, 'WB', v_raw_max_size);
v_buffer_length := v_raw_max_size;
WHILE v_buffer_offset < v_lob_size
LOOP
IF v_buffer_offset + v_raw_max_size > v_lob_size THEN
v_buffer_length := v_lob_size -v_buffer_offset;
END IF;
dbms_lob.READ(pi_file_data, v_buffer_length, v_buffer_offset, v_buffer);
utl_file.put_raw(v_file_ref, v_buffer, TRUE);
v_buffer_offset := v_buffer_offset + v_buffer_length;
END LOOP;
utl_file.fclose(v_file_ref);
END writebin;
Message was edited by:
user599879

check if this cample code helps -
CREATE OR REPLACE PROCEDURE prc_unload_blob_to_file IS
vlocation      VARCHAR2(16) := ‘LOB_OUTPUT’;
vopen_mode     VARCHAR2(16) := ‘w’;
bimax_linesize NUMBER := 32767;
v_my_vr        RAW(32767);
v_start_pos    NUMBER := 1;
v_output       utl_file.file_type;
BEGIN
FOR cur_lob IN (SELECT vmime_type,
blob_resim,
vresim,
dbms_lob.getlength(blob_resim) len
FROM tcihaz_resim a
WHERE rownum < 3 -- for test purposes
ORDER BY a.nresim_id) LOOP
v_output := utl_file.fopen(vlocation,
cur_lob.vresim,
vopen_mode,
bimax_linesize);
dbms_output.put_line(’Column length: ‘ || to_char(cur_lob.len) || ‘ for file: ‘ ||
cur_lob.vresim);
v_start_pos := 1;
IF cur_lob.len < bimax_linesize THEN
dbms_lob.READ(cur_lob.blob_resim,
cur_lob.len,
v_start_pos,
v_my_vr);
utl_file.put_raw(v_output,
v_my_vr,
autoflush => TRUE);
dbms_output.put_line(’Finished Reading and Flushing ‘ || to_char(cur_lob.len) ||
‘ Bytes’ || ‘ for file: ‘ || cur_lob.vresim);
ELSE
dbms_lob.READ(cur_lob.blob_resim,
bimax_linesize,
v_start_pos,
v_my_vr);
utl_file.put_raw(v_output,
v_my_vr,
autoflush => TRUE);
dbms_output.put_line(’Finished Reading and Flushing ‘ || to_char(cur_lob.len) ||
‘ Bytes’ || ‘ for file: ‘ || cur_lob.vresim);
END IF;
v_start_pos := v_start_pos + bimax_linesize;
WHILE (v_start_pos < bimax_linesize) LOOP
-- loop till entire data is fetched
dbms_lob.READ(cur_lob.blob_resim,
bimax_linesize,
v_start_pos,
v_my_vr);
utl_file.put_raw(v_output,
v_my_vr,
autoflush => TRUE);
dbms_output.put_line(’Finished Reading and Flushing ‘ ||
to_char(bimax_linesize + v_start_pos - 1) || ‘ Bytes’ ||
‘ for file: ‘ || cur_lob.vresim);
v_start_pos := v_start_pos + bimax_linesize;
END LOOP;
utl_file.fclose(v_output);
dbms_output.put_line(’Finished successfully and file closed’);
END LOOP;
END prc_unload_blob_to_file;
set serveroutput on
set timing on
create or replace directory LOB_OUTPUT as ‘/export/home/oracle/tutema/’;
GRANT ALL ON DIRECTORY LOB_OUTPUT TO PUBLIC;
exec prc_unload_blob_to_file ;
Column length: 3330 for file: no_image_found.gif
Finished Reading and Flushing 3330 Bytes for file: no_image_found.gif
Finished successfully and file closed
Column length: 10223 for file: OT311.gif
Finished Reading and Flushing 10223 Bytes for file: OT311.gif
Finished successfully and file closed
PL/SQL procedure successfully completedWith 9iR2 PLSQL can write binary files using UTL_FILE put_raw function, prior to Oracle9iR2 you will need to create an external procedure with Java, C, VB or some 3gl language.
Some references -
http://asktom.oracle.com/pls/ask/f?p=4950:8:::::F4950_P8_DISPLAYID:6379798216275
Oracle® Database PL/SQL Packages and Types Reference 10g Release 2 (10.2)
UTL_FILE - http://download-uk.oracle.com/docs/cd/B19306_01/appdev.102/b14258/u_file.htm#sthref14095
http://psoug.org/reference/dbms_lob.html
Metalink Note:70110.1, Subject: WRITING BLOB/CLOB/BFILE CONTENTS TO A FILE USING EXTERNAL PROCEDURES

Somewhat large binary file has trouble being opened

Hi,
I'm a new Labview user (Labview 8.0), and I'm trying to convert a
binary file into a wave file and do a lot of other things with that
binary file (like analyzing frequency via a spectrum graph). My file
works fine for files under 150MB, but once the file size is larger than
that, the computer slows way way down and I cannot collect the entire
data set. I already have 1GB of RAM (512x2) and I don't want to pay
more money if the issue can be fixed within Labview.
Is there a way to split up the binary file into little pieces when
being read, or is there another way to go about doing this? I'm using
the File Open dialog to read the file.
If anyone could provide a step-by-step solution to this, I'd be very
grateful. Thanks everyone.
Jennings

Thanks for the reply,
That would kind of work, but ultimately, I need a graph that
contains the entire file. In other words, I need to convert the binary
file to a wave file that I can view as a graph. If I only open one
half, then I would only see the graph from that first half, which is
not useful to the project. I need to view the wave file in its
entirety as a waveform graph, so is there any way to add the data from
the second half of the file to the first half?
I guess it comes down to the memory buffer size, right? Labview tries
to load the entire file onto the RAM, but I need it to load a portion
onto the RAM, save that portion on the hard drive, load a second
portion onto the RAM, and add that portion to the first portion on the
hard drive, and so forth.
Is there any function in labview that can do this? Or can I only
manipulate data using the RAM, and not the hard drive?
Thanks for any and all help!
Jennings
JLS wrote:
> Hello,
>  
> From your previous post, it sounds like you're able to open and read from the file - can you try reading half (the get file size function will be helpful) and immediately writing that half to another file?  If this works, you can try the same thing with the second half using the set file position function.  You could repeat this process until you had files which were of manageable size for your application/processing needs.
>  
> I hope this helps!
>  
> Best Regards,
>  
> JLS

NIO- how to best transfer large binary files 200mb

Hi there,
For my dissertation I am in the process of implementing a LAN- NIO based distributed system most of it is working except I cant figure out how to best transfer large files >200mb.
1. Client registers as available
2. Server notifies that a job is available on file x
3. Client requests file x
4. Server transfers file x- without reading the whole file into memory-
5- Client does its thing and tells the server the results are ready-
6. Server requests results.
This is all implemented, except for the actual file transfers, which just wont work the way I want it. I got access to a shared drive that I could copy to and from, but Java still reads all bytes fully into memory. I did a naughty workaround of calling windows native copy command ( runtime() ) which does the trick, though I would prefer that my app did the file handling since I need to know when things are done.
Can anyone provide me with a link to an example / tutorial of how to do this, or is there a better way of transferring large files like tunneling, streaming, ftp or the old way of splitting into chunks with headers (1024 bits, byteFlip() etc) how is this usually done out there in the proper world?
I did search the forum, and found hints and tips though I thought perhaps I would see if there is a best practice for this problem--
Thanks,
Thorsan

Hi,
I have tried various approaches with out much luck, think part of the problem is I have never done file transfers in java over networks before...
At the moment I let the user select files from a JFileChooser, and store the String path+filename in a list. When a file is to be transferred I do the following:
static void copyNIO(File f, String newFile) throws Exception {
        File f2 = new File(newFile);
        FileChannel src = new FileInputStream(f).getChannel();
        FileChannel dst = new FileOutputStream(f2).getChannel();
        dst.transferFrom(src, 0, src.size());
        src.close();
        dst.close();
    }This lets me copy the file to wherever I want to without going through a socket, since I am developing in a lan environment this was ok for debugging. Though sooner or later it has to go through the network.
I have allso played around with ObjectOutputStream and ObjectInputStream but again I just cant figure out how to inhibit my source from loading the whole file into memory. I want to read / write in blocks to keep the memory usage to a minimal.
Thorsan

Urgent - pls help - Problem while inserting binary file into Oracle DB

Hi,
I am trying to insert binary files into a Blob column in a Oracle 10G table.
The binary files would be uploaded by the web users and hence come as multipart request. I use apache commons upload streaming API to handle it. Finally i am getting a input stream of the uploaded file.
The JDBC code is
PreparedStatement ps=conn.prepareStatement("insert into bincontent_table values(?)");
ps.setBinaryStream(1,inStream,length);
My problem starts when i try to find the length of the stream. available() method of inputstream does not return the full length of the stream. so i put a loop to read thru the stream and find the length as shown below
int length=0;
while((v=inStream.read())!=-1)
length++;
Now, though i got the length, my stream pointer has reached the end and i cant reset it(it throws an error if i try).
So i copied the stream content to a byte array and created an ByteArrayInputStream like this.
tempByteArray=new byte[length];
stream.read(tempByteArray,0,length);
ByteArrayInputStream bais=new ByteArrayInputStream(tempByteArray);
Now if i pass this bytearray input stream instead of the normal input stream to the prepared statement's setBinaryStream() method it throws an error as
"ORA-01460: unimplemented or unreasonable conversion requested".
Now how to solve this?
My doubts are ,
1) preparedStatement.setBinaryStream(int parameterIndex, InputStream x, int length) expects an inputstream and its length. if i have the stream how to find its length with out reading the stream?
2) Also as the length parameter is a integer, what if i have a large binary file whose length runs more than the capacity of integer
3) Alternatively there is a setBlob(int i, Blob x) in prepared statement. But how to instantiate a Blob object and set it here
4) Is there any better way to do this.
Thanks in advance

"ORA-01460: unimplemented or unreasonable conversion
requested".When the setBinaryStream method is used, the driver may have to do extra work to determine whether the parameter data should be sent to the server as a LONGVARBINARY or a BLOB (reference: javadoc)
1) preparedStatement.setBinaryStream(int parameterIndex,
InputStream x, int length) expects an inputstream and its length. if i
have the stream how to find its length with out reading the stream?no. stream may have no specified length. i think you have wrong understanding about stream.
2) Also as the length parameter is a integer, what if i have a large
binary file whose length runs more than the capacity of integer
3) Alternatively there is a setBlob(int i, Blob x) in prepared statement.
But how to instantiate a Blob object and set it here
4) Is there any better way to do this.use ps.setBlob(1, instream) instead

Complexity of searhing in binary files

Hello!
I have question regarding complexity when searching in a binary file. I have a app that needs to perform a search for a certain string in a large binary file (> 1Gb). For the moment, I read one line of the file into a byte array, convert it to a string and than look for any matches with indexOf. This, however, is (yeah, you guessed right..) an extremly slow search algorithm. By the time I wrote it was a good option, but now I am thinking of re-making it.
So my question, is it any idea to try something else? Will it be faster? Is it even possible? Can one use regexp in binary files? A bit confused in this matter...
All ideas are welcome!

Yes, it true. ASCII characters are readable. Maybe i
should have used another word for "line". 1 line = 1
frame ie. each frame have a line feed attached.
The tricky part is that each frame contains a 4byte
preamble followd by a unique identifier, which I must
keep track of. That is, a search hit in one frame
should return the identifier of the frame.
Thanks for the tip about the search algorithms!
I bealive the qustion is: How do I find a specific
string in a byte array?Convert the search string to a byte array and then search for that byte array in the file byte array. This way you are only converting once.

How do I open a LeCory .trc binary file in labview

We have several Lecory scopes witch create large binary files to save room on there hard drive. the files are .rtc. I would like to open these files using labview

Alan from LeCroy Tech Support here...
Attached is a decode I put together, mostly using code from the lcwave driver. The VI will read a binary file, and return the voltage array, horizontal offset (aka "t0") and the horizontal interval (dt), among other things. It'll handle both BYTE and WORD mode files, as well as hi-Lo and Lo-Hi byte ordering.
If what you are needing is a tool to decode a large number of files that already exist, then all you need to do is use the "Batch Waveform Conversion" feature of LeCroy's ScopeExplorer application. This is available at http://www.lecroy.com/tm/Library/Software/home.asp?menuid=8. This will convert all trc files in a folder to ASCII with columns for time and voltage.
If instead you are needing to decode files so that you can then analyze the data, then the attached VI or the diadem plug in is certainly a better solution. I couldn't tell you which is faster...
Feel free to contact me directly if you have any questions!
Best Regards,
Alan
Alan Blankman, Technical Product Marketing Manager and LabVIEW Developer
LeCroy Corporation
800-553-2769 x 4412
http://www.lecroy.com
[email protected]
Attachments:
Decode waveform from file.vi ‏173 KB

Problem saving 2D array to binary file

I am getting some strange results when trying to write a transposed 2D array of Unisgned 8-bit data to a binary file. Attached is an example program ("071026_ArraySave_Bug.vi") which demonstrates the suspicious behavior.
The program generates a small 2D array of U8 data, then saves it to two temporary files using two different methods.
1) Row-by-row: The 2D array is auto-indexed, and the indexed row is written to file at once (with the prepended size option disabled).
2) Point-by-point: Two For loops are used to auto-index each element and write it to file individually.
The saved files are then reopened and the data is read into two 1D arrays for display.
The bug occurs when the generated 2D array is transposed before being saved to file. In the point-by-point method, the data is always saved to the file as desired, i.e. by row or by column depending on whether the data was transposed. However, in the row-by-row method the data is incorrectly saved when the input data is transposed. Without transposing the array, the row-by-row results are correct.
Saving the array row-by-row is significantly faster for larger arrays, so this method is desired.
Note that the option to transpose the input data in the attached VI is performed using a case structure with a boolean constant ("Bug Control"). This is important because:
1) The bug does not occur if the boolean constant is replaced with a control.
2) The bug does not occur if the case structure is replaced with a "Select" node from the Comparison Palette.
3) The bug still occurs if the case structure is removed such that the array is always transposed (i.e. keep the True case).
For now, I have been able to circumvent the bug in practice by transposing the array and then passing it through a Select with a constant True (the False case uses the untransposed array). This allows the faster row-by-row method to be used without error, but requires new memory allocation for the transposed array.
I will appreciate any help that can be given regarding this problem, or simply confirmation that it is indeed a bug in Labview 8.5. It is worth noting that the same program did not produce any errors when run in Labview 7.
Thank you for your help,
David Viggiano
Attachments:
071026_ArraySave_Bug.vi ‏29 KB

This is a known bug (CAR : 4DP855N3) , related to memory management. see this thread
If I remember well, Altenbach shown somewhere that forcing the transpose operation to produce a copy of the memory was a proper workaround.
Message Edité par chilly charly le 10-26-2007 08:40 PM
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left...
Attachments:
071026_ArraySave_Bug[1].png ‏2 KB

Multiple writes of a DBL 2D array to binary file

Hello.
Does anyone have an example of writing a 2D array to a binary file multiple times?
Specifically, I need to write a 2D array of DBL to a binary file mulitple times and then (later) be able to read it back (reconvert to DBL). All of the examples that I've found only do one write and then close the file. I need to leave the file reference open and write data to it over the period of a few days. The data comes out of a queue that is fed from a DAQ loop. (In an acquistion loop I feed a queue and in a slower loop I dequeue the data and write it to disk)
I believe it has something to do with the pos mode and offset mode but I can't figure it.
I haven't been able to successfully read the data back yet from the .bin file so I'm not sure if I'm even writing it correctly.
The closest example that I've found is the High Speed Data Logger and High Speed Data Reader but I haven't had any luck with those.
Thanks in advance.
Steve
LabVIEW 7.1
Solved!
Go to Solution.

To change the representation of a number, use the Numeric > Conversion sub-palette.
The example below shows that the wires contain the representation info independently of the destination indicators : you can connect different types of numerics to the same wire and LabVIEW will automatically adapt to the type (the coercion dots indicate that a convertion has been done). Everything you should want to know about converting numbers is explained in the LabVIEW Help : "Conversion VIs and Functions"
Message Edited by chilly charly on 07-30-2005 10:57 AM
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left...
Attachments:
Convert to single.png ‏27 KB

Large Binary file to array

Similar Messages

Maybe you are looking for