Read large binary file

How would you read a large binary file into memory please? I had a thought about creating a Byte Array on the fly, however you cannot createa byte array with a long so what happens when you reach the maximum value an integer can store?

a) You can map the file, instead of reading it physically.
b) Let's suppose that you are running Sun JVM in Windows 2000/XP/2003 and you have 4-GB of RAM in your machine (memory is so cheap nowadays...)
- Windows can not use the full 4GB (it reserves 2GB for itself, and if you buy a special server version, it reserves only 1GB for itself.)
- Java can't use more than 1.6 GB in Windows due to some arcane reasons. (Someone posted the Bug Database explanation in this forum.)
So you have an upper limit of 1.6 GB.
1.6GB is a value smaller than the size of an int, so you could have an array as big (using the class java.nio.ByteBuffer and the like). Try and see.

Similar Messages

Reading large binary files into an array for parsing

I have a large binary log file, consisting of binary data separted by header flags scattered nonuniformly thorughout the data. The file size is about 50M Byte. When I read the file into an array, I get the Labview Memory full error. The design of this is to read the file in and then parse it fro the flags to determine where to separate the data blocks in the byte stream.
There are a few examples that I have read on this site but none seem to give a straight answer for such a simple matter. Does anyone have an example of how I should approach this?

I agree with Gerd. If you are working with binaries, why not use U8 instead of doubles.
If the file is indeed 50MB, then the array should be expecting 52428800 elements, not 50000000. So if you read the file in a loop and populate an element at a time, you could run out of memory fast because any additional element insertion above 50000000 may require additional memory allocation of the size above 50000000 (potentially for each iteration). This is just speculation since I don't see the portion of your code that populates the array.
Question: Why do you need an array? What do you do with the data after you read it? I agree with Altenbach, 50MB is not that big, so working with a file of such a size should not be a problem.

Large binary file reading

Hi,
I'm currently using a java.io.BufferedInputStream to read a large binary file.
I recently discovered there is a chunk of data that shows up near the end of each file. (these files are binary and are XX to XXX mb in size)
Loading it all to memory first would kill my performance so I'd like to be able to come up with an alternate method.
Does Java or the class above offer a way I can
1) get the length of the file
2) seek to a point say 2000 bytes from the end so I can start reading the binary data?
Ideally I'd like to do a backwards read as that would be quickest. Is there a way to change the operation so that a 'read' would be reading backwards (from end to beginning)?
For me speed is the #1 thing i have to worry about. So to be able to seek forward several hundred thousand bytes at a time would help tremendously.

how does the 'skip' method work? Probably by using OS specific calls to read to a point in the file.
maybe I could
'skip' length - 20k from the start or something like
that.
Yep.

"Read from Binary File" and efficiency

For the first time I have tried using Read from Binary File on sizable data files, and I'm seeing some real performance problems. To prevent possible data loss, I write data as I receive them from DAQ, 10 times per second. What I write is a 2-D array of double, 1000 data points x 2-4 channels. When reading in the file, I wish I could read it as a 3-D array, all at once. That doesn't seem supported, so I repeatedly do reads of 2-D array, and use a shift register with Build Array to assemble the 3-D array that I really need. But it's incredibly slow! It seems that I can only read a few hundred of these 2-D items per second.
It also occurred to me that the Build Array being used in a shift register to keep adding on to the array, could be a quadratic-time operation depending on how it is implemented. Continually and repeatedly allocating bigger and bigger chunks of memory, and copying the growing array at every step.
So I'm looking for suggestions on how to efficiently store, efficiently read, and efficiently reassemble my data into the 3-D array that I need. Perhaps I could simplify life if I had "write raw data" and "read raw data" operations that write and read only the numbers and no metadata.then I could write the file and read it back in any size chunks I please -- and read it with other programs besides. But I don't see them in the menus.
Suggestions?
Ken
Solved!
Go to Solution.

I quote the detailed help from Read from Binary File:
data type sets the type of data the function uses to read from
the binary file. The function interprets the data starting at the current file
position to be count instances of data type.
If the type is an array, string, or cluster containing an array or string, the
function assumes that each instance of that data type contains size information.
If an instance does not include size information, the function misinterprets the
data. If LabVIEW determines that the data does not match the type, it sets data
to the default for the specified type and returns an error.
So I see how I could write data without any array metadata by turning off "prepend array or string size information", but I don't see any way to read it back in such bare form. If I did, I'd have to tell it how long an array to read, and I don't see where to do that. If I could overcome this, I could indeed read in much larger chunks.
I'll try the auto-indexing tunnel anyway. I didn't tell you the whole truth, the 3-D array is actually further sliced up based on metadata that I carry, and ends up as a 4-D array of "runs". But I can do that after the fact instead of trying to do it with shift registers as I'm reading.
Thanks,
Ken

Convert Large Binary File to Large ASCI File

Hello, I need some suggestions on how to convert a large binary file > 200MB to an ASCI File. I have a program that streams data to disk in binary format and now I would like to either add to my application or create a new app if necessary. Here is what I want to do:
Open the Binary File
Read a portion of the File into an array
Convert the array to ASCI
Save the array to a file
Go back and read more binary data
Convert the array to ASCI
Append the array to the ASCI file
Keep converting until the end of the binary file.
I should say that the binary data is 32-bits and I do need to parse the data; bits 0-11, bits 12-23, and bits 31-28, but I can figure that out later. The problem I see is that the file will be very large, perhaps even greater than 1GB and I don't have a clue how to read a portion of the file and come back and read another portion and then stop at the end of the file. I hope to save the data in a spreadsheet. If anyone has some experience with a similiar situation I'd appreciate any input or example code.
Thanks,
joe

sle,
In the future, please create a new thread for unrelated questions. To answer your question, you can use "Split Number" from the Data Manipulation palette.
Message Edited by jasonhill on 03-14-2006 03:46 PM
Attachments:
split number.PNG ‏2 KB

Read an avi file using "Read from binary file" vi

My question is how to read an avi file using "Read from binary file" vi .
My objective is to create a series of small avi files by using IMAQ AVI write frame with mpeg-4 codec of 2 second long (so 40 frames in each file with 20 fps ) and then send them one by one so as to create a stream of video. The image are grabbed from USB camera. If I read those frames using IMAQ AVI read frame then compression advantage would be lost so I want to read the whole file itself.
I read the avi file using "Read from binary file" with unsigned 8 bit data format and then sent to remote end and save it and then display it, however it didnt work. I later found that if I read an image file using "Read from binary file" with unsigned 8 bit data format and save it in local computer itself , the format would be changed and it would be unrecognizable. Am I doing wrong by reading the file in unsined 8 bit integer format or should I have used any other data types.
I am using Labview 8.5 and Labview vision development module and vision acquisition module 8.5
Your help would be highly appreciated.
Thank you.
Solved!
Go to Solution.
Attachments:
read avi file in other data format.JPG ‏26 KB

Hello,
Check out the (full) help message for "write to binary file"
The "prepend array or string size" input defaults to true, so in your example the data written to the file will have array size information added at the beginning and your output file will be (four bytes) longer than your input file. Wire a False constant to "prepend array or string size" to prevent this happening.
Rod.
Message Edited by Rod on 10-14-2008 02:43 PM

Issue in reading a binary file, with 'Flatten to String' data.

I'm facing issue while reading a binary file (created using LabVIEW).
I've mentioned everything (issue and method to reproduce it) within the attached VI.
Same vi is attached in 2012 and 8.0 versions.
Regards
I am not allergic to Kudos, in fact I love Kudos.
Make your LabVIEW experience more CONVENIENT.
Solved!
Go to Solution.
Attachments:
Issue in VI.vi ‏26 KB
Issue in VI (Version 8.0).vi ‏43 KB

moderator1983 wrote:
crossrulz wrote:
The reading of a string from a binary file stops at a NULL character (0x00). When the first character is 0x00, you are just reading the one character. I would suggest writing to a byte array since you are doing the inverting. And then you can read as a byte array.
crossrulz:
U rocks..!!
you have hit bull's eye...!!
After playing around a little more, I think I might have misinformed you a little. If you explicitly tell it a string, it looks for the string length at the very beginning and reads that length of bytes as a string. It appears that if you implicitly tell it to read a string (not wire the data type) it reads all of the bytes directly, including the length of string you wrote.
Regardless, my advice is the same. You should just write and read using byte array. It is less conversions if you are performing your "encryption".
EDIT: Here's a snippet of the VI I was playing with to figure this out.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
Attachments:
String Binary File.png ‏12 KB

What's the best way for reading this binary file?

I've written a program that acquires data from a DAQmx card and writes it on a binary file (attached file and picture). The data that I'm acquiring comes from 8 channels, at 2.5MS/s for, at least, 5 seconds. What's the best way of reading this binary file, knowing that:
-I'll need it also on graphics (only after acquiring)
-I also need to see these values and use them later in Matlab.
I've tried the "Array to Spreadsheet String", but LabView goes out of memory (even if I don't use all of the 8 channels, but only 1).
LabView 8.6
Solved!
Go to Solution.
Attachments:
AcquireWrite02.vi ‏15 KB
myvi.jpg ‏55 KB

But my real problem, at least now, is how can I divide the information to get not only one graphic but eight?
I can read the file, but I get this (with only two channels):
So what I tried was, using a for loop, saving 250 elements in different arrays and then writing it to the .txt file. But it doesn't come right... I used 250 because that's what I got from the graphic: at each 250 points it plots the other channel.
Am I missing something here? How should I treat the information coming from the binary file, if not the way I'm doing?
(attached are the .vi files I'm using to save in the .txt format)
(EDITED. I just saw that I was dividing my graph's data in 4 just before plotting it... so It isn't 250 but 1000 elements for each channel... Still, the problem has not been solved)
Message Edited by Danigno on 11-17-2008 08:47 AM
Attachments:
mygraph.jpg ‏280 KB
Read Binary File and Save as txt - 2 channels - with SetFilePosition.vi ‏14 KB
Read Binary File and Save as txt - with SetFilePosition_b_save2files_with_array.vi ‏14 KB

Reading a binary file to string variable does not populate correctly

Hi, I am new to Adobe Air/Flex and I'm trying to read a
binary file which also contains text in "cleartext". The problem
I'm having is that when I call FileStream.readUTFBytes method, only
the first 6 characters are showing up in my string variable
"contents" when I debug it in FlexBuilder or use a trace command
and debug it. I have also tried with other types of files but I
have a similar problem unless it's a non-binary file.
Am I doing something incorrectly or should I be reading a
binary file differently than the way I'm reading it currently?
The source code is shown below.
TIA,
Magnus
<?xml version="1.0" encoding="utf-8"?>
<mx:WindowedApplication xmlns:mx="
http://www.adobe.com/2006/mxml"
layout="absolute">
<mx:Script>
<![CDATA[
import flash.filesystem.File;
private function readFiles():void{
var feedFile:File = File.userDirectory.resolvePath( "Local
Settings/Application Data/Microsoft/Feeds/AppScout~.feed-ms" );
var stream:FileStream = new FileStream();
stream.open( feedFile, FileMode.READ );
var contents:String =
stream.readUTFBytes(stream.bytesAvailable);
trace(contents);
stream.close();
]]>
</mx:Script>
<mx:Button x="121" y="66" label="Button"
click="readFiles()" />
</mx:WindowedApplication>

It's difficult to tell what it is, it looks like a binary
pipe symbol but I can copy it from TextPad for example. Some of the
characters following it cannot be copied from TextPad which I
assume is because it's null. I can read the whole file in C#/.Net
and assign it to a string variable without any problems but perhaps
Air is somewhat limited to binary content.

Read From Binary File doesn´t work on MCB2400 in LV2009 Embedded ARM

Hello,
I try to read a Binary File from SD-Card on my MCB2400 Board with LV2009 Embedded for ARM.
But the output is always 0, if I use my VI on the MCB2400. If use the same VI on the PC it works fine with the same binary file.
The
access to the SD-Card on the MCB2400 works in other cases finde, if I
try to read from a text-file it works without any problems.
Are thre any constraints for the "Read From Binary File"-Node in Embedded in comparison to the same node on PC ?
I have noticed that there is also a problem
with the reading of textfiles. If the sice of the file is about 100Byte
it doesn´t work anymore, too. I can´t understand it, because I read
always just one Byte. And even if the implementation in Labview is so
bad that it allways reads the total file in the ram it sould work. The
MCB2400 has 32MegaByte RAM, so 100Byte or even a few MegaByte should
work.
But this doesn´t seem to be the Problem for the Binary-Problem. Because even a 50byte Binary-File doesn´t work.
bye & thanks
amin
Solved!
Go to Solution.
Attachments:
SD_Card_Read-test.vi ‏12 KB

Hello,
thank you for your Help.
But I just want to read a Binary File, which is build by another program. And this is coded with 8Bit (like a normal Binary File) and not just with 7Bit (ASCII). So the workaround doesn't work in my case.
I posted the Test-VI in my first post (here once again as picture). And it works fine on the PC, but if I try it on my MCB2400 the "Read Form Binary File" Node doesn't work.
And it is also possible to open the Bin file with the "Read Text File" Node and see the cryptic content of the Bin-File. So the Problem seems to be in the "Read Form Binary File" Node.
bye & thanks again
amin
Message Edited by aminat on 09-30-2009 03:28 AM

How can I read a binary file?

In my AIR application, I'm trying to read a binary file and then post it via an HTTPService to a remote ColdFusion component that accepts a "binary" type argument. The result of the post is always:
The FILEOBJ argument passed to the addFile function is not of type binary.
Here is how I'm reading the file in Flex, where fileObj is of type File:
var fs : FileStream = new FileStream();
var bytes : ByteArray = new ByteArray();
fs.open(fileObj, FileMode.READ);
fs.readBytes(bytes, 0, fs.bytesAvailable);
fs.close();
I then pass "bytes" as the parameter to my HTTPService's send call, like so:
var params : Object = {method : "addFile", fileObject : bytes};
service.send(params);
When I debug, bytes contains the correct file size. Is it not in binary form? I've tried several other suggestions, but with the same result, so would appreciate any help.
Thanks!

One more step...
short are U16 integer
double are double precision float
bool seem to be 2 bytes (= U16)
char are string (variable length)
rgb are U16 integer, with high order byte = 0
rect should be 4 x U16 (top, left, bottom, right)
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left...

Reading a binary file (byte swapping?): HELP!

Hi,
Below are 2 simple programs, one in C and the other in Java that read a binary file. The first 3 integer numbers read by the C program are:
910, 1024437292, 100002
The first 3 integer numbers read by the Java program are:
-1912406016, 749473597, -1568276224
The file was written with another C program. The values read by the C program listed here are correct.
I supect that I have a byte swapping problem. Do you know how I can solve this problem? I really can't change the format of the input file.
Thanks for your help,
Miguel
#include <stdio.h>
void main(int argc, char *argv[]) {
long buff[600];
FILE *fp = fopen("c:\\inetpub\\wwwroot\\super\\wvdot3.awg","r");
fread(buff,sizeof(int),512,fp);
int sz = sizeof(int);
import java.io.*;
public class test {
public static void main(String[] args) {
try {
DataInputStream din = new DataInputStream(new BufferedInputStream
(new FileInputStream("c:\\inetpub\\wwwroot\\super\\wvdot3.awg")));
int nrecs = din.readInt();
int version = din.readInt();
int routetype = din.readInt();
System.out.println("read: " + nrecs + " " + version + " " + routetype);
} catch (IOException exc) {
System.out.println("IOException: " + exc);

Yeah, the problem is byte ordering. I had to deal with the same thing, since Java is always big endian, and intel and DEC Alpha are both little endian. I assume you're original program runs on Intel.
I ended up creating a new LittleEndianDataInputStream. Unfortunately, you can't just extend DataInputStream since all it's methods are final. You can however copy the source, rename it, and then change all the multi-byte numeric methods to reverse the byte ordering.
For example, here is the code for the original and little endian version of readInt()
<code>
// DataInputStream
public final int readInt() throws IOException {
InputStream in = this.in;
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
if ((ch1 | ch2 | ch3 | ch4) < 0)
throw new EOFException();
return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
// LittleEndianDataInputStream
public int readInt() throws IOException
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
if ((ch1 | ch2 | ch3 | ch4) < 0)
throw new EOFException();
// here's the change
return ( (ch4 << 24) + (ch3 << 16) + (ch2 << 8) + (ch1 << 0) );
</code>

"Read From Binary File" function Help ambiguity

I must be getting tired, but for some reason a doubt crept in my mind as I was designing a new piece of code this morning:
"is the "Read From Binary File" using the last file position or is it starting from the beginning of the file?"
"That's a stupid question", I told myself.
"I used this function a million times and have always assumed it is reusing the last file position. Moreover, there is no file offset input to that function, so WTH am I afraid of?"
So, for kicks, I fired up the Help window and read the following description (*):
Reads binary data from a file and returns it in data. How the data is read depends on the format of the specified file. This function does not work for files inside an LLB.
(*) BTW, has anybody ever complained that you can't select and copy anything from the floating Help Window?
Not much there. I particularly admire the phrasing of the second sentence... What about: "This function can do a lot of things, but it would much to complex to describe this is extensive details, so if you are asking, you probably can't afford using it"?
Anyhow, I clicked on the "Detailed Help" and got this (among other things):
Use the Set File Position function if you need to perform random access.
WHAT? I am pretty darn sure I DO NOT USE the Set File Position when I read a file in successive and contiguous chunks. I just pass the file refnum into a shift register and back to the function and that's it.
Now, the description of the "Refnum Out" ouput says: If file is a refnum or if you wire refnum out to another function, LabVIEW assumes that the file is still in use until you close it. Translated in plain English, is that supposed to mean that if the file is not closed it is open, or is that implying that it contains more info that just "the file is open and can be found here"?
I started searching around and finally ended up with the entry for "refnums, file I/O". Down the bottom of the (long) article, I found this under the heading "References to Objects or Applications" (but nothing specific to files, BTW):
...LabVIEW creates a refnum associated with that file, device, or network connection...
[...] LabVIEW remembers information associated with each refnum, such as the current location for reading from or writing to the object and the degree of user access, so you can perform concurrent but independent operations on a single object. If a VI opens an object multiple times, each open operation returns a different refnum. LabVIEW automatically closes refnums for you when a VI finishes running, but it is a good programming practice to close refnums as soon as you are finished with them to most efficiently use memory and other resources.
So it seems that my recollection was correct. I do not know what the "degree of user access" for a file is, but that's not the topic of today's post.
So, my point is: the Help File for this function is incomplete or ambiguous at best. Please correct it. And provide a link to the "refnum, file I/O" Help entry in its detailed Help. It would H E L P...
Thanks for reading.

Reading in succesive chunks is *NOT* random access. An open file always has
a current position, which is updated with each read or write operation.
You only need to set the file position if you want to start elsewhere.
LabVIEW Champion . Do more with less code and in less time .

Any info on CRC, checksum, or other file integity VIs for large binary files?

Working on send rather large binary files (U16 stream to file) via internet. Would like to check for file integity via CRC or comparable checksum. Would appreciate any comments/suggestions

Hi Brian,
You said;
"Would appreciate any comments/suggestions".
You did not mention what transport mechanism you plan on using.
As I understand ALL of the standard mechanism use CRC of some form to ensure the validity of the packet BEFORE it is ever passed up the OSI 7-Layer model.
TCP/IP based protocols will see to it that all of the segments of a transfer are completed and in order.
UDP on the other hand is a broadcast type protocol and does not ensure any packets are recieved.
So,
At the very worst you should be able to handle your "sanity checks" by simply using a sequence value that is included in your out-going message. The reciever should just have to check if the current seq value is equal to the previous +1.
I co-developed an app that ut
ilized this technique to transfer status messages from a RT platform to a Windows machine. The status messages in this app where concidered FYI, so the sequence counter served as a way of determining if anything was missed.
I am insterested in others thoughts on this subject.
Ben
Ben Rayner
I am currently active on.. MainStream Preppers
Rayner's Ridge is under construction

Reading a binary file using the type cast function is not working

Hello
I am trying to read a binary file. Using the Read from binary file and type cast functions I can actually get the information from the file.
I set an indicator in the “Normal” option in Properties. I can read the header and footer but not the body of the file; For the body of the file all I can see is characters of the type
“C¾ Ü Qþ X@ “. One of the other options, Password, seems to provide an output of just the characters that were originally written, but then again, I can not read the characters because it is only asterisks and I can not copy them onto a word processor either.
I tried setting the indicator in Hexadecimal mode, then creating a local variable that acts as a control and use the type cast function again to see if could convert the Hexadecimal string to readable ASCII characters; unfortunately, this did not work either.
I don’t know what else I should try or if there is something that I may not be doing right and for that reason I am not getting the desired results. I hope someone has an idea about this issue.
Regards,
Roberto

Thank you for your reply,
Well, actually I don’t have any information about this file. I will try to obtain information about it though.
This is a file that contains velocity information; there are 65 channels that form a velocity profile in a pipe. Also, there are 4096 velocity profiles. I don’t know what is the format of these values.
The software that creates the file provides a text file. Using this text file the velocity information can be processed and then plotted.
I want to avoid all these intermediate steps and read, process and plot everything using only LabView. BTW, I am using LabView 8.2
Roberto

Read large binary file

Similar Messages

Maybe you are looking for