Large binary file reading

Hi,
I'm currently using a java.io.BufferedInputStream to read a large binary file.
I recently discovered there is a chunk of data that shows up near the end of each file. (these files are binary and are XX to XXX mb in size)
Loading it all to memory first would kill my performance so I'd like to be able to come up with an alternate method.
Does Java or the class above offer a way I can
1) get the length of the file
2) seek to a point say 2000 bytes from the end so I can start reading the binary data?
Ideally I'd like to do a backwards read as that would be quickest. Is there a way to change the operation so that a 'read' would be reading backwards (from end to beginning)?
For me speed is the #1 thing i have to worry about. So to be able to seek forward several hundred thousand bytes at a time would help tremendously.

how does the 'skip' method work? Probably by using OS specific calls to read to a point in the file.
maybe I could
'skip' length - 20k from the start or something like
that.
Yep.

Similar Messages

Read large binary file

How would you read a large binary file into memory please? I had a thought about creating a Byte Array on the fly, however you cannot createa byte array with a long so what happens when you reach the maximum value an integer can store?

a) You can map the file, instead of reading it physically.
b) Let's suppose that you are running Sun JVM in Windows 2000/XP/2003 and you have 4-GB of RAM in your machine (memory is so cheap nowadays...)
- Windows can not use the full 4GB (it reserves 2GB for itself, and if you buy a special server version, it reserves only 1GB for itself.)
- Java can't use more than 1.6 GB in Windows due to some arcane reasons. (Someone posted the Bug Database explanation in this forum.)
So you have an upper limit of 1.6 GB.
1.6GB is a value smaller than the size of an int, so you could have an array as big (using the class java.nio.ByteBuffer and the like). Try and see.

Convert Large Binary File to Large ASCI File

Hello, I need some suggestions on how to convert a large binary file > 200MB to an ASCI File. I have a program that streams data to disk in binary format and now I would like to either add to my application or create a new app if necessary. Here is what I want to do:
Open the Binary File
Read a portion of the File into an array
Convert the array to ASCI
Save the array to a file
Go back and read more binary data
Convert the array to ASCI
Append the array to the ASCI file
Keep converting until the end of the binary file.
I should say that the binary data is 32-bits and I do need to parse the data; bits 0-11, bits 12-23, and bits 31-28, but I can figure that out later. The problem I see is that the file will be very large, perhaps even greater than 1GB and I don't have a clue how to read a portion of the file and come back and read another portion and then stop at the end of the file. I hope to save the data in a spreadsheet. If anyone has some experience with a similiar situation I'd appreciate any input or example code.
Thanks,
joe

sle,
In the future, please create a new thread for unrelated questions. To answer your question, you can use "Split Number" from the Data Manipulation palette.
Message Edited by jasonhill on 03-14-2006 03:46 PM
Attachments:
split number.PNG ‏2 KB

Binary File Read

Hello everyone,
I am not that comfortable with binary file reading. I am trying to learn.
I have a vi which logs a sine wave in a binary file.
The Read vi which I have written is not giving the expected data.
Thanks in advance
cooldude
Attachments:
Log Data.vi ‏9 KB
Read Data.vi ‏10 KB

Attachments:
Log.JPG ‏20 KB
Read.JPG ‏13 KB

Any info on CRC, checksum, or other file integity VIs for large binary files?

Working on send rather large binary files (U16 stream to file) via internet. Would like to check for file integity via CRC or comparable checksum. Would appreciate any comments/suggestions

Hi Brian,
You said;
"Would appreciate any comments/suggestions".
You did not mention what transport mechanism you plan on using.
As I understand ALL of the standard mechanism use CRC of some form to ensure the validity of the packet BEFORE it is ever passed up the OSI 7-Layer model.
TCP/IP based protocols will see to it that all of the segments of a transfer are completed and in order.
UDP on the other hand is a broadcast type protocol and does not ensure any packets are recieved.
So,
At the very worst you should be able to handle your "sanity checks" by simply using a sequence value that is included in your out-going message. The reciever should just have to check if the current seq value is equal to the previous +1.
I co-developed an app that ut
ilized this technique to transfer status messages from a RT platform to a Windows machine. The status messages in this app where concidered FYI, so the sequence counter served as a way of determining if anything was missed.
I am insterested in others thoughts on this subject.
Ben
Ben Rayner
I am currently active on.. MainStream Preppers
Rayner's Ridge is under construction

Reading large binary files into an array for parsing

I have a large binary log file, consisting of binary data separted by header flags scattered nonuniformly thorughout the data. The file size is about 50M Byte. When I read the file into an array, I get the Labview Memory full error. The design of this is to read the file in and then parse it fro the flags to determine where to separate the data blocks in the byte stream.
There are a few examples that I have read on this site but none seem to give a straight answer for such a simple matter. Does anyone have an example of how I should approach this?

I agree with Gerd. If you are working with binaries, why not use U8 instead of doubles.
If the file is indeed 50MB, then the array should be expecting 52428800 elements, not 50000000. So if you read the file in a loop and populate an element at a time, you could run out of memory fast because any additional element insertion above 50000000 may require additional memory allocation of the size above 50000000 (potentially for each iteration). This is just speculation since I don't see the portion of your code that populates the array.
Question: Why do you need an array? What do you do with the data after you read it? I agree with Altenbach, 50MB is not that big, so working with a file of such a size should not be a problem.

UTL_FILE write_error when writing large binary files to unix os

I am trying to write large files to a folder in unix from a table containing a BLOB object. The procedure below is called by another procedure I have written to do this. It works in windows environment fine with files up to 360MB. When I run this exact same procedure in UNIX I get an initialization error. When I change the WB in the fopen call to W it works. I can store all the files I want up to 130MB in size. The next size larger file I have is 240MB and it fails after writing the first 1KB passing the utl_file.write_error message. If someone can help me to diagnose the problem, I would really appreciate it. i have been trying everything I can think of to get this to work.
Specifics are, the windows version is 10GR2, on unix we are running on Sun Solaris 9 using 9iR2
PROCEDURE writebin(pi_file_name IN VARCHAR2, pi_file_url IN VARCHAR2, pi_file_data IN BLOB)
IS
v_file_ref utl_file.file_type;
v_lob_size NUMBER;
v_raw_max_size constant NUMBER := 32767;
v_buffer raw(32767);
v_buffer_offset NUMBER := 1;
-- Position in stream
v_buffer_length NUMBER;
BEGIN
-- WB used in windows environment. W used in unix
v_lob_size := dbms_lob.getlength(pi_file_data);
v_file_ref := utl_file.fopen(pi_file_url, pi_file_name, 'WB', v_raw_max_size);
v_buffer_length := v_raw_max_size;
WHILE v_buffer_offset < v_lob_size
LOOP
IF v_buffer_offset + v_raw_max_size > v_lob_size THEN
v_buffer_length := v_lob_size -v_buffer_offset;
END IF;
dbms_lob.READ(pi_file_data, v_buffer_length, v_buffer_offset, v_buffer);
utl_file.put_raw(v_file_ref, v_buffer, TRUE);
v_buffer_offset := v_buffer_offset + v_buffer_length;
END LOOP;
utl_file.fclose(v_file_ref);
END writebin;
Message was edited by:
user599879

check if this cample code helps -
CREATE OR REPLACE PROCEDURE prc_unload_blob_to_file IS
vlocation      VARCHAR2(16) := ‘LOB_OUTPUT’;
vopen_mode     VARCHAR2(16) := ‘w’;
bimax_linesize NUMBER := 32767;
v_my_vr        RAW(32767);
v_start_pos    NUMBER := 1;
v_output       utl_file.file_type;
BEGIN
FOR cur_lob IN (SELECT vmime_type,
blob_resim,
vresim,
dbms_lob.getlength(blob_resim) len
FROM tcihaz_resim a
WHERE rownum < 3 -- for test purposes
ORDER BY a.nresim_id) LOOP
v_output := utl_file.fopen(vlocation,
cur_lob.vresim,
vopen_mode,
bimax_linesize);
dbms_output.put_line(’Column length: ‘ || to_char(cur_lob.len) || ‘ for file: ‘ ||
cur_lob.vresim);
v_start_pos := 1;
IF cur_lob.len < bimax_linesize THEN
dbms_lob.READ(cur_lob.blob_resim,
cur_lob.len,
v_start_pos,
v_my_vr);
utl_file.put_raw(v_output,
v_my_vr,
autoflush => TRUE);
dbms_output.put_line(’Finished Reading and Flushing ‘ || to_char(cur_lob.len) ||
‘ Bytes’ || ‘ for file: ‘ || cur_lob.vresim);
ELSE
dbms_lob.READ(cur_lob.blob_resim,
bimax_linesize,
v_start_pos,
v_my_vr);
utl_file.put_raw(v_output,
v_my_vr,
autoflush => TRUE);
dbms_output.put_line(’Finished Reading and Flushing ‘ || to_char(cur_lob.len) ||
‘ Bytes’ || ‘ for file: ‘ || cur_lob.vresim);
END IF;
v_start_pos := v_start_pos + bimax_linesize;
WHILE (v_start_pos < bimax_linesize) LOOP
-- loop till entire data is fetched
dbms_lob.READ(cur_lob.blob_resim,
bimax_linesize,
v_start_pos,
v_my_vr);
utl_file.put_raw(v_output,
v_my_vr,
autoflush => TRUE);
dbms_output.put_line(’Finished Reading and Flushing ‘ ||
to_char(bimax_linesize + v_start_pos - 1) || ‘ Bytes’ ||
‘ for file: ‘ || cur_lob.vresim);
v_start_pos := v_start_pos + bimax_linesize;
END LOOP;
utl_file.fclose(v_output);
dbms_output.put_line(’Finished successfully and file closed’);
END LOOP;
END prc_unload_blob_to_file;
set serveroutput on
set timing on
create or replace directory LOB_OUTPUT as ‘/export/home/oracle/tutema/’;
GRANT ALL ON DIRECTORY LOB_OUTPUT TO PUBLIC;
exec prc_unload_blob_to_file ;
Column length: 3330 for file: no_image_found.gif
Finished Reading and Flushing 3330 Bytes for file: no_image_found.gif
Finished successfully and file closed
Column length: 10223 for file: OT311.gif
Finished Reading and Flushing 10223 Bytes for file: OT311.gif
Finished successfully and file closed
PL/SQL procedure successfully completedWith 9iR2 PLSQL can write binary files using UTL_FILE put_raw function, prior to Oracle9iR2 you will need to create an external procedure with Java, C, VB or some 3gl language.
Some references -
http://asktom.oracle.com/pls/ask/f?p=4950:8:::::F4950_P8_DISPLAYID:6379798216275
Oracle® Database PL/SQL Packages and Types Reference 10g Release 2 (10.2)
UTL_FILE - http://download-uk.oracle.com/docs/cd/B19306_01/appdev.102/b14258/u_file.htm#sthref14095
http://psoug.org/reference/dbms_lob.html
Metalink Note:70110.1, Subject: WRITING BLOB/CLOB/BFILE CONTENTS TO A FILE USING EXTERNAL PROCEDURES

Somewhat large binary file has trouble being opened

Hi,
I'm a new Labview user (Labview 8.0), and I'm trying to convert a
binary file into a wave file and do a lot of other things with that
binary file (like analyzing frequency via a spectrum graph). My file
works fine for files under 150MB, but once the file size is larger than
that, the computer slows way way down and I cannot collect the entire
data set. I already have 1GB of RAM (512x2) and I don't want to pay
more money if the issue can be fixed within Labview.
Is there a way to split up the binary file into little pieces when
being read, or is there another way to go about doing this? I'm using
the File Open dialog to read the file.
If anyone could provide a step-by-step solution to this, I'd be very
grateful. Thanks everyone.
Jennings

Thanks for the reply,
That would kind of work, but ultimately, I need a graph that
contains the entire file. In other words, I need to convert the binary
file to a wave file that I can view as a graph. If I only open one
half, then I would only see the graph from that first half, which is
not useful to the project. I need to view the wave file in its
entirety as a waveform graph, so is there any way to add the data from
the second half of the file to the first half?
I guess it comes down to the memory buffer size, right? Labview tries
to load the entire file onto the RAM, but I need it to load a portion
onto the RAM, save that portion on the hard drive, load a second
portion onto the RAM, and add that portion to the first portion on the
hard drive, and so forth.
Is there any function in labview that can do this? Or can I only
manipulate data using the RAM, and not the hard drive?
Thanks for any and all help!
Jennings
JLS wrote:
> Hello,
>  
> From your previous post, it sounds like you're able to open and read from the file - can you try reading half (the get file size function will be helpful) and immediately writing that half to another file?  If this works, you can try the same thing with the second half using the set file position function.  You could repeat this process until you had files which were of manageable size for your application/processing needs.
>  
> I hope this helps!
>  
> Best Regards,
>  
> JLS

Binary file read byte unmatch

I tried to read one line of binary file as the top of the screenshot.
The result is as the left side. It shows the first part with text are correct. The ending part is not match with the raw one.
I also tried read by byte, but still the same.
Any suggestion, thanks.
Solved!
Go to Solution.
Attachments:
shot.JPG ‏48 KB

What program created this binary file?
If you can figure out how the file was created, then you can figure out how to read it.
We can't help because all you've shown are some bytes and say that some are right and some are wrong. What is the "raw one" you are referring to? What do you expect it to look like?
Message Edited by Ravens Fan on 01-26-2010 11:15 PM

Binary file read with bit offset..not byte offset.

hi,
i want to read a binary file with some offset ..not in bytes... offset in bits..my data is a single bit data stream size of 1GB ,after skipping particular bit offset i want to read the data in byte order..
Indrajit
| [email protected] | [email protected] .

hi , Gerd W
i am not able to follow your answer.... here with i attached a file to describe my query more clearly. Actually two channels of 1 bit data stream stored in two different files ,and the file size will be more than 10 GB.
i want to delay the channel B by 3 bits and i want to perform bitwise operation between channel A and shifted version of channel B..till the end of the data..
now i am doing this without applying any delay by choosing a small block by block which corresponds to 1 sec of data .
Indrajit
| [email protected] | [email protected] .
Attachments:
bitwsie.png ‏16 KB
bitwsie.jpg ‏52 KB

Binary file readed from Labview and Borland C++ v.5

Hi Everybody!
My problem is reading binary file from Labview 7.0 and Borland C++ version 5 is not the same.
Only one difference is "space". Reading from Labview I got "00" (decimal) and reading from Borland C++ I got "32".
I also try to read this binary file with Matlab 6 and I also got "00" as well as those from Labview.
I really struck with this problem.. Anybody has an idea about this problem.
Thanks a lot for anybody help!
Nok

Hi,
These threads regarding reading of binary files was recently discussed in this forum
You might find some answers for your query
http://forums.ni.com/ni/board/message?board.id=170&message.id=99016&requireLogin=False
and
http://forums.ni.com/ni/board/message?board.id=170&message.id=153565&requireLogin=False
Regards
Dev

Binary file reading

Dear All,
I have Binary file. I want to read Bytes based on Patterns and each channel value (Byte values -- > Float conversion) and storing it into Index array.
Please check the attachment for pattren format.
Here, the biggest problem is we will get only pattern style, Based on pattern the data is already stored in Binary file just we need to read it means we need to set the Timestamp based on this.
Please give me some idea to read this kind of pattern files.
BR,
Guru
Munna
Attachments:
Data Reading.xlsx ‏15 KB
Data Reading.xlsx ‏15 KB

Hi Munna,
that pattern is quite hard to understand. It seems there is some information missing…
In your pattern table the data size for e.g. ch1 is given as "1 byte", but in your time schedule you use two bytes for ch1. How are those "ch start position bytes" to be identified?
Byte values -- > Float conversion
How do you want to convert those bytes to Float (SGL or DBL)?
storing it into Index array
What is an "index array"?
It seems you have to create a table from your pattern which identifies the bytes in the file. That seems to be the trickiest part as in your pattern#2 example you already have made a mistake (for ch4 there is no marking at timestamp 70ms)…
Best regards,
GerdW
CLAD, using 2009SP1 + LV2011SP1 + LV2014SP1 on WinXP+Win7+cRIO
Kudos are welcome

Binary File Reader to a Chart

Hello All,
Newbie having some trouble with reading a chart from a binary file. My problem may be with trying to use a binary file, I am not sure. The x axis time is not the same length as the actual test time. I have acquired a rms voltage from a VCR RF envelope. I'm using a 6009 DAQ with LV 7.1 on Win2K. I have attached the recording vi and the reading vi and a sample file. I have tried different things including the examples that came with lab view. I need to write a small sized file because this test will last for 4 hours. I have another application that will run for 5 days. I appreciate any help or ideas.
Thanks,
bh3560
Attachments:
Chart write_Read.zip ‏140 KB

Hi,
The program you have attached acquires, saves and reads data in ASCII ( floating numbers) and not binary.
To get an idea on how to write and read data in binary format, look at attached examples( taken from examples shipped with labview)
Hope this helps
Regards
Dev
Attachments:
Cont Acq&Graph Voltage-To File(Binary).vi ‏88 KB
Graph Acquired Binary Data.llb ‏72 KB

Binary file reading confusion

Hi there,
I'm trying to read a partially binary file with the following structure: The first three lines of the file are regular strings (one can see them correctly when opening the file in a text editor), the rest of the file's contents are integers in their binary form (one cannot see them correctly when opening the file in a text editor).
How can I read such a file?
Thanks in advance!
+P.S.: If you're interested, I'm trying to read a .PPM file (look here to find out about the format).+

pfrc wrote:
Hi there,
I'm trying to read a partially binary file with the following structure: The first three lines of the file are regular strings (one can see them correctly when opening the file in a text editor), the rest of the file's contents are integers in their binary form (one cannot see them correctly when opening the file in a text editor).
How can I read such a file?
Thanks in advance!
+P.S.: If you're interested, I'm trying to read a .PPM file (look here to find out about the format).+Use a BufferedInputStream wrapped around a FileInputStream. Read and bytes and until you've hit two end-of-lines, then put those bytes into a String constructor. Note that you'll have to know which line end convention is used, or write your code so as to allow for any of the common ones. After that, continue reading bytes and processing however is appropriate for you.

Blob for binary file, read/write problems

Hi,
I am relatively new to this type of development so apologies if this question is a bit basic.
I am trying to write a binary document (.doc) to a blob and read it back again, constructing the original word file. I have the following code for reading and writing the file:
private void save_addagreement_Click(object sender, EventArgs e)
// Save the agreement to the database
int test_setting = 0;
// create an OracleConnection object to connect to the
// database and open the connection
string constr;
if (test_setting == 0)
constr = "User Id=royalty;Password=royalty;data source=xe";
else
constr = "User ID=lob_user;Password=lob_password;data source=xe";
OracleConnection myOracleConnection = new OracleConnection(constr);
myOracleConnection.Open();
// create an OracleCommand object to hold a SQL statement
OracleCommand myOracleCommand = myOracleConnection.CreateCommand();
myOracleCommand.CommandText = "insert into blob_content(id, blob_column) values 2, empty_blob())";
OracleDataReader myOracleDataReader = myOracleCommand.ExecuteReader();
// step 2: read the row
OracleTransaction myOracleTransaction = myOracleConnection.BeginTransaction();
myOracleCommand.CommandText =
"SELECT id, blob_column FROM blob_content WHERE id = 2";
myOracleDataReader = myOracleCommand.ExecuteReader();
myOracleDataReader.Read();
Console.WriteLine("myOracleDataReadre[\"id\"] = " + myOracleDataReader["id"]);
OracleBlob myOracleBlob = myOracleDataReader.GetOracleBlobForUpdate(1);
Console.WriteLine("OracleBlob = " + myOracleBlob.Length);
myOracleBlob.Erase();
FileStream fs = new FileStream(agreement_filename.Text, FileMode.Open, FileAccess.Read);
Console.WriteLine("Opened " + agreement_filename.Text + " for reading");
int numBytesRead;
byte[] byteArray = new byte[fs.Length];
numBytesRead = fs.Read(byteArray, 0, (Int32)fs.Length);
Console.WriteLine(numBytesRead + " read from file");
myOracleBlob.Write(byteArray, 0, byteArray.Length);
Console.WriteLine(byteArray.Length + " written to blob object");
Console.WriteLine("Blob Length = " + myOracleBlob.Length);
fs.Close();
myOracleDataReader.Close();
myOracleConnection.Close();
This gives the following console output:
myOracleDataReadre["id"] = 2
OracleBlob = 0
Opened D:\sample_files\oly_in.doc for reading
56832 read from file
56832 written to blob object
Blob Length = 56832
My write to file code is:
private void save_agreement_to_disk_Click(object sender, EventArgs e)
string filename;
SaveFileDialog savedoc = new SaveFileDialog();
if (savedoc.ShowDialog() == DialogResult.OK)
filename = savedoc.FileName;
// create an OracleConnection object to connect to the
// database and open the connection
OracleConnection myOracleConnection = new OracleConnection("User ID=royalty;Password=royalty");
myOracleConnection.Open();
// create an OracleCommand object to hold a SQL statement
OracleCommand myOracleCommand = myOracleConnection.CreateCommand();
myOracleCommand.CommandText =
"SELECT id, blob_column " +
"FROM blob_content " +
"WHERE id = 2";
OracleDataReader myOracleDataReader = myOracleCommand.ExecuteReader();
myOracleDataReader.Read();
Console.WriteLine("myOracleDataReader[id] = " + myOracleDataReader["id"]);
//Step 2: Get the LOB locator
OracleBlob myOracleBlob = myOracleDataReader.GetOracleBlobForUpdate(1);
Console.WriteLine("Blob size = " + myOracleBlob.Length);
//Step 3: get the BLOB data using the read() method
byte[] byteArray = new byte[500];
int numBytesRead;
int totalBytes = 0;
FileStream fs = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.Write);
while ((numBytesRead = myOracleBlob.Read(byteArray, 0, 500)) > 0)
totalBytes += numBytesRead;
fs.Write(byteArray, 0, byteArray.Length);
Console.WriteLine("numBytes = " + numBytesRead + " totalBytes = " + totalBytes);
Console.WriteLine((int)fs.Length + " bytes written to file");
fs.Close();
myOracleDataReader.Close();
myOracleConnection.Close();
This gives the following console output:
myOracleDataReader[id] = 2
Blob size = 0
0 bytes written to file
If I manually add the blob file using the following:
DECLARE
my_blob BLOB;
BEGIN
-- load the BLOB
my_bfile := BFILENAME('SAMPLE_FILES_DIR', 'binaryContent.doc');
SELECT blob_column
INTO my_blob
FROM blob_content
WHERE id = 1 FOR UPDATE;
DBMS_LOB.FILEOPEN(my_bfile, dbms_lob.file_readonly);
DBMS_LOB.LOADFROMFILE(my_blob, my_bfile, DBMS_LOB.GETLENGTH(my_bfile), 1, 1);
DBMS_LOB.FILECLOSEALL();
COMMIT;
END;
COMMIT;
The write to file works perfectly. This tells me that there must be something wrong with my code that is writing the blob to the database. I tried where possible to following the Oracle article using large objects in .NET but that (along with most things on the internet) focus on uploading text files.
Thanks in advance.
Chris.

myOracleCommand.CommandText = "insert into blob_content(id, blob_column) values 2, empty_blob())";
OracleDataReader myOracleDataReader = myOracleCommand.ExecuteReader();
This looks wrong, you shouldn't be using ExecuteReader unless you expect to get a result back. Try using ExecuteNonQuery to do the insert.

Large binary file reading

Similar Messages

Maybe you are looking for