Searching binary data file

Hi I have a data file which acts as a database. I need to search this file for records. I am trying to implement this in the most efficient way I have the following extremes
1)To search using RandomAccessFile each record directly from the file
2)Load the entire file into memory. By encapsulating each record in a lava object.
1st option would save me a lot of memory but time consuming . The second option would consume a lot of memory and might be unsuitable for a large amount of records.
I have chosen a middle path but I am not sure if it will work properly.
My current proposal is to use a scaled down Object to represent a record
Integer values which are mostly 8 bit and 16 bit value will be stored an a array of integers where the necessary values are stored and extracted using bitwise operations.
My problem is characters I was thinking of storing the hash code of the Strings but the search protocol requires that you should be able to search even if a part of the String is given.
Regarding this problem I looked up the java 1.5 API doc for String.hashCode. Where I determined the hash code of a substring is smaller than of the main String. My question is? Is it possible to use this as a search condition to initially eliminate certain records and then read the character sequence from the file to confirm my match.
I hope you can understand what I have typed above and any help would be appreciated.

Why not turn this thing into a database and actually use database drivers to query it, and let the database worry about those kinds of things rather than basically reinvent them yourself?

Similar Messages

To attach/upload binary data file(to vendor) from EP(sends from) to abap

Hi All,
I have requirement Enterprise portal(JAVA) will send the file data in binary format to abap.we have to attach the binary data(file) to content server.
that is for vendor/custmer(XK02/XD02) will have attachments in the sap screen.
similarly the above screens are provided using enterprise portal (using java) also attachment feature also needs to provide.
so whenever user imports file form EP screen, they will convert the file into binary format and sends to abap through proxy(custom proxy) abap should attach the binary file to vendor/customer.
I have checked so_object_upload,so_object_insert, binary relation create -- these FMs are helped using dialog box or providing the path of the file. but couldnt use when the file is in binary format and if it sent form proxy interface.
could anyone help me in resolving the requirement to upload binary data.

I use "call function 'ARCHIVOBJECT_CREATE_TABLE'" to process a binary table. However, in my situation, I'm storing in Documentum. Once that FM completed, I follow up with call function 'ARCHIV_CONNECTION_INSERT' to store the reference to the document and the archived object in SAP tables. In your case, you need something that processes the binary table, or you need to utilize one of the "SCMS* " function modules to put your binary table into something that you can use with the other functions.
The upload from desktop can be done using BIN filetype for GUI_UPLOAD class or function module, or from apps server, you can read your file into an xstring with open dataset....in binary mode....read dataset into your xstring, close dataset.

Download Binary Data File over Serial Port

I need help setting up a .vi that I can use to download a binary data file from a dos-based system. I will send the system a command, say ascii "DOWNLOAD", and then the remote system will send me a binary data file that will be approx. 600MB. I won't be doing any display/manipulation using labview, I will just need to have the option available in LV to recieve the file.
Any examples would help a programmer that is new to LV and is getting a little desperate.

A VISA Read doesn't care whether the data it's getting is binary or ASCII. Where you might have problems is detecting that file transfer is complete. If you have enable termination character set to True in the VISA Configure Serial Port, the read will stop as soon as the termination character is detected. The default termination character is 0xA or a linefeed. You probably don't want to use this. If the file is terminated with a character or string of characters that you now will be unique, you can use this. If there is no unique termination character at the end of the file, the next best thing is to know the file size exactly and set the byte count of VISA Read to that number. Based on file size and baud rate, you would also need to adjust the VISA timeout value. Another way to do this is to transfer the data in pieces and write to a file as soon as the data is received. The reading of data would then be terminated when there are no more bytes available at the serial port. I've attached a LabVIEW 7.0 program that should give an idea of the last approach.
Attachments:
File Transfer.vi ‏57 KB

B&K Portable Signal Analyzer Binary Data Files

I am currently using a B&K 2144/7651 Portable Signal Analyzer. I'd like to start a project to display the binary data files created by the analyzer in Labview. I have the technical documentation that specifies the formatting of the binary files and they have example code written for Pascal but I have no idea how to translate that to Labview. If anyone has experience with these data files or could give me a hint on how to parse the Labile files I would appreciate it.
I can post the technical doumentation but I have to scan it first.
Thanks
Eric
Solved!
Go to Solution.

Hi Arvin,
Thanks much for the Kudos!
Here's a first "possbly useful" LLB. The main thing is the data arrays are being populated!
I hope you'll forgive me for not unpacking the entire "Setup" section - - I wasn't sure you needed it all(?) As is, enough Setup is unpacked to interpret the Data correctly plus a bunch more.
Please give it a test-drive and let me know if you need any unrepresented Setup values (or all of them.)
Cheers!
"Inside every large program is a small program struggling to get out." (attributed to Tony Hoare)
Attachments:
7651.FileReader.llb ‏1475 KB

How can I read a binary file stream with many data type, as with AcqKnowledge physio binary data file?

I would like to read in and write physiological data files which were saved by BioPac�s AcqKnowledge 3.8.1 software, in conjunction with their MP150 acquisition system. To start with, I�d like to write a converter from different physiodata file format into the AcqKnowledge binary file format for version 3.5 � 3.7 (including 3.7.3). It will allow us to read different file format into an analysis package which can only read in file written by AcqKnowledge version 3.5 � 3.7 (including 3.7.3).
I attempted to write a reader following the Application Note AS156 entitled �AcqKnowledge File Format for PC with Windows� (see http://biopac.com/AppNotes/ app156Fi
leFormat/FileFormat.htm ). Note the link for the Mac File format is very instructive too - it is presented in a different style and might make sense to some people with C library like look (http://biopac.com/AppNotes/ app155macffmt/macff.htm).
I guess the problem I had was that I could not manage to read all the different byte data stream with File.vi. This is easy in C but I did not get very far in LabView 7.0. Also, I was a little unsure which LabView data types correspond to int, char , short, long, double, byte, RGB and Rect. And, since it is for PC I am also assuming the data to be written as �little endian� integer, and thus I also used byte swap vi.
Two samples *.acq binary files are attach to this post to the list. Demo.acq is for version 3.7-3.7.2, while SCR_EKGtest1b.acq was recorded and saved with AcqKnowledge 3.8.1, which version number is 41.
I would be grateful if you someone could explain how to handle such binary file stream with LabView and send an example to i
llustrate it.
Many thanks in advance for your help.
Donat-Pierre
Attachments:
Demo.acq ‏248 KB
SCR_EKG_test1b.acq ‏97 KB

The reading of double is also straight forward : just use a dble float wired to the type cast node, after inverting the string (indian conversion).
See the attached example.
The measure of skin thickness is based on OCT (optical coherent tomography = interferometry) : an optical fiber system send and received light emitted to/back from the skin at a few centimeter distance. A profile of skin structure is then computed from the optical signal.
CC
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left...
Attachments:
Read_AK_time_info.vi.zip ‏9 KB

How to covert a flat txt file to a binary dat file

hi, am working on a project. i have to insert all the data into a dat file in a binary format. my input file is a flat text file. i have written the code for this job. the code executes ok but it doesnt' write into the dat file. please help me out.
here is my method for binary conversion---- >
public static void BinaryOutFile(String fileName){
     try{
          File fileOut=new File(fileName);
          DataOutputStream out=new DataOutputStream(new BufferedOutputStream(
                                   new FileOutputStream(fileOut)));
          for(int i=0;i<array.size();i++)
               String con=(String)array.get(i);
     //System.out.println(""+con);
               if(!con.equals(""))
                    out.writeChars(con);
               out.writeChar(',');
     catch(IOException io){
          System.out.println("Error! Can not write to teh specified file.");
}

no i never flush or close the output stream.
i just want to dump the value of the database (the flat file) in a dat file.
here is my whole code---->
import java.io.*;
import java.util.*;
import java.util.StringTokenizer;
public class Flatgen{
private static ArrayList array= new ArrayList();
public static void main(String args[]){
     String line=null;
     String value=null,
     actualValue=null;
     StringTokenizer token1=null,
          token2=null;
     BufferedReader buf=null;
     try{
          File inFile=new File("input.txt");
     buf=new BufferedReader(new InputStreamReader(new FileInputStream(inFile)));
     catch(FileNotFoundException e){
          System.out.println("Cannot open the source file: ");
          System.exit(0);
     try{
          while((line=buf.readLine())!= null){
          token1=new StringTokenizer(line,",");
          int tokenizeCount=0;
          while(token1.hasMoreTokens()){
               value=token1.nextToken();
               token2=new StringTokenizer(value,":");
               for(int i=0;i<2;i++)
                    actualValue=token2.nextToken();
          if(tokenizeCount>0)
               Integer.parseInt(actualValue);
               array.add((Object)actualValue);
               tokenizeCount++;
     catch(IOException e){
          System.out.println("Cannot read the input file");
          System.exit(0);
     BinaryOutFile("dbfile.txt");
public static void BinaryOutFile(String fileName){
     try{
          File fileOut=new File(fileName);
          DataOutputStream out=new DataOutputStream(new BufferedOutputStream(
                                   new FileOutputStream(fileOut)));
          for(int i=0;i<array.size();i++)
               String con=(String)array.get(i);
     //System.out.println(""+con);
               if(!con.equals(""))
                    out.writeChars(con);
               out.writeChar(',');
     catch(IOException io){
          System.out.println("Error! Can not write to teh specified file.");
}

Binary Data file displays large gaps in the data. Will use of a queue help?

Ultimately I'd like to save 15 channels of data (1 channel at 20kHz, 6 at 2kHz, and 8 at 100 Hz) while displaying the waveforms on the front panel.
Currently, I am saving in binary format without any decimation of the data (therefore saving all channels at 20kHz). Decimating the waveforms can be done without affecting the display, and will definitely reduce the final size of the data files, but will slow down the processing. I've seen examples of using queues to limit data loss, but do not know if it will be advantageous in this case.
Any comments and suggestions will be greatly appreciated.
Attachments:
Cont_Acq_to_File_(binary)_NM-no_dec.llb ‏546 KB
Cont_Acq_to_File_(binary)_NM-dec ‏531 KB

So let's say that I'm going to use queues. Is it better to have an acquisition loop and then a 'everything else' loop (decimation and data recording) or is it possible (and better?) to use 3 loops? I'm assuming with each queue segment, the time spent reading and writing the data will increase, and therefore slow down my vi even more than it is doing now.
And another question: in the case of the tabbed case structure, my charts are only active when the corresponding tab is selected, correct? If I would then put the recording of data in each case, then the data would be recorded for only those signals which are displayed on that tab. I'm not completely confident that I understand how that structure works, or how I can make the data recording more efficient.
Thank you for all of your help, and for the compliment.

How do I import binary data files (with header info) into DIAdem?

We have lots of data acquired by nCode datagate. Files are binary data with header info. How do I import this into DIAdem? Typical (short) file attached. Also of interest, can I stream data from Sony PC208/SIR1000 recorders?
Attachments:
Propshft.dac ‏4 KB

I have successfully imported your file into DIAdem using the nCode file filter included with DIAdem. You might have to make that file filter known to DIAdem by following these steps (this assumes you have version 8 of DIAdem):
1. Go to the 'Window' menu in DIAdem and select 'Close all'
2. Go to the 'Settings' menu and select 'GPI-DLL Registration'
3. Click the 'Add' button in the window and select the 'GfSnCode' DLL in the 'DIAdem/AddInfo' directory
4. The program will prompt you to restart.
5. After you have restarted DIAdem, go to the DATA icon, select 'File - Open' from the DATA menu and in the dialog that appears, choose 'nSoft Data File Format' in the 'Files of type:' field.
6. Select your data file and load
That's it.
With regards to your Sony reco
rder data, I would need to get my hands on an actual data file to try it out. In general, DIAdem can import ANY binary file format by creating a header in the 'File - Import via header' section in DATA. If you have lots of data files from the SONY recorder, a DLL can be created for that specific file format with the free DLL toolkit included with DIAdem.
Leave me a message here if you have any additional questions.
Otmar
Otmar D. Foehner
Business Development Manager
DIAdem and Test Data Management
National Instruments
Austin, TX - USA
"For an optimist the glass is half full, for a pessimist it's half empty, and for an engineer is twice bigger than necessary."

Opening a binary data file with a local java application from a weblink...

I don't know if this is the right forum for my query, but I would like to be able to click on a data file from a download link on a webpage and be able to have the file "open" into the java application that processes the particular file (as opposed to selecting the "save to disk" option or having the browser display a bunch of funky characters). If the application is not already running, have an instance start up and process the data file and display the information. What steps would I need to take to get this to happen? The application is executed in a windows environment through executing a .bat file and not an executable .jar file. Any thoughts? TIA!

This is entirely a matter of configuring the Windows file associations so that files with that extension are started using your designated process. Depending on the particulars you may be able to establish the association from the context (right-click) menu, or you may need to use Folder Options rom the Control Panel.
In any case, this is a not a Java question, but Windows; if further help is needed you should go to a Windows help site.

How can I read a binary file stream with many data type, as with AcqKnowledge physio binary data file?

I would like to read in and write physiological data which was saved by Biopac�s AcqKnowledge 3.8.1 software, in conjunction with their MP150 acquisition system. To start with, I�d like to write a converter from different physiodata file format into the AcqKnowledge binary file format for version 3.5 � 3.7 (including 3.7.3). It will allow us to read different file format into an analysis package which can only read in file written by AcqKnowledge version 3.5 � 3.7 (including 3.7.3).
I attempted to write a reader following the Application Note AS156 entitled �AcqKnowledge File Format for PC with Windows� (see http://biopac.com/AppNotes/app156FileFormat/FileFormat.h
tm ). Note the link for the Mac File format is very instructive too - it is presented in a different style and might make sense to some people with C library like look (http://biopac.com/AppNotes/app155macffmt/macff.htm) .
I guess the problem I had was that I could not manage to read all the different byte data stream with File.vi. This is easy in C but I did not get very far in LabView 7.0. Also, because it is for PC I am assuming the data to be written as �little endian� integer, and thus I also used byte swap vi.
I would be grateful if you someone could explain how to handle such binary file stream with LabView and send an example to illustrate it.
Many thanks in advance for your help.
Donat-Pierre

One more step...
short are U16 integer
double are double precision float
bool seem to be 2 bytes (= U16)
char are string (variable length)
rgb are U16 integer, with high order byte = 0
rect should be 4 x U16 (top, left, bottom, right)
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left...

Binary data files

I am attempting to compile a code which I write/read files in binary format. With the intel fortran compiler, I use the option "-assume byterecl" and all open/read statements in my code specify RECL=8. It's not obvious to me by reading the manual how I should compile this code using Solaris. Any help would be greatly appreciated.

I assume you are using direct-access I/O. Oracle Solaris Studio Fortran assumes record lengths are in terms of bytes, so there is no need for an option to say so. If you are running using Intel or AMD CPUs, the files generated by OSS Fortran and files generated by Intel's Fortran are generally compatible as long as only intrinsic data types are read and written. Intel Fortran and OSS Fortran sometimes choose different layouts for derived types, which makes reading and writing derived type in binary form incompatible in general.
Are trying to transfer data between SPARC and Intel CPUs? If so, OSS Fortran includes features to make the process easier, but expect it to be difficult.
Robert Corbett

Need to read data files originally created in LabVIEW in C++

I have originally created binary files in LabVIEW. These files are made by flattening labview data to a string, then converting the string to a U8 byte array which is then saved to a binary data file. A separate file is used to keep track of the variables written to the data file. (FAT) a 2D-String array (first element = variable name, 2nd element = staring byte location in .dat file, 3rd element = byte length of variable) is flattened to U32 and then written to a binary file.
Opening the files in LabVIEW follows the inverse proceedure. The FAT file is read as a U32 binary data, converted to a byte array, converted to a string, and then unflattened from the string into a LabVIEW 2D-string data type. From this the starting location and length of the variable is extracted. These are used to read the data file, in U8, which is then converted bu a string and then unflattened to LabVIEW data according to the LabVIEW data type.
I now need to now open these files in a Linux environment using C++. I am at a loss at how to emulate all of this data-type conversion, flattening, and unflattening in C++. Does anyone have any ideas?

Faraclas wrote:
> Thanks rolfk, I will do the reading. But just to make sure you don't misunderstand
> me, everything works beautifully in LabVIEW.
I gathered that. But reading through the description gave me the
impression that a few conversions/typecasts may actually be superfluous
without any change to the actual data.
> The problem I am having is trying to open up these files in C++. This system has
> been around for a couple of years now, so I cannot change the data structure in
> the LabVIEW programs (well I could but I don't want to because it is really a very
> flexible system that works well).
Well, it really isn't that difficult, it just won't be a one hour job
for sure. A few guidelines:
1) LabVIEW flattened data contains the actual data as is for all fixed
size elements (scalars and clusters of scalars). Any variable sized data
(strings, arrays) is prepended with an int32 for each dimension
containing the number of elements for that dimension.
2) LabVIEW flattened data is always in Big Endian (Most significant Byte
first) format. This means for normal C(++) programs developed on an x86
system you will have to byteswap every single 16 bit and 32 bit integer.
3) LabVIEW Typecast really is similar to Flattened data but no
prepending of the number of elements is done for variable sized data.
And Typecast can't convert any type containing variable sized data, but
any arbitrary complex datatype only containing fixed size data, or a
simple array or string can be typecasted.
Rolf Kalbermatter
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions

Scripting: browse for data files in a folder and all subfolders below

Hello,
i'm looking for a command to search for data files (*.dat) in a folder and all subfolders below.
I checked out command DirListGet. This command only search in the declared folder and not in the subfolders below.
Anyone an idea?
I don't want to use a loop-structure for finding subfolders and browse for the data files.
Mr. Buddy

dim result : result = DirListGet("C:\tmp", "*.dat", "filename", "FullFilenamesRecursive")
dim fl : for each fl in result
MsgBox fl
Next
Works fine for me and even recursive.
Alternatively but with mucg more effort and the same result.
Option Explicit 'Forces the explicit declaration of all the variables in a script.
dim folderPath : folderPath = "C:\tmp"
dim files : files = GetFileListRecursive(folderPath)
dim fl : for each fl in files
MsgBox fl
Next
Function GetFileListRecursive(folderPath)
dim fso : Set fso = CreateObject("Scripting.FileSystemObject")
dim results : results = Array()
GetFiles fso, folderPath, results
GetFileListRecursive = results
End Function
Sub GetFiles(fso, folderPath, results)
dim folderObj : Set folderObj = fso.GetFolder(folderPath)
dim f : for Each f In folderObj.Files
if(0 = StrComp(fso.GetExtensionName(f), "dat", 1)) then
dim index : index = ubound(results) + 1
redim Preserve results(index)
results(index) = f.Path
End If
next
dim d : for Each d In folderObj.SubFolders
GetFiles fso, d.Path, results
next
End sub

Using a servlet to send binary data

There is a section of our intranet that need to hold documents that have to be available only for a certain group of users. Here's my approach (if you think of a better one, please tell me):
1-all the files are in a directory upper than WWWRoot so that nobody can directly download them
2-there is a servlet that first: authentificate the user with username-password; and then lists all available documents.
3-When one of these docs is clicked, I want my servlet to open the file using filesystem File object and then copy the content of the selected file in the http response body. What object and method should I use to do so? I dont know if I should use FileReader or something else. Dont forget it will not only has to work with text files, but also with binary data files like swf, powerpointpresentation and the like. So I fear that objects such as FileRead will mess special characters up and corrupt data.
What would you advice me? Use FileReader or another object? Or maybe a completly different approach for the problem? And if you want to give me some sample codes, it would be appreciated.
Thx for your help

There is something I dont understand.. its bizarre. Usually I dont understand why a script is not working, but now, I dont understand why a scritp is working..
Here are the codes that copy from a file to the servlet's ouput:
//basic response configurations
response.setContentType( mimeType );
ServletOutputStream out = response.getOutputStream();
//copy a file f to the servlet's output stream
File f = new File( path );
FileInputStream fis= new FileInputStream(f);
int available = fis.available();
byte[] b = new byte[available];
fis.read(b);
out.write( b );
out.flush();
They work fine. What I dont understand is that I tried them with a file of about 5 megs, and they still worked! I expected them to fail because the variable available is of the type int and then it implies that the maximum file size permitted by the algorithm should be 65536 bytes/1024=64 ko... isn't it????

Data plug-in for binary data with byte streams of variable length

Hi there,
I would like to write a data plug-in to read binary data from file and I'm using DIAdem 10.1.
Each data set in my file consists of binary data with a fixed structure (readable by using direct access channels) and of a byte stream of variable length. The variable length of each byte stream is coded in the fixed data part.
Can anyone tell me how my data plug-in must look like to read such kind of data files?
Many thanks in advance!
Kind regards,
Stefan

Hi Brad,
thank you for the very quick response!
I forgot to mention, that the data in the byte stream can actually be ignored, it is no data to be evaluated in DIAdem (it is picture data and the picture size varies from data set to data set).
So basically, of each data set I would like to read the fixed-structure data (which is the first part of the data set) and discard the variable byte stream (last part of the data set).
Here is a logical (example) layout of my binary data file:
| fixedSize-Value1 | fixedSize-Value2 | fixedSize-Value3 (=length of byte stream) | XXXXXXXXXXXXX (byte stream)
| fixedSize-Value1 | fixedSize-Value2 | fixedSize-Value3 (=length of byte stream) | XXXXXX (byte stream)
| fixedSize-Value1 | fixedSize-Value2 | fixedSize-Value3 (=length of byte stream) | XXXXXXXXXXXXXXXXXXXX (byte stream)
What I would like to show in DIAdem is only fixedSize-Value1 and fixedSize-Value2.
´
If I understood right, would it be possible to set the BlockLength of each data set by assigning Block.BlockLength = fixedSize-Value3 and to use Direct Access Channels for reading fixedSize-Value1 and fixedSize-Value2 ?
Thank you!
Kind regards,
Stefan

Searching binary data file

Similar Messages

Maybe you are looking for