Adding large arrays to a text file as new columns

I am trying to merge several large text data files into a single file in Labview 8.0. The files are too large to read in all at once (9-15 million lines each), so decided I need to read them in as smaller chunks, combine the arrays, and write them to a new file.
The reason there are three separate data files was for speed and streaming purposes in the project, and the users wanted the raw, unadulterated data written to file before any kind of manipulation took place.
My VI:
1. Takes a header generated from another VI and writes it to the output file.
2. Creates a time column based on sample rate and the total number of data points
3. Reads in 3 files that each have text data (each data point is 9 bytes wide, there are up to 15 million data points per file.
4. Each iteration of the for loop writes a chunk of 10 to 100 thousand points (Somewhere in there seems to be the fastest it will do), formatted with the time column on the left, then the three data columns, until it's done. I haven't quite figured out how to write the last iteration if there are fewer data points than the chunk size.
Anyways, the main thing I was looking for was suggestions on how to do this faster. It takes about a minute per million points on my laptop to do this operation, and though I recognize it is a lot of data to be moving around, this speed is painfully slow. Any ideas?
Attachments:
Merge Fast Data.vi ‏67 KB

Thanks for the tip. I put the constants outside the array and noticed a little improvement in the speed. I know I could improve the speed by using the binary file VI's but I need the files as tab delimited text files to import them into MATLAB for another group to do analysis. I have not had any luck converting binary files into text files. Is there an easy way to do that? I don't know enough about binary file systems to use them. I looked at the high speed data logger examples but they seemed complicated and hard to adapt to what I need to do. Creating the binary header file seemed like a chore.
I am up for more advice on the VI I posted, or suggestions on different ways to convert a binary file to a MATLAB readable text file.
Thanks!

Similar Messages

Write arrays into a text file in different columns at different times

Hi,
I have a problem write data into a text file. I want to write 4 1D arrays into a text file. The problem is that I need to write it at different time and in different column (in other word, not only append the arrays).
Do you have an idea to solve my problem?
Thank you

A file is long a linear string of data (text). In order ro insert columns, you need to rewrite the entire file, because colums are interlaced over the entire lenght of the file.
So:
read file into 2D array
insert columns using array operations
write resulting 2D array to file again.
(Only if your colums are guaranteed to be fixed width AND you know the final number of colums, you could write the missing columns as spaces and then overwrite later. Still, it will be painful and inefficient, because column data are not adjacent in the file.)
LabVIEW Champion . Do more with less code and in less time .

Reading in an array from a text file

I'm trying to use a text file to load in some configuratin
data (using actionscript 3) I have it working ok for simple stuff
like gamename=Chess&gamescore=100 , etc... but some of the data
needs to be in an array. Can someone please point me in the right
direction to how I can read in an array from a text file?
Thanks!

the easiest way is to create a string with the (soon-to-be)
array elements separated by a delimiter (like a double comma). read
in your string, then use the split() method of strings to split
your string into an array.

Load & retrieve text files from lob column

Dear All,
I have created a table with CLOB column in it.
Now I have to inset text files into that column and retrieve the file/ file contents from that column through stored proc.
Can u please tell me how can i do that??
Thank you,
Gautam

Examine two packages, Dbms_Lob and Utl_File. You are building your file from clob 'piece-by-piece' until you reach the end of clob.
Pseudo
- open file for writing (utl_file)
- get the length of clob
- get first part of clob (dbms_lob.substr)
- apend part of clob (from previous) to the file
- while not end of clob repeat previous two steps

Read text file into clob column

Dear Oracle users and Oracle support,
I have a text file that includes hundreds of data entries. The format is below. What I need to do is read each entry (
<DATALOAD ................</DATALOAD>) into the CLOB column as a table record. Then create a loop in the table to convert each record into XML data type. I have general idea on how to convert CLOB to XML. The difficult part to me is read the each entry in the text file to CLOB table. Please let me know what technique I should use. Any recommendation and sample code are welcome! I appreciate!
Thanks,
Bing
<DATALOAD ................</DATALOAD>
<DATALOAD ................</DATALOAD>
<DATALOAD ................</DATALOAD>
<DATALOAD ................</DATALOAD>

RBYL wrote:
Hi,
Thank you for your response. What I want to achieve is read the text file into ClOB column. There are hundreds reocords in the text file. The format is below. Each entry is '<DATALOAD (sensitive data here, use....... instead)</DATALOAD>' that needs to be read into clob table as a record. That is basically what I need to achieve.
<DATALOAD ................</DATALOAD>
<DATALOAD ................</DATALOAD>
hundreds of them here........
<DATALOAD ................</DATALOAD>
<DATALOAD ................</DATALOAD>So, is it really a text file or is it a well structured XML file?
Just reading it into a CLOB to process is not likely to be the best way.
If each line of the file is a record, then you're likely to be better using something like External Tables.
If it's a structured XML file, then it can be read using CLOB functionality into an XMLTYPE datatype and then shredded down into relational table structures.
Be more clear in what your requirements are and we can help you better.
{message:id=9360002}

Retrieving two dimensional arrays from a text file.

Good evening
I am having a problem with reading a text file, and putting this info into a 2D array to be put into a table.
I have the table already(but no code to add rows to the table incase the array increases)
The array is called Product[a][ b ]
where a is the product number(the row), and b are details about this product.
The file i want to be read looks like this:
data.txt
CPU|AMD|X2, 64bit, 2GHz|$150|9
Video card|NVidia|7800 GTX 256MB|$400|4
e.g. product[2][4] would equal 4
Basically: Type, manufacturer, specifications, price, amount in stock.
the "splitter" or "field terminator" is "|"
This is what i want it to do:
Store fileline into single string,
split string and put it in product[ j ][ i ]
repeat with next lines until nothing is read anymore.
This is the code I have so far:
try{
        FileReader textFileReader = new FileReader("C://BACKUP2//data.txt");
        BufferedReader textReader = new BufferedReader(textFileReader);
        for(int j=0; j<0; j++){
             for(int i=0; i<6; i++){ //i starts at 0, condition: i is under 6, 1 is added to i at each loop/update.
                   ProductT = textReader.readLine();
                   textReader.close();
                   for(String Product[j] : ProductT.split("|")){ //this line is completly wrong, but i don't know how else to do it.
     // for counting for the number of columns
catch (IOException e) {System.out.println(e);}
I don't even think this code makes any bloody sense, I don't know if I'm approaching it correctly.
Would anyone be gracious enough to give me a helping hand and tell me what/how to change my code?
from what I see, the problem lies less in the loops, but more in the splitting of the string. no function seems to like 2D arrays :(

So this is where I stand:
     //Reading
    try{
        FileReader textFileReader = new FileReader("C://BACKUP2//data.txt");
        BufferedReader textReader = new BufferedReader(textFileReader);
        for(int j=0; j>0; j++){ // this will go on forever now or what?
// yes, this will loop forever, which is fine as long as you break out of the loop when you reach the end of the file (which is indicated by textReader.readLine() returning null)
             // replace the for(String currentField : ProductT.split("|")) loop with this one
             //for(int i=0; i<6; i++){ //i starts at 0, condition: i is under 6, 1 is added to i at update.
                  String inputline = textReader.readLine();    // read a single line from the file each time through the loop
                  //while (inputline != null) {
                        if (inputline != null) {    // check if the end of the file has been reached
                       //ProductT = textReader.readLine(); //put to single string
                   //textReader.close(); //shut it off    // move this line to the end of processing, you don't want to close the file until you've read everything from it.
                   //for(String currentField : ProductT.split("|")){   //Split string...      according to JBuilder "The local variable currentField is never read"
                         // switch this loop to use the regular for( ; ; ) type, that way you will have another counter to use as the second index in the product assignment
                                     //currentField = Product[j];
                    product[i][] = currentField; // use the index of the second loop as the second index here
else {
break; // break out of the loop when end of file is reached
     //} // the for i etc. end here
} // End for j et cetera...
     } //end Try
catch (IOException e) {System.out.println(e);}
//cut the crap
// close the reader here
ok - so I am not quite sure about what you mean here
"Assign each of the 5 fields to the five [][j]
positions of the array. "
should i put 5 "fors" in there? - currently the
program is compiling - although they do not appear in
the table.I put comments in the code that should answer those questions.
The code for the table is
Object[][] data = {
          {Product[0][0], Product[0][1],
               Product[0][2], Product[0][3], Product[0][4]},
               {Product[1][0], Product[1][1],
Product[1][2], Product[1][3],
uct[1][3], Product[1][4]},
                      {Product[2][0], Product[2][1],
Product[2][2], Product[2][3],
ct[2][3], Product[2][4]},
                          {Product[3][0], Product[3][1],
Product[3][2],
Product[3][2], Product[3][3], Product[3][4]},
                {Product[4][0], Product[4][1],
Product[4][2],
Product[4][2], Product[4][3], Product[4][4]},
;If I do not declare any of these strings (e.g. I have
only declared Product[0-2][] oldschol style) an error
occurs, telling me that
Caused by:
java.lang.ArrayIndexOutOfBoundsException: 3
     at data.<init>(data.java:112)
     ... 5 more
When you declare product, you'll need to give it a size for both dimensions. The first dimension will be the number of lines to be read from the file. You'll need to know this before allocating the array, so either count them in the file, (if that's good enough for this assignment) or loop through the file once before reading it to count the lines. The second dimension of the array will be 5, since that's how many fields each line has.
btw; do I give those duke stars when all problems of
my life(currently: reading the two dimensional
string) have been resolved?I don't know about [i]all the problems of your life, but when the topic of the thread has been solved, then give them to whoever helped. :)

How to populate an array with random text files.

I am making a Jeopardy program. I have my program set up so that it retrieves 5 random text files. I just want to know how I populate one array with all the lines from the text files my program is retrieving.

You can read a textfile line by line and add each line to an ArrayList. An ArrayList is very much like an array only that it's "open ended". You can start adding lines without first knowing how many you're going to get. If you still want an ordinary "static" array when you're finished reading lines you can easily get one from the ArrayList and then drop the ArrayList.

Extra space adding during spool data into text file

I'm using sql developer to query oracle tables. I can export data using GUI of sql developer, but I wish to use spool query to export the data. Using below query, I was able to export the data to specified location, but my final text file consist of additional spaces after each line. These additional spaces between each line is causing an error to upload the text file in different location.
Did any one face have this issue? Below is my query
set head off
SET FEEDBACK OFF
SET ECHO OFF
SET TRIMSPOOL ON
SET TRIMOUT ON
spool c:\test.txt
select cast(memberid||',12'as varchar2(20))
from abc;
spool off
set head on
set ECHO on
i'm using trimspool, and trimout, but it is not removing any blank spaces.

Sorry, I changed my export.sql to the query you provided.
SET HEAD OFF FEED OFF ECHO OFF
SET LIN 120 PAGES 0 TRIMS ON
SPOOL c:\test.txt
SELECT memberid || ',12' FROM abc;
SPOOL OFF
But,the spool query is still adding additional spaces making length of each line 43 characters in final text file output.

Adding 2 arrays to a Spreadsheet file Help!!

I'm trying to write to a spreadsheet file and I want the date and time in one column and the data from my channels in another. I'm running the problem that my time is in a string format and my channels are in a number format. Attached is what my program currently looks like.
Any help would be great..
Thanks,
Pier
Attachments:
Array Problem.jpg ‏121 KB

Paulo,
I dont understand what you mean about putting the write to spreadsheet in. Do you mean putting it in twice would solve this or not? I'm looking for a top header to be as follows:
Time/Date | Channel 1 name | Channel 2 name | etc....
Could you possibly show me what you mean with the previous example I sent you.
Thanks,
Piero
Hello,
You have your file wrong formatted, because, the first time you write to it, you write as a normal text file, and that will be how excel will interpret it. You have to replace the "Write characters to file.vi" by the "Write to Spreadsheet file string" that I sent you.
If you allow me, I would like to make some extra comments on your code:
1 - When you connect an array to a for loop, there is a property of the tunnel (the point where the array goes in the loop) that is auto-indexing. Right click it and select it. This replace all those index array that you have inside your for loops and the array size wired to the N input of the loop. This will automatically index the array and the loop will automatically run n times where n is the number of elements of the array.
2 - I'm not sure about this, because I didn't run your program, because I don't have a DAQ board installed in this PC, but I think that the secont For loops that you have (the little one) is doing nothing there.
3- try to keep your code more clean, you have it spread for a very big area with a lot of empty spaces and apparentl~y there is no flow from left to right in your code that makes it more difficult to understant it.
Hope this helps,
Paulo

Adding own name in Receiver Text file in receiver FCC

hi,
In my receiver file CC(using file content conversion)
I want to add some names on the receiver text file name automatically.
sender is R3 that is IDOC as MI.
In that field called CURCY on one segment and country one segment.
ex.
Now the receiver file name in FTP server (example)
Result_report_20070721-104810-765.txt
i want to get CURCY, COUNTRY field value on receiver file name in FTP server (it varies)
example:
Result_report_USD_USA_20070721-104810-765.txt
Result_report_EURO_DE_20070721-104810-765.txt
Result_report_EURO_FR_20070721-104810-765.txt
Thanks

hi,
I am getting error in my message monitoring like message are in HOLDING bec of mistake which i made in Receiver FILE CC
as i said:
I followed my help.sap.com link to do this req.
In my receiver file CC(using file content conversion)
I want to add some names on the receiver text file name automatically.
sender is R3 that is IDOC as MI.
In that field called CURCY on one segment and country one segment.
ex.
Now the receiver file name in FTP server (example)
Result_report_20070721-104810-765.txt
i want to get CURCY, COUNTRY field value on receiver file name in FTP server (it varies)
example:
Result_report_USD_USA_20070721-104810-765.txt
Result_report_EURO_DE_20070721-104810-765.txt
Result_report_EURO_FR_20070721-104810-765.txt
My message mapping is
Source is IDOC occurence is 1
Receiver File adapter: 0 to unbounded
My message type on receiver is
MT_Receiver
DT_Receiversub1 (under MT_receiver)
COUNTRY (third value under DT_DT_Receiversub1)
I have mentioned in receiver FILE CC advanced variable substitution like this:
variable Name : var1
Reference: payload:DT_Receiversub1,3
I am not receiving my file in Receiver FTP server.
The mesage are on HOLDING in message monitoring
Thanks

Adding And Removing in a text file

Hi
I'm trying to make a VI that generates a textfile which saves some data each second and give it some timeframe so that it can maintain only the data that has been generated last 10 seconds or so
First I have made 2 different files, one is for saving some data each second into a text file, and the other one is for removing the data
but when i combined those two it doesn't seem to work.
Could anyone have a look at the file and comment on that?
thanks
Attachments:
test2.vi ‏17 KB
Time(Sub).vi ‏8 KB

A couple comments on what you are trying to do. First, doing this with a text file is going to be inherently difficult as you are going to be doing a lot of rewriting of the file. Second, having two VIs (1 to write the data and 1 to remove the old data) is a bad approach - why not design the algorithm so the file is only written to one time for each update cycle? Stop trying to make a poor design work and spend a little time thinking outside the box.
Hint: think about how you calculate a running average in LV, and what if the indicator out wasn't just an indicator but a file write?
Mike...
Certified Professional Instructor
Certified LabVIEW Architect
LabVIEW Champion
"... after all, He's not a tame lion..."
Be thinking ahead and mark your dance card for NI Week 2015 now: TS 6139 - Object Oriented First Steps

Cannot find symbol error -- array fill from text file

When I compile my program I receive a cannot find symbol error for the variable ayears. I thought this snippet would fill the array ayears that could be accessed later in the program but I am getting the error message from my buildGUI() class. What could I have done wrong?
(my file options.txt contains data separated by a comma and a space)
public void getOption(){
     InputStream istream;
     File options = new File("options.txt");
     istream = new FileInputStream(options);
     try {
                           StringBuffer sbuff = new StringBuffer();
                           BufferedReader inbuff = new
BufferedReader(new FileReader(options));
                           String line = "";
                           while((line = inbuff.readLine()) != null) {
                           System.out.println(line);
          sbuff.append(line);
                           inbuff.close();
                          String fileData = sbuff.toString();
          String[] splitData = fileData.split(", ");
          String[] ayears = new String[splitData.length];
     catch(Exception e){
     JOptionPane.showMessageDialog(null,
     "Could not find specified file", "Error Message",
     JOptionPane.ERROR_MESSAGE);
}

Okay -- that helped. I've avoided that error -- now on to the next one, why won't my array fill? It's going to be a long night.
Thanks for the help.

Read from text file and seperate column

hello,
I have a problem reading my text file. This text file is downloaded from weather link. All data will be changed upon the selection of user. my problem is I can read the data column by column but i cannot retrive the right column header for each data. I also attached the test file. Can somebody help me? Thanks in advance..
Attachments:
download.txt ‏8 KB
read text file.JPG ‏11 KB
read data.JPG ‏43 KB

Hi
Sorry, I missed that one. The file formatting is strange though. It uses spaces to separate columns but also within columns. I attached a modified VI that tries to take care of that, but it fails at the columns "Solar Energy" and "Hi Solar Rad.". This is very difficult to read programmatically as there's one space each between "Solar", "Hi", and "Solar" again. So how do we distinguish between column separator and space within column?
Is download.txt the original file? Or was it probably modified by an editor that would replace tabs by space characters?
Attachments:
test-mod-2.vi ‏11 KB

Writing Records in a Text File to Specific Columns in the Text File

Help Please! I am a noooooooooobbbbbb!
I have searched this forum looking for a procedure that will write records from my database into certain col (positions) in a text file. Example:
1234 TONY TEST 84889922 Y 34 TORCHCOMP 34
5678 BOB JOHNS 43534534 N 56 SAFDWEERE 65
I am thinking I would like to setup variaables like:
Consumer_ID = Consumer(position 1,5)
Name = name(position 10,20)
Invoice_No = Invoice(position 30,40)
etc.
Any input will be greatly appreciated
Thank You

Hi,
That's a formatting issue.
Formatting is best done in the front end. For example, using SQL*Plus:
COLUMN     consumer_id     FORMAT 9999     HEADING     c_id
COLUMN     filler_6_9     FORMAT A4     HEADING     " "
COLUMN     name          FORMAT A11
COLUMN     filler_21_29     FORMAT A9     HEADING     " "
COLUMN     invoice_no     FORMAT 9999999999
SET     COLSEP          ""
SET     PAGESIZE     9999
SPOOL     x.dat
SELECT     consumer_id
,     ' '          AS filler_6_9
,     name
,     ' '          AS filler_21_29
,     invoice_no
FROM     table_x;
SPOOL     OFFIf you have to do it in the back end, use functions like TO_CHAR and RPAD to make each part exactly the right length, then concatenate them all into one big string:
SET     PAGESIZE     0
SPOOL     x.dat
SELECT     TO_CHAR ( consumer_id
          , 'fm99999'
          )          -- 1-5
||     '    '               -- 6-9
||     RPAD ( name
          , 11
          )               -- 10-20
||     '         '          -- 21-29
||     TO_CHAR ( invoice_no
          , 'fm9999999999'
          )          -- 30-40
FROM     table_x;
SPOOL     OFFEdited by: Frank Kulash on May 22, 2009 10:40 AM

Fetching a text file into CLOB column in Oracle!

Can anyone please help me to find out how to fetch a text file present on the network on to the CLOB column in Oracle 8i?
I dont want to use BFILE for this.
Please help its urgent.
Love
Prathab
null

Prathab,
This is an example from the SQL package doc for DBMS_LOB, that reads from a bfile, and store in a lob.
CREATE OR REPLACE PROCEDURE Example_l2f IS
lobd BLOB;
fils BFILE := BFILENAME('SOME_DIR_OBJ','some_file');
amt INTEGER := 4000;
BEGIN
SELECT b_lob INTO lobd FROM lob_table WHERE key_value = 42 FOR UPDATE;
dbms_lob.fileopen(fils, dbms_lob.file_readonly);
dbms_lob.loadfromfile(lobd, fils, amt);
COMMIT;
dbms_lob.fileclose(fils);
END;
Hope it helps.
Eric
null

Adding large arrays to a text file as new columns

Similar Messages

Maybe you are looking for