Parsing Huge Fixed length files

Hi All,
I have a requirement to parse a fixed lenght file which is of size 100MB. I have 1.7 million records in that file where each and every record is of length 41. When I am trying to load the file using the LineNumberReader and using the substring it's throwing java.lang.OutMemoryError. I tried with the -Xmx1500M, it's the same. I am also using the reflection API to populate the java beans with the parsed values.
I am thinking a way that I read a chunk of data from the file and parse it, read the next chunk parse it etc... Is there a way like this in java 1.5 or 1.6? I heared about the nio package in JDK1.5. I am looking at it If that can helpful to me.
If any body have an idea or best practice how can we deal huge fixed lenght files in java, please suggest me.
Thank you.

krishna2581 wrote:
I realised that I forgot to mention few points thats why reposted any problem with that?The problem is that if I see only one of your posts, and somebody else sees only the other one, we may end up wasting time duplicating each others' work in answering you. Either continue in your original thread--edit the first post if nobody has replied--or if you must start a new thread, provide a link to it from the original so that the discussion is all in one place.

Similar Messages

MFL Exception while parsing fixed length file

Hello Folks,
I am getting a MFLException when I run a fixed length file through a MFL Format tester.
<MFLException>
<ErrorMessage>Did not completely process all input data.</ErrorMessage>
<Details>
<Detail>
<Name>ErrorCode</Name>
<Value>-3</Value>
</Detail>
<Detail>
<Name>DataOffset</Name>
<Value>2720</Value>
</Detail>
</Details>
</MFLException>
Added a Filler to handle any new line characters at the EOF.
Any ideas what's wrong with my MFL ?
Cheers,
Abhijeet

Hi Atheek,
Thanks for the response. I sent an email to you gmail id with MFL & sample message being used.
Cheers,
Abhijeet

Prasing a huge Fixed lenght file

Hi All,
I have a requirement to parse a fixed lenght file which is of size 100MB. I have 1.7 million records in that file where each and every record is of length 41. When I am trying to load the file using the LineNumberReader and using the substring it's throwing java.lang.OutMemoryError. I tried with the -Xmx1500M, it's the same.
I am thinking a way that I read a chunk of data from the file and parse it, read the next chunk parse it etc... Is there a way like this in java 1.5 or 1.6? I heared about the nio package in JDK1.5. I am looking at it If that can helpful to me.
If any body have an idea or best practice how can we deal huge fixed lenght files in java, please suggest me.
Thank you.

BigDaddyLoveHandles wrote:
abillconsl wrote:
BigDaddyLoveHandles wrote:
abillconsl wrote:
prase noe record at a tiemI need to get outside and breathe some fresh air. That sentence looked fine to me, at first.That's because you've been looking at generics and closures for too long.
... Sorry, I could not resist - I will stop now (starting to sound like an old hen, I am) ... you've been a good sport. ;-)Maybe it's time for me to learn Scala.Scala got my interest for a spell. Then I read a bit about it and it started to sound an awful lot like someone trying to draw all kinds of attention to themselves while trying to ride the "latest" popular programming trends wave.

External tables-Fixed length file

Hi All,
I have a fixed length file that i load daily using an External table. Recently, one of the field, IP length was changed and customer wants to send both old records with 8 byte length and new records with 11 byte length in the same data file, until complete migration takes place.
Will it be possible for External tables to handle this requirement?. Or Is there any other possibility to treat it.
The old file contains 104 fields with IP field position form 490 to 498. Total
The new file contains 104 fields with the IP position from 490 to 501.
Thanks,
Sri.

If the two record types are mixed in the same file, then you will have problems loading them. I can see two possible solutions, in no particular order of preference (using your example data):
1. Redefine the external table something like:
Position (record_type (1:1)
version (2:5)
data (6:41))then parse the remaining fields based on the version number when you select from the external table.
2. Create two external tables over the same file, one for version 1.00 and one for version 1.01 using the LOAD WHEN clause to determine which set of data to load when you select. Something like:
CREATE TABLE version1 ...
ORGANIZATION EXTERNAL ...
ACCESS PARAMETERS
(RECORDS DELIMITED BY newline
LOAD WHEN (version = 1.00)
< definition for the old format >
and
CREATE TABLE version101 ...
ORGANIZATION EXTERNAL ...
ACCESS PARAMETERS
(RECORDS DELIMITED BY newline
LOAD WHEN (version = 1.01)
< definition for the new format >Then yor processing would use something like:
SELECT ip, last_name
FROM version1
UNION ALL
SELECT ip, last_name
FROM version101HTH
John

Please help with reading fixed-length file

I am reading a fixed-length file from a batch program and I need to read through each line, parsing out records in each, and replacing the first character in each line after I finish processing. I am using a RandomAccessFile, but I am not sure if this is best for my needs.
Here is essentially the code that I am using:
RandomAccessFile raf = new RandomAccessFile(File, "rw");
int pointer = 0;
int recordLength = 65;
int counter = 0;
string test = "X";
pointer = raf.getFilePointer();
while ((line = raf.readLine()) != null) {
//set file pointer
seek(counter * recordLength);
counter ++;
if (line.substring(0, 1).equals(test)) {
continue;
var1 = line.substring(7, 22);
var2 = line.substring(23, 41);
var3 = line.substring(42, 52);
var4 = line.substring(53, 53);
//PROCESSING
raf.writeChar(test);

Thanks for the suggestion. I think that the
BufferedReader is the way that I will go. Do you
happen to know the best way to update the 1st Char of
each line as I'm reading each line using Buffered
Writer?Hi,
To update the first char of each line, you could read each line of the file into a StringBuffer, modify the value, and then write the value back out to the file.
I had to do something similar.
eg.
StringBuffer buf = new StringBuffer();
BufferedReader in = new BufferedReader(
    // data file source
new FileReader(db_location));
String s;
// while there are still items to read
while ((s = in.readLine()) != null)
    System.out.println("reading file");
    // split into component parts so we can modify
    StringTokenizer t = new StringTokenizer(s,",");
    int id = Integer.parseInt(t.nextToken());
    String code = t.nextToken();
    int quantity = Integer.parseInt(t.nextToken());
    // copy component parts into StringBuffer
    buf.append(id+",");
    buf.append(code+",");
    buf.append("\n");
     // use StringBuffer methods here to isolate first character
     // use StringBuffer methods to replace first value of input in buffer
     // eg.
          buf.replace(index,length,","+ new_code);
// print out the StringBuffer to rewrite updated file
PrintWriter output = new PrintWriter(
    new BufferedWriter(
    new FileWriter(db_location, false)));
output.println(buf.toString());
output.close();As always, there is more than one way to skin a cat. But this is my way.
I think that JDK 1.4 has some new methods, but not 100% sure of that.
//Daniel.

Gui_download issue - trailing spaces getting truncated for fixed length fil

Hi All,
I have a requirement where I need to download an internal table as a fixed length file.
The code is as follows:
CALL FUNCTION 'GUI_DOWNLOAD'
EXPORTING
BIN_FILESIZE =
FILENAME = L_FILE
FILETYPE = 'ASC'
APPEND = 'X'
WRITE_FIELD_SEPARATOR = ' '
HEADER = '00'
TRUNC_TRAILING_BLANKS = ' '
WRITE_LF = ' '
COL_SELECT = ' '
COL_SELECT_MASK = ' '
DAT_MODE = ' '
IMPORTING
FILELENGTH =
TABLES
DATA_TAB = IT_TEXT
EXCEPTIONS
FILE_WRITE_ERROR = 1
NO_BATCH = 2
GUI_REFUSE_FILETRANSFER = 3
INVALID_TYPE = 4
NO_AUTHORITY = 5
UNKNOWN_ERROR = 6
HEADER_NOT_ALLOWED = 7
Each row in the internal table IT_TEXT is 242 chars long.
The FM is truncatinf the trailing blanks on the file. How do I get the FM to not truncate the trailing blanks in each row?
My internal table has multiple rows and the number of rows on the table should be same as the number of rows on the downloaded file.
I tried setting the WRITE_LF parameter to space.
In this case, the trailing spaces are not truncated(which is as per my requirement), BUT all the rows in the internal table appear in a single line on the downloaded file instead of multiple rows.
I also tried setting the TRUNC_TRAILING_BLANKS field to space but that does not work either. Spaces at the end of the row are still truncated.
so the requirement is: the spaces at the end of each row should not be truncated and
each row on the internal table should have a corresponding row on the downloaded file.
(it is a fixed length file)
I also tried using the following code
class cl_abap_char_utilities definition load.
DATA: BEGIN OF IT_TEXT OCCURS 0,
       TEXT(242) TYPE C,
       cr_lf TYPE c VALUE cl_abap_char_utilities=>cr_lf,
      END OF IT_TEXT.
when i compile, i get the following error
The type "CL_ABAP_CHAR_UTILITIES" is unknown.
Im using R/3 4.6C. Could this be a problem?
Please suggest a solution for this problem.
Thanks!
Sandeep
Edited by: sandeep reddy on Jul 25, 2008 7:16 PM

Hi,
Try this..This worked..Add a dummy character at the end of the internal table...Then pass trunc_trailing_blanks   = ' '...
PARAMETERS: p_file TYPE rlgrap-filename
            DEFAULT 'c:\test_download.txt'.
DATA: BEGIN OF s_data,
        data TYPE char10,
        dummy,      " Added this.
      END OF s_data.
DATA: t_data LIKE TABLE OF s_data.
s_data-data = 'Test'.
APPEND s_data TO t_data.
s_data-data = 'Test2'.
APPEND s_data TO t_data.
s_data-data = 'Test3'.
APPEND s_data TO t_data.
s_data-data = 'Test4'.
APPEND s_data TO t_data.
* Download.
DATA: v_file TYPE string.
v_file = p_file.
CALL FUNCTION 'GUI_DOWNLOAD'
EXPORTING
    filename                = v_file
    trunc_trailing_blanks   = ' '
TABLES
    data_tab                = t_data
EXCEPTIONS
    file_write_error        = 1
    no_batch                = 2
    gui_refuse_filetransfer = 3
    invalid_type            = 4
    no_authority            = 5
    unknown_error           = 6
    header_not_allowed      = 7
    separator_not_allowed   = 8
    filesize_not_allowed    = 9
    header_too_long         = 10
    dp_error_create         = 11
    dp_error_send           = 12
    dp_error_write          = 13
    unknown_dp_error        = 14
    access_denied           = 15
    dp_out_of_memory        = 16
    disk_full               = 17
    dp_timeout              = 18
    file_not_found          = 19
    dataprovider_exception = 20
    control_flush_error     = 21
    OTHERS                  = 22.
Thanks
Naren

XSLT Mapping : XML to Fixed Length File

Hi,
I have to code a XSLT mapping which converts the XML into a Fixed Length File Format. I am getting the output but it has some garbage values (Some extra spaces in front of first record and also extra blank lines before the first record)
I am pasting my xsl sheet :
<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
     <xsl:output method="text" indent="yes" media-type="text/plain"/>
     <xsl:template match="Employees">
          <xsl:for-each select="Employee">
               <xsl:value-of select="Name"/>
               <xsl:value-of select="ID"/>
               <xsl:value-of select="ADD"/>
               <xsl:text>
</xsl:text>
          </xsl:for-each>
     </xsl:template>
My input XML file is as follows:
<?xml version="1.0"?>
<p1:Test02 xmlns:p1="http://www.infosys.com/xi/training/hyd/66289">
        <Employees>
          <Employee>
             <Name>Anurag</Name>
             <ID>1121</ID>
             <ADD>Hyderabad</ADD>
         </Employee>
         <Employee>
             <Name>Divya</Name>
             <ID>1122</ID>
             <ADD>Hyderabad</ADD>
         </Employee>
         <Employee>
             <Name>Rasmi</Name>
             <ID>1123</ID>
             <ADD>Bangalore</ADD>
            </Employee>
     </Employees>
</p1:Test02>
And the output i am receiving is as follows:
    Anurag1121Hyderabad
Divya1122Hyderabad
Rasmi1123Bangalore
Please do help.....

hi,
>>>>
<xsl:output method="text" indent="yes" media-type="text/plain"/>
you allow the spaces by using indent="yes"
try with indent="no"
Regards,
michal
<a href="/people/michal.krawczyk2/blog/2005/06/28/xipi-faq-frequently-asked-questions"><b>XI / PI FAQ - Frequently Asked Questions</b></a>

Upload a Fixed Length file in terms of Bytes..

Hi,
Here is my query.
I have a fixed length file that I need to upload into my program from my presentation server.
The file is in a Shift-JIS Format.
The file is a fixed length format. But it is fixed interms of the number of bytes that each column occupies.
Eg. The 1st column takes 8 bytes, the second 15 bytes, so on and so forth. We do not know the no. of characters each column takes... just the numbe of bytes.
This is how I had approached the upload.
I created an internal table with just one field of type XSTRING.
I used GUI_UPLOAD FM with CODEPAGE = `8000`.
But i noticed during debugging that in each record, the moment a SPACE occured in the input file, it would stop reading and go to the next record in the file. Meaning, I loose all the data after the first occurance of SPACE.
Am I missing something here?? Why does the FM truncate after the first SPACE. ??
Do I need to declare the internal table in any other format..??

" May be placing a carriage return end of each records
" will solve your problem
class cl_abap_char_utilities definition load.
data : begin of itab,
        field1(1) type c,
        field2(2) type c,
        field3(3) type c,
        field4(4) type c,
crlf(2) type c value cl_abap_char_utilities=>cr_lf. "<<<See this line<<<
data : end of itab.
Data : begin of itab1 occurs 0.
        Field(20) type c.
Data : end of itab1.
Loop at itab.
     Move itab to itab1.
     Append itab1.
Endloop.
Open dataset ........
Loop at itab1.
   Transfer itab1 TO dataset.
Endloop.
a®

Creating a Fixed Length File

Greetings,
I'm creating an application that need to create a fixed length file on a UNIX system and need help. I have an internal table(s) which contain structures with fields of different lengths (type c) and so I have a routine that concatenates these fields into a single record to be sent to a file using the open dataset. This process is squeezing out all my spaces and so my fixed length file is lost. Can someone assist in creating a fixed length file from an internal table without using delimiters?
Thanks!

" May be placing a carriage return end of each records
" will solve your problem
class cl_abap_char_utilities definition load.
data : begin of itab,
        field1(1) type c,
        field2(2) type c,
        field3(3) type c,
        field4(4) type c,
crlf(2) type c value cl_abap_char_utilities=>cr_lf. "<<<See this line<<<
data : end of itab.
Data : begin of itab1 occurs 0.
        Field(20) type c.
Data : end of itab1.
Loop at itab.
     Move itab to itab1.
     Append itab1.
Endloop.
Open dataset ........
Loop at itab1.
   Transfer itab1 TO dataset.
Endloop.
a®

Reading a fixed length file

Hi All,
I am trying to read a fixed length file with .DAT extension through an FTP Adapter. I am using a read(polling) operation.
In the file there are three records(H,D,T). The length of all the three records is more than 100.When I am trying to build a schema for that file using native format builder, maximum position that i get is 100 and if i manually try to put the position beyond 100 say 120,while building the schema and then run the composite using the generated schema, the file is not polled from the location So if anyone could help me to clarify my doubts regarding reading of fixed length files.
a. Can i read i fixed length file which contains records whose position is more than 100.
b. If yes, then how do i do that?
Any help is appreciated.
Thanks in Advance.

Here's one way to start. Extend this class for the particular functionality you want:import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
/** Processes binary files which have embedded data records. */
public abstract class BinaryInputFile extends File
* Class constructor.
* @param fileName The input file name.
* @throws NullPointerException if the <code>pathname</code> parameter is <code>null</code>
protected BinaryInputFile(String fileName) throws NullPointerException
   super(fileName);
* Reads the file, one line at a time, passing each line to the subclass� process()
* function.
* @throws IOException if an error occurs.
public void process() throws IOException
   DataInputStream stream = new DataInputStream(new FileInputStream(this));
   process(stream);
* Process the file data.
* @param stream The input stream.
protected abstract void process(DataInputStream stream);
}

Reading fixed length file with different record types

Hi,
I need to read a fixed-length file with different record types, but the record identifier is in 31st position and not in 1st position.
But if I give 31 as position in File adpater wizard, BPEL takes whole 1-31 as identifier.
How we need to read such files.
Thanks
Ravdeep

hi ,
u cannot use the default wzard for this
use some thing like this nxsd:lookAhead="30" nxsd:lookFor="S"have a look at the below link it has some examples
http://download.oracle.com/docs/cd/B31017_01/integrate.1013/b28994/nfb.htm

How to handle a fixed length file without newline?

Hi Experts,
I'd like to handle a fixed length file without newline by sender file adapter.
A file like following.
It contains three recores."AAXBBBXCCCCX" is one record.
AA1BBB1CCCC1AA2BBB2CCCC2AA3BBB3CCCC3
I tried that following two parameters set. But only first recored was read.
fieldFixedLengths
fieldFixedLengthType
Please tell me how to handle.
Thanks
Shinya Kawagoe.

For this case we wrote a simple Adapter Module inserting an end of line character after an offset.
This way it can be reused in many interfaces.
And reading the whole file may not be an option in case of large source files. May cause performance / memory issues.
eolbean.offset = <recordLlen>
XMLPayload xmlpayload = msg.getDocument();
byte[] content = xmlpayload.getContent();
byte crlf = 0x0A;
int current = 0;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int lines = content.length / recordLen;
do
     lines--;
     baos.write(content, current, recordLen);
     if (lines > 0) // if other lines, eol required
          baos.write(crlf);
          current += recordLen;
} while (lines > 0);
xmlpayload.setContent(baos.toByteArray());
baos.close();
Audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS,     MODULE + " Done EOLing.");

How to create a fixed length file

Can anyone suggest a simple technique for creating fixed
length files? CSV and delimited files are pretty simple using
CFFILE. Is there an elegant way to create fixed length
files?

Simplest: Use LJustify() or RJustify() on your lines of data.
EG:
<cfset FileLine = RJustify (LineOfText, 128)>

Fixed length file sending to user.

My output filxed length file contains records of length 750 chars.
My problem is I need to emailed such file to user...so the user can download the file & if error is there then, user can correct the values in this file and then rerun the program with this corrected fixed length file.
I can send this fixed lenth file as an attachment also.
How can I send, this fixed length file to the user mail id ?
Could you please help me with the code?
THANKS IN ADV.

it is possible in case of binary files not text files just read Thomas comments in the Blog.
Sending E-Mail from ABAP - Version 610 and Higher - BCS Interface

Fixed Length File attachment to Mail Receiver

Hi,
I have an interface requirement IDOC - PI - Fixed Length File attached to email.
Is it possible to do this? I need to pass the IDOC through a mapping and content conversion to build the fixed length file, I then need to attach this file to an email and send it to a user.
Any help would be greatly appreciated.
Thanks
Gareth

Hi,
I actually need to convert the IDOC to a different file format before sending it on. Eg.
<IDOC>
     <Segment 1...n>
           <username>User</username>
           <address>Address 1</Address>
     </Segment>
     <Segment>
           <username>User2</username>
           <address>Address 2</Address>
     </Segment>
<IDOC>
Converted to flat file with structure:
User......Address 1.....
User2....Address 2.....
The file needs to have fields of Fixed Length (thats what the .... after each field is to represent)
So I think the Data Type I am using for the file needs to go through Content Conversion to do this change, and then the file attached to an email.
I hope this makes sense.

Parsing Huge Fixed length files

Similar Messages

Maybe you are looking for