Prasing a huge Fixed lenght file

Hi All,
I have a requirement to parse a fixed lenght file which is of size 100MB. I have 1.7 million records in that file where each and every record is of length 41. When I am trying to load the file using the LineNumberReader and using the substring it's throwing java.lang.OutMemoryError. I tried with the -Xmx1500M, it's the same.
I am thinking a way that I read a chunk of data from the file and parse it, read the next chunk parse it etc... Is there a way like this in java 1.5 or 1.6? I heared about the nio package in JDK1.5. I am looking at it If that can helpful to me.
If any body have an idea or best practice how can we deal huge fixed lenght files in java, please suggest me.
Thank you.

BigDaddyLoveHandles wrote:
abillconsl wrote:
BigDaddyLoveHandles wrote:
abillconsl wrote:
prase noe record at a tiemI need to get outside and breathe some fresh air. That sentence looked fine to me, at first.That's because you've been looking at generics and closures for too long.
... Sorry, I could not resist - I will stop now (starting to sound like an old hen, I am) ... you've been a good sport. ;-)Maybe it's time for me to learn Scala.Scala got my interest for a spell. Then I read a bit about it and it started to sound an awful lot like someone trying to draw all kinds of attention to themselves while trying to ride the "latest" popular programming trends wave.

Similar Messages

Parsing Huge Fixed length files

Hi All,
I have a requirement to parse a fixed lenght file which is of size 100MB. I have 1.7 million records in that file where each and every record is of length 41. When I am trying to load the file using the LineNumberReader and using the substring it's throwing java.lang.OutMemoryError. I tried with the -Xmx1500M, it's the same. I am also using the reflection API to populate the java beans with the parsed values.
I am thinking a way that I read a chunk of data from the file and parse it, read the next chunk parse it etc... Is there a way like this in java 1.5 or 1.6? I heared about the nio package in JDK1.5. I am looking at it If that can helpful to me.
If any body have an idea or best practice how can we deal huge fixed lenght files in java, please suggest me.
Thank you.

krishna2581 wrote:
I realised that I forgot to mention few points thats why reposted any problem with that?The problem is that if I see only one of your posts, and somebody else sees only the other one, we may end up wasting time duplicating each others' work in answering you. Either continue in your original thread--edit the first post if nobody has replied--or if you must start a new thread, provide a link to it from the original so that the discussion is all in one place.

Fixed lenght file and row separator

Hi experts.
Im using open data set and close datd set commands to create a text file in background.
It is separated by comma.when a field is blank it just displays , , .
I want the file to be a fixed length file. I want to see as many spaces as the length on the field if it is initial.
Also i see a smaal box as the row separator. I want a new line separator ( ASCII char) as the row separator.
Any suggestions ?
Thank You
Radhika.

Hi,
Instead of assigning the work area to file, use MOVE command :
gt_file = gt_data. " Comment this line
Move gt_data to gt_file. " Add this line
Use cl_abap_char_utilities=>NEWLINE for line seaparator.
Best regards,
Prashant

How to create flat file with fixed lenght records

I need help to export an Oracle table to a flat file with fixed lenght and without columns separator.
the fixed length is the more important demand.
My table have 50 columns with varchar, date and number .
Date and number columns may be empty, null o with values.
Thanks a lot for any help.
[email protected]

Hi,
You can use this trick:
SQL>desc t
Name                                      Null?    Type
NAME                                               VARCHAR2(20)
SEX                                                VARCHAR2(1)
SQL>SELECT LENGTH(LPAD(NAME,20,' ')||LPAD(SEX,1,' ')), LPAD(NAME,20,' ')||LPAD(SEX,1,' ') FROM T;
LENGTH(LPAD(NAME,20,'')||LPAD(SEX,1,'')) LPAD(NAME,20,'')||LPA
                                      21                    aF
                                      21                    BM
                                      21                    CF
                                      21                    DM
4 rows selected.
SQL>SELECT * FROM t;
NAME                 S
a                    F
B                    M
C                    F
D                    M
4 rows selected.Regards

Problem with RSCRM_BAPI (extraction to Fixed Lenght txt file)

Hi Experts,
I have a question about a RSCRM_BAPI limitation:
I want to extract a Query, using this transaction, to a fixed lenght .txt file. The query has a tabular structure like this:
CHARACT1 | CHARACT2 | CHARACT3 | CHARACT4 | KEYFIG_MONTH1 | KEYFIG_MONTH2 | .... | KEYFIG_MONTH12.
so i have a tabular structure with some CH and a KF repeated for the 12 month of the year.
The problem is that the fixed lenght .txt file extracted with RSCRM_BAPI stops to the month 11, month 12 is discarded. If i try to extract a .csv file with the same query all the 12 month are included in the file.
Question: Is it possible that RSCRM_BAPI has a row lenght limitation considering the fixed lenght .txt file?
Many thanks in advance.
Fabio

How did you sechudle the Query/Report Using BAPI, i'm having similar Problem. Please Can you help me out.
My problem is , i have a query which is schedule for every month 2 wed and a bapi is associated to The query.
for some reason's the query is not working properly .
Can you please help.

Gui_download issue - trailing spaces getting truncated for fixed length fil

Hi All,
I have a requirement where I need to download an internal table as a fixed length file.
The code is as follows:
CALL FUNCTION 'GUI_DOWNLOAD'
EXPORTING
BIN_FILESIZE =
FILENAME = L_FILE
FILETYPE = 'ASC'
APPEND = 'X'
WRITE_FIELD_SEPARATOR = ' '
HEADER = '00'
TRUNC_TRAILING_BLANKS = ' '
WRITE_LF = ' '
COL_SELECT = ' '
COL_SELECT_MASK = ' '
DAT_MODE = ' '
IMPORTING
FILELENGTH =
TABLES
DATA_TAB = IT_TEXT
EXCEPTIONS
FILE_WRITE_ERROR = 1
NO_BATCH = 2
GUI_REFUSE_FILETRANSFER = 3
INVALID_TYPE = 4
NO_AUTHORITY = 5
UNKNOWN_ERROR = 6
HEADER_NOT_ALLOWED = 7
Each row in the internal table IT_TEXT is 242 chars long.
The FM is truncatinf the trailing blanks on the file. How do I get the FM to not truncate the trailing blanks in each row?
My internal table has multiple rows and the number of rows on the table should be same as the number of rows on the downloaded file.
I tried setting the WRITE_LF parameter to space.
In this case, the trailing spaces are not truncated(which is as per my requirement), BUT all the rows in the internal table appear in a single line on the downloaded file instead of multiple rows.
I also tried setting the TRUNC_TRAILING_BLANKS field to space but that does not work either. Spaces at the end of the row are still truncated.
so the requirement is: the spaces at the end of each row should not be truncated and
each row on the internal table should have a corresponding row on the downloaded file.
(it is a fixed length file)
I also tried using the following code
class cl_abap_char_utilities definition load.
DATA: BEGIN OF IT_TEXT OCCURS 0,
       TEXT(242) TYPE C,
       cr_lf TYPE c VALUE cl_abap_char_utilities=>cr_lf,
      END OF IT_TEXT.
when i compile, i get the following error
The type "CL_ABAP_CHAR_UTILITIES" is unknown.
Im using R/3 4.6C. Could this be a problem?
Please suggest a solution for this problem.
Thanks!
Sandeep
Edited by: sandeep reddy on Jul 25, 2008 7:16 PM

Hi,
Try this..This worked..Add a dummy character at the end of the internal table...Then pass trunc_trailing_blanks   = ' '...
PARAMETERS: p_file TYPE rlgrap-filename
            DEFAULT 'c:\test_download.txt'.
DATA: BEGIN OF s_data,
        data TYPE char10,
        dummy,      " Added this.
      END OF s_data.
DATA: t_data LIKE TABLE OF s_data.
s_data-data = 'Test'.
APPEND s_data TO t_data.
s_data-data = 'Test2'.
APPEND s_data TO t_data.
s_data-data = 'Test3'.
APPEND s_data TO t_data.
s_data-data = 'Test4'.
APPEND s_data TO t_data.
* Download.
DATA: v_file TYPE string.
v_file = p_file.
CALL FUNCTION 'GUI_DOWNLOAD'
EXPORTING
    filename                = v_file
    trunc_trailing_blanks   = ' '
TABLES
    data_tab                = t_data
EXCEPTIONS
    file_write_error        = 1
    no_batch                = 2
    gui_refuse_filetransfer = 3
    invalid_type            = 4
    no_authority            = 5
    unknown_error           = 6
    header_not_allowed      = 7
    separator_not_allowed   = 8
    filesize_not_allowed    = 9
    header_too_long         = 10
    dp_error_create         = 11
    dp_error_send           = 12
    dp_error_write          = 13
    unknown_dp_error        = 14
    access_denied           = 15
    dp_out_of_memory        = 16
    disk_full               = 17
    dp_timeout              = 18
    file_not_found          = 19
    dataprovider_exception = 20
    control_flush_error     = 21
    OTHERS                  = 22.
Thanks
Naren

XSLT Mapping : XML to Fixed Length File

Hi,
I have to code a XSLT mapping which converts the XML into a Fixed Length File Format. I am getting the output but it has some garbage values (Some extra spaces in front of first record and also extra blank lines before the first record)
I am pasting my xsl sheet :
<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
     <xsl:output method="text" indent="yes" media-type="text/plain"/>
     <xsl:template match="Employees">
          <xsl:for-each select="Employee">
               <xsl:value-of select="Name"/>
               <xsl:value-of select="ID"/>
               <xsl:value-of select="ADD"/>
               <xsl:text>
</xsl:text>
          </xsl:for-each>
     </xsl:template>
My input XML file is as follows:
<?xml version="1.0"?>
<p1:Test02 xmlns:p1="http://www.infosys.com/xi/training/hyd/66289">
        <Employees>
          <Employee>
             <Name>Anurag</Name>
             <ID>1121</ID>
             <ADD>Hyderabad</ADD>
         </Employee>
         <Employee>
             <Name>Divya</Name>
             <ID>1122</ID>
             <ADD>Hyderabad</ADD>
         </Employee>
         <Employee>
             <Name>Rasmi</Name>
             <ID>1123</ID>
             <ADD>Bangalore</ADD>
            </Employee>
     </Employees>
</p1:Test02>
And the output i am receiving is as follows:
    Anurag1121Hyderabad
Divya1122Hyderabad
Rasmi1123Bangalore
Please do help.....

hi,
>>>>
<xsl:output method="text" indent="yes" media-type="text/plain"/>
you allow the spaces by using indent="yes"
try with indent="no"
Regards,
michal
<a href="/people/michal.krawczyk2/blog/2005/06/28/xipi-faq-frequently-asked-questions"><b>XI / PI FAQ - Frequently Asked Questions</b></a>

Fixed lengh file processing

Hi All,
I'm doing fixed lenghth file by small example. but my receiver adapter is showing following error
Receiver Adapter v2123 for Party '', Service 'F1_BS':
Configured at 2006-04-19 17:28:19 BST
No message processing until now
Following is my configuration
My file
123456
venugopalsirangi
Sender communication channel
Record structure: H1,,sub1,
FNAME               Value
H1.fieldNames           KF
H1.fieldFixedLengths     6
H1.fieldFixedType      char
H1.endSeparator          'nl'
sub1.fieldNames          KF,Lname,Sname
sub1.fieldFixedLengths     4,5,7
sub1..fieldFixedType      char
sub1.endSeparator     'nl'
H1.keyFieldValue     '144857'
sub1.keyFieldValue     'venu'
Receiver communication channel
Record structure:
FNAME               Value
H1.fieldNames           KF
H1.fieldFixedLengths     6
H1.fieldFixedType      char
H1.endSeparator          'nl'
sub1.fieldNames          KF,Lname,Sname
sub1.fieldFixedLengths     4,5,7
sub1..fieldFixedType      char
sub1.endSeparator     'nl'
H1.keyFieldValue     '144857'
sub1.keyFieldValue     'venu'
Regards,
venu.

Hi Prateek,
XML file is successfully picked up by the adpater without content conversion.
If I do content conversion it is not receiver adapter not processing the message.
MY file.
0112345010101
021111112222
03100001111112222
03100011111212223
041000011111
021231116722
03100781119012332
041005611001
059453287699
MY xml file
<?xml version="1.0" encoding="UTF-8" ?>
- <ns0:LSOUT_MT xmlns:ns0="urn://LengthSpecific">
- <Header>
<Key1>01</Key1>
<name>12345</name>
<date>010101</date>
</Header>
- <Hbatch>
<Key2>02</Key2>
<hvalue1>11111</hvalue1>
<hvalue2>12222</hvalue2>
</Hbatch>
- <body>
<Key3>03</Key3>
<bvalue1>10000</bvalue1>
<bvalue2>111111</bvalue2>
<bvalue3>2222</bvalue3>
</body>
- <tbatch>
<Key4>04</Key4>
<tvalue1>10000</tvalue1>
<tvalue2>11111</tvalue2>
</tbatch>
- <trailer>
<Key5>05</Key5>
<value1>94532</value1>
<value2>87699</value2>
</trailer>
</ns0:LSOUT_MT>
MY content conversion of sender adapter is:
record struct :Header,1,Hbatch,,body,,tbatch,*,trailer,1
Header.fieldFixedLengths 2,5,6
Header.keyFieldValue 01
Header.fieldNames Key1,name,date
the same approach i used for Hbatch, body and tbatch..
My content conversion for receiver adapter is.
Record structure: Header,Hbatch,body,tbatch,trailer
Header.fieldFixedLengths 2,5,6
Header.fieldNames Key1,name,date
Header.processFieldNames fromConfiguration
rest as above.
Regards,
venu.

Is it possible to import a Fixed Text file that does not have a header?

I'm trying to import a Fixed Text file through Import Manager, but the text file I am receiving does not have a header on it (the first row is valid data). When I try importing this, Import Manager uses the first row of data as the headings. Is there any way to turn this off and tell Import Manager there is no header data in the file?
Thanks!

Hi Ryan ,ur requirement can be met only when u import the file as ODBC .for this follow the below steps.
1)Go to control panel,in ur settings,select administrative tools
2) In that select DATA SOURCE(ODBC ).
3)Go to SYSTEM DSN tab select tab .
4)Select ur file type MICROSOFT TEST DRIVER (.txt,.csv)
5)Double click on "Import DSN" and in the blank give the name of DATA SOURCE NAME any name u want to give.
6)Uncheck the use current directory box and select the directory ur file is it will automatically show ur source file select it and click ok.
7) Now when u import ur file as ODBC give the name of ur DATA source name which u have given and it will show ur file in the source preview itself .
In this way u can import ur file as ODBC.
Hope this may help u.
Regards
Ankit

External tables-Fixed length file

Hi All,
I have a fixed length file that i load daily using an External table. Recently, one of the field, IP length was changed and customer wants to send both old records with 8 byte length and new records with 11 byte length in the same data file, until complete migration takes place.
Will it be possible for External tables to handle this requirement?. Or Is there any other possibility to treat it.
The old file contains 104 fields with IP field position form 490 to 498. Total
The new file contains 104 fields with the IP position from 490 to 501.
Thanks,
Sri.

If the two record types are mixed in the same file, then you will have problems loading them. I can see two possible solutions, in no particular order of preference (using your example data):
1. Redefine the external table something like:
Position (record_type (1:1)
version (2:5)
data (6:41))then parse the remaining fields based on the version number when you select from the external table.
2. Create two external tables over the same file, one for version 1.00 and one for version 1.01 using the LOAD WHEN clause to determine which set of data to load when you select. Something like:
CREATE TABLE version1 ...
ORGANIZATION EXTERNAL ...
ACCESS PARAMETERS
(RECORDS DELIMITED BY newline
LOAD WHEN (version = 1.00)
< definition for the old format >
and
CREATE TABLE version101 ...
ORGANIZATION EXTERNAL ...
ACCESS PARAMETERS
(RECORDS DELIMITED BY newline
LOAD WHEN (version = 1.01)
< definition for the new format >Then yor processing would use something like:
SELECT ip, last_name
FROM version1
UNION ALL
SELECT ip, last_name
FROM version101HTH
John

Upload a Fixed Length file in terms of Bytes..

Hi,
Here is my query.
I have a fixed length file that I need to upload into my program from my presentation server.
The file is in a Shift-JIS Format.
The file is a fixed length format. But it is fixed interms of the number of bytes that each column occupies.
Eg. The 1st column takes 8 bytes, the second 15 bytes, so on and so forth. We do not know the no. of characters each column takes... just the numbe of bytes.
This is how I had approached the upload.
I created an internal table with just one field of type XSTRING.
I used GUI_UPLOAD FM with CODEPAGE = `8000`.
But i noticed during debugging that in each record, the moment a SPACE occured in the input file, it would stop reading and go to the next record in the file. Meaning, I loose all the data after the first occurance of SPACE.
Am I missing something here?? Why does the FM truncate after the first SPACE. ??
Do I need to declare the internal table in any other format..??

" May be placing a carriage return end of each records
" will solve your problem
class cl_abap_char_utilities definition load.
data : begin of itab,
        field1(1) type c,
        field2(2) type c,
        field3(3) type c,
        field4(4) type c,
crlf(2) type c value cl_abap_char_utilities=>cr_lf. "<<<See this line<<<
data : end of itab.
Data : begin of itab1 occurs 0.
        Field(20) type c.
Data : end of itab1.
Loop at itab.
     Move itab to itab1.
     Append itab1.
Endloop.
Open dataset ........
Loop at itab1.
   Transfer itab1 TO dataset.
Endloop.
a®

Creating a Fixed Length File

Greetings,
I'm creating an application that need to create a fixed length file on a UNIX system and need help. I have an internal table(s) which contain structures with fields of different lengths (type c) and so I have a routine that concatenates these fields into a single record to be sent to a file using the open dataset. This process is squeezing out all my spaces and so my fixed length file is lost. Can someone assist in creating a fixed length file from an internal table without using delimiters?
Thanks!

" May be placing a carriage return end of each records
" will solve your problem
class cl_abap_char_utilities definition load.
data : begin of itab,
        field1(1) type c,
        field2(2) type c,
        field3(3) type c,
        field4(4) type c,
crlf(2) type c value cl_abap_char_utilities=>cr_lf. "<<<See this line<<<
data : end of itab.
Data : begin of itab1 occurs 0.
        Field(20) type c.
Data : end of itab1.
Loop at itab.
     Move itab to itab1.
     Append itab1.
Endloop.
Open dataset ........
Loop at itab1.
   Transfer itab1 TO dataset.
Endloop.
a®

Reading a fixed length file

Hi All,
I am trying to read a fixed length file with .DAT extension through an FTP Adapter. I am using a read(polling) operation.
In the file there are three records(H,D,T). The length of all the three records is more than 100.When I am trying to build a schema for that file using native format builder, maximum position that i get is 100 and if i manually try to put the position beyond 100 say 120,while building the schema and then run the composite using the generated schema, the file is not polled from the location So if anyone could help me to clarify my doubts regarding reading of fixed length files.
a. Can i read i fixed length file which contains records whose position is more than 100.
b. If yes, then how do i do that?
Any help is appreciated.
Thanks in Advance.

Here's one way to start. Extend this class for the particular functionality you want:import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
/** Processes binary files which have embedded data records. */
public abstract class BinaryInputFile extends File
* Class constructor.
* @param fileName The input file name.
* @throws NullPointerException if the <code>pathname</code> parameter is <code>null</code>
protected BinaryInputFile(String fileName) throws NullPointerException
   super(fileName);
* Reads the file, one line at a time, passing each line to the subclass� process()
* function.
* @throws IOException if an error occurs.
public void process() throws IOException
   DataInputStream stream = new DataInputStream(new FileInputStream(this));
   process(stream);
* Process the file data.
* @param stream The input stream.
protected abstract void process(DataInputStream stream);
}

Reading fixed length file with different record types

Hi,
I need to read a fixed-length file with different record types, but the record identifier is in 31st position and not in 1st position.
But if I give 31 as position in File adpater wizard, BPEL takes whole 1-31 as identifier.
How we need to read such files.
Thanks
Ravdeep

hi ,
u cannot use the default wzard for this
use some thing like this nxsd:lookAhead="30" nxsd:lookFor="S"have a look at the below link it has some examples
http://download.oracle.com/docs/cd/B31017_01/integrate.1013/b28994/nfb.htm

How to handle a fixed length file without newline?

Hi Experts,
I'd like to handle a fixed length file without newline by sender file adapter.
A file like following.
It contains three recores."AAXBBBXCCCCX" is one record.
AA1BBB1CCCC1AA2BBB2CCCC2AA3BBB3CCCC3
I tried that following two parameters set. But only first recored was read.
fieldFixedLengths
fieldFixedLengthType
Please tell me how to handle.
Thanks
Shinya Kawagoe.

For this case we wrote a simple Adapter Module inserting an end of line character after an offset.
This way it can be reused in many interfaces.
And reading the whole file may not be an option in case of large source files. May cause performance / memory issues.
eolbean.offset = <recordLlen>
XMLPayload xmlpayload = msg.getDocument();
byte[] content = xmlpayload.getContent();
byte crlf = 0x0A;
int current = 0;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int lines = content.length / recordLen;
do
     lines--;
     baos.write(content, current, recordLen);
     if (lines > 0) // if other lines, eol required
          baos.write(crlf);
          current += recordLen;
} while (lines > 0);
xmlpayload.setContent(baos.toByteArray());
baos.close();
Audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS,     MODULE + " Done EOLing.");

Prasing a huge Fixed lenght file

Similar Messages

Maybe you are looking for