Input file size limition in File Reader Adapter

Hi,
We have a File Read Activity(using File Reader Adapter) which works on a Polling model. Whenever a input file is found in a designated directory it invokes our BPEL web service. During load testing we found out that if input file size exceeds 7 MB then we get "insufficient heap space" error. We found a reference to this size limitation in http://download.oracle.com/docs/cd/E11036_01/integrate.1013/b28994.pdf which has below lines..
"Files that are 7MB or larger in size cannot be delivered. As an alternative, you can debatch large files (if they have multiple messages), and publish these files in messages of size less than 7 MB. This alternative is applicable only to structured files (comma-delimited or fixed position), which contain more than one message. It is not applicable to binary or XML files."
Our customer will have the data in XML format and the size of the files will be pretty huge(running into 49, 69 MBs). Do we have any workaround here or we are left with no option but to ask the customer to split the files and load it? The later option is currently difficult one for the customer since when split there is a possibility of XML becoming invalid. Any pointers in this issue is appreciated.
Thanks!
Balaji

Hi,
BPEL can not process more than 7 MB size xml structred file. but,debatching will work for xml files also provided if the file has repeated multiple structures (means multiple repeated root elements).
Regards,
Dhanumjay

Similar Messages

File Read Adapter

I have got a requirement where I will be having a synchronous process which takes a file name as input and my process as to read that file.
How should I create the file adapter without having the knowledge of the file name?

The concern I have is that in your question it looks like you have a sync process call another BPEl process which calls the file adapter.
I don't know what functionality you are trying to acheive but I suggest the that File adapter is used to invoke the process, not the sync process. Therefore you can use the file adapter hearders to get the file names etc.
cheers
James

File read adapter issue

Hi,
1. Instead of reading file from local pc I need to read from server. how do I do that.
2. Or when I tried to deploy the bpel process(that reads file from local pc) in bpel server, it gives error saying invalid input directory.
Thanks, Ram

Hi Ram,
1) if u want to read the file from the remote machine then you have to use FTP
adapter. N you have to install FTP Server on the Remote machine.
2) You are getting the error saying "invalid input directoy" coz i think you have
specified the local directory in File Adapter wizard 'n you are deploying tht
process on some other machine. Is tht the case..???
/Mishit

How to ensure File Read Adapter handles like SQLLOADER's TRAILING NULLCOLS

I have to read a CSV file which has a specific format but sometimes the trailing columns values can be missing and i would like to handle it in such way that they values are treated as null. This is similar to the SQLLoader's TRAILING NULLCOLS clause.
Currently if my data is as shown below:
As-Of Date,As-Of-Time,Bank ID,Bank Name,State,Acct No,Acct Type,Acct Name,Currency,BAI Type Code,Tran Desc,Debit Amt,Credit Amt,0 Day Flt Amt,1 Day Flt Amt,2+ Day Flt Amt,Customer Ref No,Value Date,Location,Bank Reference,Tran Status,Descriptive Text,Descriptive Text1,Descriptive Text2,Descriptive Text3,Descriptive Text4,Descriptive Text5,Descriptive Text6
20061031,23:59:00,121000248,"WELLS FARGO BANK, N.A.",CA,4121235097,COMMERCIAL DDA,Silicon Image - A/P,USD,475,CHECK PAID,55.86,0,0,0,0,51689,10/31/2006,,IA000313659233,POSTED,,,,,,,
20061031,23:59:00,121000248,"WELLS FARGO BANK, N.A.",CA,4121235097,COMMERCIAL DDA,Silicon Image - A/P,USD,475,CHECK PAID,1377.57,0,0,0,0,51685,10/31/2006,,IA000210166161,POSTED
20061031,23:59:00,121000248,"WELLS FARGO BANK, N.A.",CA,4121235097,COMMERCIAL DDA,Silicon Image - A/P,USD,475,CHECK PAID,1435,0,0,0,0,51621,10/31/2006,,IA000627084628,POSTED
It reads the first row properly , but when in encounters the second row POSTED column , it tries to find the terminated column and since it is not present it reads data from the next line until it finds a comma.
This is not the way i would like to handle it.
We get around 1000-2000 records in that CSV from the Bank and they cant change the way they are sending the file. So , please help in resolving this issue.
Thanks
Sridhar

Hi thanks for the reply
well i've been having a play around but haven't got it to work yet,
Here's what i have so far
private void jButton2MouseClicked(java.awt.event.MouseEvent evt) {
         BufferedReader sb = new BufferedReader();
       sb.append(""+jTextField1.setText()+",");
       sb.append(""+jFormattedTextField1.setText()+"\n");
       sb.append(""+jTextField2.setText()+",");
       sb.append(""+jFormattedTextField2.setText()+"\n");
       sb.append(""+jTextField3.setText()+",");
       sb.append(""+jFormattedTextField3.setText()+"\n");
       sb.append(""+jTextField4.setText()+",");
       sb.append(""+jFormattedTextField4.setText()+"\n");
       sb.append(""+jTextField5.setText()+",");
       sb.append(""+jFormattedTextField5.setText()+"\n");
       ReadFromFile(sb.toString());
    private void ReadFromFile(String s) {
        try {
        BufferedReader in = new BufferedReader(new FileReader("/users/data.txt"));
        String str;
        while ((str = in.readLine()) != null) {
            Process(str);
        in.close();
    } catch (IOException e) {
    }// TODO add your handling code here:
                }im thinking the problem is the parts that say sb.append but i don't know what to in it's place?

How to give two Input File from Adapter Framework to EJB Modules

hi,
i'm using one EJB Module in SAP XI, which converts EDI file
format to XML file format. But now i'm facing a problem, that is,
in SAP XI we can able to give one file as an Input at a time, but
in my case it is <input_file_name>.edi, but my issue is i need
to refer another file also, if it is possible.?
then,
What and how we can able to use this, any clue
inputModuleData.getSupplementalData("<Name>");
or else,
we can able to achieve this through Configuration settings,
Option: addAdditionalFile
please send your suggestion as well.
thanks,
rajesh

hi check the development manual for reference
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/f76f9f90-0201-0010-bf84-d0409ea425f1
Note: reward points if solution found helpfull
Regards
Chandrakanth.k

SOAP message size limitation for sender soap adapter

Hi All,
We are facing critical production issue in case of sender SOAP Adapter,
If the sender soap message is having 114359 Bytes than the Third party is getting exception and SOAP request is not reaching XI.
If the message size is less then 100kbytes then no exception will come.
Is this a limitation that SOAP message size should not exceed 100kbyte?
Thnaks in advance
Best Regards,
Harleen Kaur Chadha

Hi ,
Thanks for your inputs,Could you please tell me which hardware configurations are you talking about?
Are you people talking about harware configurations for XI?
Best Regards,
Harleen Kaur Chadha

What does Java do with the input file?

Hello everyone --
I have a freaky Java problem. I'm running a program that takes an input file, reads it one line at a time, and determines whether the info I'm looking for is contained in it or not. For each line it reads, it decides if that line is good or bad, and broadcasts the findings in a System.out.println(). Every line gets one. When I run the program, I redirect the output to a log file. What's going on when the input file is 13000+ lines long and the log file is only 2500 lines long? This is tweakin' me out.
Thanks for the advice,
-Kwj.

Here's what I have:
boolean isGood = true;
int count = 1;
while((line = in.readLine()) != null)
if (/*some condition is met*/)
{*do some stuff*/}
else
isGood = false;
System.out.println("***" + line);
if (isGood)
/*do some more stuff*/
System.out.println(count + ". " + line)
count++;
}//process all lines
As far as I can tell, I have every line accounted for in the else statement or the last if statement. If the line doesn't meet the req., it doesn't get to go on to the last if processing. All the good ones get processed. I think.

PDF file size limited to graphics memory in Reader?

I've created a form (in LiveCycleDS) that allows for an unlimited number of photos to be loaded into it. I put an image into a subform that is duplicated every time a user clicks a button thus creating an unlimited number of images that can hold photos. I then extended it with Reader Extensions.
I'm running into a problem when I try to load a large number of photos into the form. It gets to about 47MB of images when it locks-up Adobe reader. I've been able to bring up other applications after the lock-up and when I switch back to Reader the artifacts of the other application are then displayed within Reader.
What is the practical limit to the size of a file created with LiveCycle? Is it tied to the amount of graphics memory a computer has? My machine has 2GB of RAM while my video card has only 384MB. I haven't been able to figure out if there is a file size limitation.

It's entirely normal for file size to increase. PDFs are much more compressed than print streams. For some printers the print size is almost constant (a huge bitmap), for others, it's a collection of graphical items and the size varies enormously. Rarely anything you can do.

Passing file read location as a parameter or dynamica to FTP write adapter.

Hi All,
We have a requirement to FTP files to a location. We are planning to use FTP adapter in SOA 11g.
But our requirement is to pass the file read location as a parameter as in case of failure we are planning to read file from the archive location.
But FTP location to which file is written or ftped will always remain constant. Only the read location from which this file is picked needs to be dynamic.
Any pointers on this will be of great help.
Thanks,
Sowmya

Hi,
In 11g we can pass filename and directory name at runtime through properties on receive activity.
If you double click on the Receive activity (connecting to file Adapter), then on Properties tab, you will find three properties jca.file.Directory, jca.file.Filename and jca.file.Size. Select the property which ever you want and assign variable/expression to it.
Value assigned to Variable can be passed runtime as an input to the process.

Need Help-SOA 11g File Adapter unable to delete input file and its crashing

Hi All
Please find the details below:
1. We have created a simple SOA composite to Read file from an input directory, archive the file in an archive directory using Inbound File Adapter Read
and then use Outbound File Adapter Write to move the file to a output directory.
2. File Adapter needs to delete the file after successful read/retrieval.
3. We are using the "Use Trigger File" for invoking the file adapter. This is a new feature in SOA 11g
4. Also we are using the option of reading the file as an attachment as we are not doing any transformation in the composite
Issue Details_
1. When the trigger file is put in the input directory for the first time, the File Adapter reads the file, archives it and moves it to the output directory
2. However it does not delete the input file from the input directory and raises Fatal Exception mentioned below:
[*2011-01-12T16:55:48.639+05:30] [soa_server1] [WARNING] [] [oracle.soa.adapter] [tid: weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@19c243d]*
[userId: <anonymous>] [ecid: 0000IptyLrL9_aY5TrL6ic1DBOS_000009,0] [APP: soa-infra] File Adapter FileAdapterTriggerFilePOC PostProcessor::
Delete failed, the operation will be retried for max of [0] times
[2011-01-12T16:55:48.639+05:30] [soa_server1] [WARNING] [] [oracle.soa.adapter] [tid: weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@19c243d]
[userId: <anonymous>] [ecid: 0000IptyLrL9_aY5TrL6ic1DBOS_000009,0] [APP: soa-infra] File Adapter FileAdapterTriggerFilePOC [[
BINDING.JCA-11042
File deletion failed.
File deletion failed.
File : C:\Dibya\AttachmentTest\InputDir\TestFile3.txt could not be deleted.
Delete the file and restart server. Contact oracle support if error is not fixable.
If any one has faced similar issues, kindly provide pointers on how to resolve it.
Regards,
Dibya

Hi,
Using the file adapter, you can poll from multilple locations...
Keep the following property in your .jca file
<property name="DirectorySeparator" value="," />
While giving the path in File Adapter configuration, keep comma and give the next location....then the file will be picked up from the locations you gave....
Hope this helps...
Thanks,
N

File Sender adapter not reading the first line

Hi,
I have a scenario configured from file to jdbc adapter.
However, I found a problem in my file sender adapter. My file adpater always not picking up the first line from my text even I have set the 'Document Offset' to 0.
Any Ideas? Thanks.
Regards,
Pua

Hi Latika Sethi,
My complete input text file is as below, it contains only 2 testing records:-
H1|DIZ1                          |A0016507    |10000020|09/2007
H2|ABC0001
D|P|0001|Gaji Pokok       |   1,000.09
D|D|0002|Denda              |   1,000.00
D|P|0003|Elaun               |   1,000.00
H1|PUA1                        |A0016508    |10000021|09/2007
H2|ABC0002
D|P|0001|Gaji Pokok       |   2,744.09
D|D|0002|Denda              |   2,000.00
D|P|0003|Elaun               |   2,000.00
After the message mapping, I found the pipeline existed in sxmb_moni as below:-
<?xml version="1.0" encoding="utf-8"?>
<ns:mt_rmp03B_filejdbc_sender xmlns:ns="urn:rmp03:pdrm:ips:jdbcjdbc">
<PaySlip>
..<Header2>
.... <IC>ABC0001</IC>
.. </Header2>
.. <Details>
.... <Jenis>P</Jenis>
.... <No>0001</No>
.... <Deskripsi>Gaji Pokok</Deskripsi>
.... <Jumlah>1,000.09</Jumlah>
.. </Details>
</PaySlip>
<PaySlip>
.. <Header1>
.... <Nama>PUA1</Nama>
.... <KWSP>A0016508</KWSP>
.... <NoGaji>10000021</NoGaji>
.... <Bulan>09/2007</Bulan>
.. </Header1>
.. <Header2>
.... <IC>ABC0002</IC>
.. </Header2>
.. <Details>
.... <Jenis>P</Jenis>
.... <No>0001</No>
.... <Deskripsi>Gaji Pokok</Deskripsi>
.... <Jumlah>2,744.09</Jumlah>
.. </Details>
</Payslip>
There are 2 payslips as for the top payslip node...Header1 tag is missing. It means that during the file sender step....the first line of my record which is :-
"H1|DIZ1                          |A0016507    |10000020|09/2007"
is not read into the xi pipeline. Basically this is the problem i faced.
Has somebody facing the same problem before? Currently I have no choice but moved the First line of the text into Second line and left the first line of the text become null/ empty line. As such both records can be successfully read by the XI.
However what I wondered is sometimes clients will do insert the records into the first line and this might made some data loss.
Thanks...

Nio ByteBuffer and memory-mapped file size limitation

I have a question/issue regarding ByteBuffer and memory-mapped file size limitations. I recently started using NIO FileChannels and ByteBuffers to store and process buffers of binary data. Until now, the maximum individual ByteBuffer/memory-mapped file size I have needed to process was around 80MB.
However, I need to now begin processing larger buffers of binary data from a new source. Initial testing with buffer sizes above 100MB result in IOExceptions (java.lang.OutOfMemoryError: Map failed).
I am using 32bit Windows XP; 2GB of memory (typically 1.3 to 1.5GB free); Java version 1.6.0_03; with -Xmx set to 1280m. Decreasing the Java heap max size down 768m does result in the ability to memory map larger buffers to files, but never bigger than roughly 500MB. However, the application that uses this code contains other components that require the -xMx option to be set to 1280.
The following simple code segment executed by itself will produce the IOException for me when executed using -Xmx1280m. If I use -Xmx768m, I can increase the buffer size up to around 300MB, but never to a size that I would think I could map.
try
String mapFile = "C:/temp/" + UUID.randomUUID().toString() + ".tmp";
FileChannel rwChan = new RandomAccessFile( mapFile, "rw").getChannel();
ByteBuffer byteBuffer = rwChan.map( FileChannel.MapMode.READ_WRITE,
0, 100000000 );
rwChan.close();
catch( Exception e )
e.printStackTrace();
I am hoping that someone can shed some light on the factors that affect the amount of data that may be memory mapped to/in a file at one time. I have investigated this for some time now and based on my understanding of how memory mapped files are supposed to work, I would think that I could map ByteBuffers to files larger than 500MB. I believe that address space plays a role, but I admittedly am no OS address space expert.
Thanks in advance for any input.
Regards- KJ

See the workaround in http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038

File Adapter: Fixed length file read fails when all data not present

Hi
We have a BPEL process that reads fixed length data files. It works fine when all the data elements are available in the file but fails with 'rejected:10002' when even a single data is missing.
How to handle this situation in BPEL file adapter?
Are we doing something wrong or is this a normal functionality.
If yes, then is there any work around for this as this is a very usual business condition which may occur, where all data elements are not mandetory.
fixedLength
==========
2,3,3,2
Data - Successful
============
1234567890
2345678901
3456789012
Data - Failed
===========
1234567890
2345678901
345678901
Thanks in advance
Buddhadev

Hi Naveen,
      Do check the following things,
>>Note : I have been asked to give the Transport Protocol as "NFS" (Whether this is the problem???) I have summarized the complete details below. Please help me
       1.If your file resides on your local network/local computer give NFS(Network file system). if your file resides on a FTP location give FTP and also give the FTP log on parameters.
Additional Parameters
File_MT.fieldFixedLengths 10,10,5
File_MT.fieldNames VendorNumber,VendorName,City
File_MT.fieldSeparator
File_MT..processFieldNames fromConfiguration
       2.If this structure does not match with the input file structure the file adapter wont pick up the file. So check for the help document provided by SAP in the following path.
help.sap.com --> Documentation --> SAPNetWeaver --> SAPNetWeaver '04 --> English --> process integration --> SAP Exchange Infrastructure --> connectivity --> Adapters --> File Adapter
       Your file contains three records
V123456789 A123456789 Bosto
V234567890 B123456789 Atlan
V334587900 C123456789 Austi
You have mentioned the fieldSeperator as space but there is no File_MT.endSeparator '/n' which differentiates between each and every row (record).
        Parameters for Record set Structures mentioned in the sender adapter configuration does not match with the actual file structure .
        Try giving exact structures in the configuration of sender file adapter.
regards,
Aravindh.

How to transfer input file into XI through sender file adapter

Hi
I am quite new to XI. Want to know how I can transfer a file into XI through sender file adapter. I dont have any input file, so I have to create a sample input file and I want to transfer that into XI. Can you please advise me of the steps involved in that. I have finished work in design and directory, now I want to push file the sample file into XI. The receiver adapter is an IDoc adapter. And also a very trivial question. How I can place my sample file into my source directory. Thanks

Hi Rajiv,
If your transport protocol is NFS then you need to put the file in the XI server. Because it becomes the Network File System. For that you can just type the ip address of the XI server in the "run" command or in the browser with "
" then in any of the Folders you can paste your file. But ensure that this is the path you give in your Sender File Adapter.
If your transport protocol is FTP then you need to see that ftp is running in your system. for that go to Control Panel>Add/Remove Programs>Add/Remove Components. In that check the IIS server and go to details and see that ftp is also checked. Now ftp is running in your system.
Now open your C:> drive >InetPub>ftproot . and create a folder here and paste your file. In the configuration of your File sender adapter just mention the name of the folder.
For your information, you can create a file with xml tags and namespace and message type manually (or) you can directly get it from the test tab of message mapping and generate source code.

File adapter input file format

Here the communication scenario is as follows: read a file and send a message via XI to an inbound interface.
A file adapter service has been defined. Message protocol 'file content conversion' has been chosen for the file adapter service. Next the file format has to be defined in communiction channel settings.
Is it possible to define the input record structure in the file to be red itself?
The first line will list all column names. The data records will follow in line two up till end of file.
Regards
Klaus

Now I can add some more information to the facts.
In XI 2.0 the adapter engine was used to connect the integration engine to external systems. The adapter engine converted input file contents to a XML message. The structure of the records in the input file has to be defined. There was one option to read this structure from the first line of the file itself. This option was called 'fromFile'.
Now the current situation in XI 3.0 is that the integration engine reads the input file. Conversion of file contents is still available but the option to read the record structure from the input file doesn't exist anymore.
Does anybody know whether I am right and this functionality isn't available anymore or how to achieve the desired system function in XI 3.0.
Regards
Klaus

Input file size limition in File Reader Adapter

Similar Messages

Maybe you are looking for