Huge Files Processed in XI

Hi Experts,
Scenario: File ---> XI -
>JMS
I have a requirement where XI will processed above 5MB files uing content conversion. Is there any performance issue in XI to proceesed huge files.
Does XI support above 5MB files? If XI processed huge file then what would be the limitation? what would be the best approach to achive the sceario.
Would apreciate your inputs.
Regards
Sameer,

Thanks Udo for your reply.
Since I am not using BPM in this scenario. As the file contains above 22,000 records so can we split the file at every 5000th record and process it by using Recordsets per Message in sender FCC.
I am using Graphical mapping (1:1) with FCC both the side i,e FILE --- XI --- JMS
Is there any performance issue in XI while doing this?
Sameer!

Similar Messages

Huge size file processing in PI

Hi Experts,
1. I have seen blogs which explains processing huge files. for file and sftp
SFTP Adapter - Handling Large File
File/FTP Adapter - Large File Transfer (Chunk Mode)
Here also we have constrain that we can not do any mapping. it has to be EOIO Qos.
would it be possible, to process 1 GB size file and do mapping? which hardware factor will decide that sytem is capable of processing large size with mapping?
is it number of CPUs,Applications server(JAVA and ABAP),no of server nodes,java,heap size?
if my system if able to process 10 MB file with mapping there should be something which is determining the capability.
this kind of huge size file processing will fit into some scenarios. for example,Proxy to soap scenario with 1GB size message exchange does not make sense. no idea if there is any web service will handle such huge file.
2. consider pi is able to process 50 MB size message with mapping. in order to increase the performance what are the options we have in PI
i have come across these two point many times during design phase of my project. looking for your suggestion
Thanks.

Hi Ram,
You have not mentioned what sort of Integration it is.You just mentioned as FILE.I presume it is FILE To FILE scenario.In this case in PI 711 i am able to process 100MB(more than 1Million records ) file size with mapping(File is in the delta extract in SAP ECC AL11).In the sender file adapter i have chosen recordset per message and processed the messages in bit and pieces.Please note this is not the actual standard chunk mode.The initial run of the sender adapter will load the 100MB file size into the memory and after that messages will be sent to IE based on recordset per message.If it is more than 100MB PI Java starts bouncing because of memory issues.Later we have redesigned the interface from proxy to file asyn and proxy will send the messages to PI in chunks.In a single run it will sent 5000 messages.
For PI 711 i believe we have the memory limtation of the cluster node.Each cluster node can't be more than 5GB again processing depends on the number of Java app servers and i think this is no more the limitation from PI 730 version and we can use 16GB memory as the cluser node.
this kind of huge size file processing will fit into some scenarios. for example,Proxy to soap scenario with 1GB size message exchange does not make sense. no idea if there is any web service will handle such huge file.
If i understand this i think if it is asyn communication then definitely 1GB data can sent to webservice however messages from Proxy should sent to PI in batches.May be the same idea can work for Sync communication as well however timeouts in receiver channel will be the next issue.Increasing time outs globally is not best practice however if you are on 730 or later version you can increase timeouts specific to your scenario.
To handle 50 MB file size make sure you have the additional java app servers.I don't remember exactly how many app server we have in my case to handle 100 MB file size.
Thanks

Unable to process huge files in SFTP BASED PROXY SERVICE

Hi,
I credted osb project which will have sftp based proxy and sftp based business service.After completing the development i tested the fie transfer through WINSCP ,but my sftp based proxy was picking the files but i am unable to see the files in the output directory for huge files,for small file it is coming after aquite long time .please assist me .

If you enable content streaming then you cant access the file/message content within the Proxy service and can not perform actions like Replace or perform transformations on the content. Use streaming for pass through scenarios only. If you want to read a large file and also perform transformations on content I would recommend using JCA Adapter for FTP and try reading records from file in batches, a few records at a time. Just so we know the requirement better, what is the file size and what is the format of the content?

How to process huge files using File Adapter without ESR and no FCC?

Hi All,
I came to a scenario where i need to pick the file from R/3 sys which of around 500MB and need to transport it to the Third party sys using PI 7.1 . This is just simply a pass through data i.e.., no mapping is required (no ESR). and i don't to make use of FCC or BPM
My question is can the File Adapter can pick the file which is of 500MB size?
If not then what would be best solution to do it?
Do i need to divide the file into smaller sizes say around 100MB a file each in R/3 sys and then pick those files .
can i do without dividing the files in the R/3 sys?
Thanks in advance.
Sai

Hi Bhaskar,
If you zip probably the size would come less which may increase his performance a little. But I feel SAP also dont recommend this much load with PI.
If I am in his shoes then I will do a simple FTP without using PI itself. Else will look into other options. Thats my two cents.
Hope this alexs blog will give an idea to Sai:
/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover
Regards,
---Satish
Edited by: Satish Reddy on Jul 20, 2011 3:02 PM

Upload huge file size: "The page cannot be displayed" browser

What are precautions to be take for uploading a huge file
I have an upload file operation in the web application. The web server is OC4J.
//UploadForm.jsp
<FORM NAME="InputForm" ACTION="UploadAction.jsp" METHOD="POST" enctype=multipart/form-data>
<input type="file" name="fileName">
</FORM>
After I deploy the application to the web server, if I upload a small file size, it is fine.
But if i upload huge data ,data is uploading but if any action button is clicked
After 30 seconds, it has error "The page cannot be displayed" shown on the web browser.

Hi All,
I have a similar time out problem, it ll be great if some one provide a solution.
The problem I'm facing is becasue of the time out setting in the proxy server which is 1.5 mins. The app server is sending Response properly but after some 2 mins or so because the processing is a bit complicated (3 API calls done intermittently for different validations).
I cant request for change in proxy server time out setting. So I have to handle this with in my application.
It will be greatful if anyone helps us out of this.
Thanks
Noufal

Handle huge file size in file receiver

I have file(NfS) to file(Ftp) scenario the receiver file failing because of my huge data(nearly 20KB) i am using Message mapping functionality i cant use file check machanisam to split my file .could you share some ideas how can i achieve this requirement .
file ---> PI (7.4 java)--->File(Ftp) 20 to 25 KB)
how can i handle huge data in FIle receiver side ?

Hi Raman,
First you need to identify which step is taking more time. Adapter or transformation. Please share more details about your scenario.
If your scenario is pass through (with no transformation), then you can use the chunk mode. Please refer the below blog
File/FTP Adapter - Large File Transfer (Chunk Mode)
PI/XI: PI 7.3 processing of large files - teaser
If your file is having multiple recordset then you can pick the fix no of record set. Please refer the below blog
Night Mare-Processing huge files in SAP XI
regards,
Harish

OutOfMemory error on large file processing in sender file adapter

Hi Experts,
I got file to IDOC scenario, sender side we are using remote adapter engine, files are processing fine when the size is below 5mb, if the file size is more than 5mb we are getting java.lang.OutOfMemoryError: java heap space. can anyone suggest me what parameter i need to change in order to process more than 5mb.

Hi Praveen,
Suggestion from SAP is not to process huge files at a time. Instead, you can make the into chunks and process.
To increase the heap Memory:
For Java heap-size and other memory requirements, Refer to the following OSS note for details:
Note 723909 - Java VM settings for J2EE 6.40/7.0
Also check out the OSS Note 862405 - XI Configuration Options for Lowering Memory Requirements
There are oss notes available for the Java Heap size settings specific to the OS environment too.
Thanks,

HUGE file size for Pages (iWork '06) documents

I'm in the process of evaluating Pages in the hopes of dumping the last of my Microsoft programs (ie, Word.) I'm impressed with the ease of use; especially when it comes to being a little creative.
My great concern is the file size. I created a Pages document of 6 pages. It had text, images and tables. This relatively small document is a whopping 17.7MB. When I export this document to a Word doc and open it, it looks identical. The file size of the Word doc is only 1.4 MB. A HUGE difference. Is this a known issue or is there some sort of compression setting I'm missing?
I typically work with large docs and I don't want Pages files hogging all of my drive space.

The package shows image1.tif, which I added on the
first page. There's also an image1_filtered-9.tiff
which is HUGE. It's about 3MB larger than the
original file. It looks like Pages possibly makes a
copy of an image when you apply an effect like a drop
shadow?
So if you don't want huge files, I guess you have to
stay away from the image effects...which is actually
what made Pages interesting.
It is the adjustment tool which is causing the ballooning of your files.
If you use the levels adjustement, you will end up with an image which is labelled ~levelled.tiff. If you use the various other parts of the adjustment screen you will get files which are labelled ~-filtered.tiff, with or without version numbers.
There is no optimization of the various files as they are saved, and no user control over their generation. It would be much nicer if the adjustments weere done on the fly, or at least if a size-specific thumbnail were made for on-screen representation.
For now, however, if you are concerned with the size of your files in pages (and keynote, since it also happens there) then do the adjustments elsewhere. I suggest iPhoto, if that is where you have stored your items. You can always make your placed photo an image placeholder and replace it with alternative adjustments later.
(I tested this with a 4k file made for Livejournal, saved as a jpg. The resultant filtered and levelled tiffs were 40k. A considerable increase.)

Java.io.IOException during large file processing on PI 7.1

Hello Colleagues,
for a large file scenario on our PI 7.1 System we have to verify with big file size we are able to process over PI.
During handing over the large file (200 MB XML) form the Adapter Frame Work (File Adapter) to the Integration Engine we receive following error:
Transmitting the message to endpoint http://<host>:<port>/sap/xi/engine?type=entry using connection File_http://sap.com/xi/XI/System failed, due to: com.sap.engine.interfaces.messaging.api.exception.MessagingException: Error transmitting the message over HTTP. Reason: java.io.IOException: Error writing to server.
The message processing stopped and message still lies at Adapter Frame Work. Large files up to 100 MB we are able to process successfully.
Please, could you tell me why this happened and how we are able to solve it?
Because there is not a java.outofmemory exception however a IO exception i think it could be an memory issue?!
Many thanks in advance!
Regards,
Jochen

Hi Jochen,
Indeed the error is IO Error and it is because the Adapter engine was not able to send the message to Integration server. But it happens due to memory/heap size issues.
Look at these thread, they are having the same problem. Please try the remedy measures suggested by them
Mail to Proxy scenario with attachment. AF channel error.
Error with huge file
problem with big file in file-to-proxy scenario
Is there any additional information in Adapter messaging tool.?
Regards
Suraj
Edited by: S.R.Suraj on Oct 1, 2009 8:55 AM

Large file processing issue

Hi,
A 2MB source file is found to be generating a file of over 180 MB causing it to fail in pre prod and production. The processes successfully in Development Box where there is no web dispatcher or restrictions on size.
The recommendation from SAP is that we try to reduce the outout file size.
Kindly look into the issue ASAP.
Appreciate your help.
Thanks,
Satya Kumar

Hi Satya,
There are many ways are available check the below links
/people/stefan.grube/blog/2007/02/20/working-with-the-payloadzipbean-module-of-the-xi-adapter-framework
/people/aayush.dubey2/blog/2007/10/10/zip-transfer-unzip-increase-the-performance-of-your-java-abap-applications
/people/pooja.pandey/blog/2005/10/17/number-formatting-to-handle-large-numbers
/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover
/people/alessandro.guarneri/blog/2006/03/05/managing-bulky-flat-messages-with-sap-xi-tunneling-once-again--updated
One more way is we have to develope the ZIP Adapter and send the zip file after processing again we have to unzip the file.
Regards
Ramesh

Large File Processing Problem

HI Group,
I am facing problem in XI while processing 48 MB File through File adapter,I have used Content Conversion in the design.
I am using normal 64Bit operating system with Max of 2GB heap size,still I am facing the problem,Can any body tell me how much Heap size I required to process 48 MB size file through XI?

Hi,
Refer following SAP note-for this go to www.service.sap.com/notes
File adapter faq- 821267
Java Heap - 862405
for java settings- 722787
This blog may give some insight-/people/sap.user72/blog/2004/11/28/how-robust-is-sap-exchange-infrastructure-xi
/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi
btw, if the error tells that, trailer is missing.. where are you getting this error ?
Regards,
Moorthy

Maintain Server parameters to allow huge file size to pass through XI

Hello SAP Gurus,
i would like to know how can i check this Server parameters in SAP PI Configuration :
u2022 UME Parameters : To look into the pool size and poolmax wait parameters - UME recommended parameters (like: poolmaxsize=50, poolmaxwait=60000)
u2022 Tuning Parameters: To look/define the Message Size Limit u201Clike: EO_MSG_SIZE_LIMIT = 0000100u201D under tuning category
u2022 ICM Parameters: To consider ICM parameters (ex: icm/conn_timeout = 900000. icm/HTTP/max_request_size_KB = 2097152)
This is required for allowing huge file size to pass through XI.
Regards,
Ravi

Hi Michael,
I did the settings you just mentioned in the link.
I got through the First step where the file got processed and 50000 record data got divided into on 50000 idocs.
But when its is posting this in BW system, then the outbound status of the message is " Message has error status on outbound side"
And when i see BW there are no idocs in that system. Is it that the RFC connection between XI and BW had timed out and so it did not got processed at outbound side. Or there is some other setting required ?
Also, when we tried it with 20 K file then the outbound status is also successful.
Please let me know how to resolve this issue .
Regards,
Ravi

Splitting a huge file in chunks.

We have a requirement to split file containing 23 million records in chunks of 5000 records.I have used "Recordsets per message" option but it splits just 2.25 lakh records and from that records there will not be any idocs to be posted.Does anyone has idea on this?

Hi Sravya
Have a look at these threads,
Processing huge file loads through XI
Handling Large Files in XI
cheers
Sameer

Remote huge file load problem

---Following is the code to load a huge image file (50-100M)
InputStream in = new FileInputStream(readFile);
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
while((i = in.read(buffer)) != -1){
out.write(buffer, 0, i);
in.close();
byte[] result = out.toByteArray();
---It will crash (or hang) on toByteArray() when file size exceed 18mb.
Is there any way to load large files?
Thanks

1)I am new to the java forum. May I know what is
documentation for java BUT?http://java.sun.com/docs/index.html
2)Also may I know where I can find topics about
processing the file without loading the whole of it
into memory?
3) Actuall we are loading this file into bytearray in
java at client side(PC or MAC) and use rpc to
transfer to server(UNIX) - the C code in server side
will take this byte buffer and use fwrite to create
this huge file. Is there a better way than this?a) Break the file into smaller chunks and send the chunks together with a chunk identifer to the server. The server will then re-construct the file from the chunks.
b) Install an ftp server on the server machine and user FTP to sent the file.

Error message by periodic weekly: No output from the 1 file processed

Hi there,
since four weeks, I got a problem with the maintenance script periodic weekly. Up to December 22nd, the script did, what it should do: rebuilding the database of locate and whatis, rotating log-files. Since one week later, I got the error message: No output from the 1 file processed.
Normally, I use Anacron to do the job. When I noticed the problem, I tried to start the script with Tinker Tool System getting the same result. Another try using the Terminal (sudo periodic weekly) also failed. The commands locate and whatis are working, locate.updatedb and makewhatis also. I'm running 10.4.8; in the past, I did not have such problems. Anyone with an idea or solution?
Thanks
Klaus
MacBook Pro Mac OS X (10.4.8)

Hi Gary,
here is the output you were asking for:
Last login: Thu Jan 25 20:03:55 on console
Welcome to Darwin!
DeepThought:~ dirk$ sudo /private/etc/periodic/weekly/500.weekly; echo $?
Password:
Sorry, try again.
Password:
Rebuilding locate database:
Rebuilding whatis database:
find: /usr/local/man: No such file or directory
makewhatis: /usr/share/man/man1/fetchmailconf.1.gz: No such file or directory
Rotating log files: ftp.log lpr.log mail.log netinfo.log ipfw.log ppp.log secure.log
access_log error_log
Running weekly.local:
Rotating psync log files:/etc/weekly.local: line 17: syntax error near unexpected token `)'
/etc/weekly.local: line 17: `if [ -f /var/run/syslog.pid ]; then kill -HUP 0 80 79 81 0cat /var/run/syslog.pid | head -1); fi'
2
DeepThought:~ dirk$ ls -loe /private/etc/periodic/weekly/500.weekly
-r-xr-xr-x 1 root wheel - 2532 Jan 13 2006 /private/etc/periodic/weekly/500.weekly
DeepThought:~ dirk$
It seems, Rogers idea, PsynX respectively the deficient uninstalling by me is responsible for my problems, is correct. Should I remove the whole file weekly.local or should I only remove the content? I prefer removing the whole file, because it was created while installing PsyncX. The date of creation is the same as the date of installing the app (December 25).
Klaus
By the way: it seems to me, the solution of my problem is in sight. So I want to thank you all for the amazing aid I got from you!

Huge Files Processed in XI

Similar Messages

Maybe you are looking for