Handling large files with FTP in XI

Hi All,
I have a file scenario where i have to post the file with size of more than 500MB and with the fields upto 700 in each line.
The other scenario worked fine if the file size is below 70MB and less number of fields.
Could anyone help me in handling such scenario with large file size and without splitting the file.
1) From your previous experience did you use any Tools to help with the development of the FTP interfaces?
2) The client looked at ItemField, but is not willing to use it, due to the licensing costs. Did you use any self-made pre-processors?
3) Please let me know the good and bad experiences you had when using the XI File Adapter?
Thanks & Regards,
Raghuram Vedagiri.

500 MB is huge. XI will not be able to handle such a huge payload for sure.
Are you using XI as a mere FTP or are you using Content Conversion with mapping etc?
1. Either use a splitting logic to split the file outside XI ( using Scripts ) and then XI handles these file.
2. or Quick Size your hardware ( Java Heap etc ) to make sure that XI can handle this file ( not recommended though). SAP recommends a size of 5 MBas the optimum size.
Regards
Bhavesh

Similar Messages

Is it possible to upload large files through FTP to server with iWeb?

Is it possible to upload large files through FTP to server with iWeb like for example with Cyberduck?
I don't need to publish website with files, I just need to upload files on server. Can it be done somehow with iWeb?

Here's some info about FTP...
http://www.iwebformusicians.com/Search-Engine-Optimization/Upload.html
Make sure you are allowed to store files on your server. Most hosting services don't allow anything to be stored that can't be accessed via the internet. Check the fine print!

Handling large messages with MQ JMS sender adapter

Hi.
Im having trouble handling large messages with a MQ JMS sender adapter.
The messages are around 35-40MB.
Are there any settings I can ajust to make the communication channel work?
Error message is:
A channel error occurred. The detailed error (if any) : JMS error:MQJMS2002: failed to get message from MQ queue, Linked error:MQJE001: Completion Code 2, Reason 2010, Error Code:MQJMS2002
The communication channel works fine with small messages!
Im on SAP PI 7.11, MQ Driver is version 6.
Best Regards...
Peter

The problem solved itself, when the MQ server crashed and restarted.
I did find a note that might could have been useful:
Note 1258335 - Tuning the JMS service for large messages or many consumers
A relevant post as well: http://forums.sdn.sap.com/thread.jspa?threadID=1550399

Handling large files in scope of WSRP portlets

Hi there,
just wanted to ask if there are any best practices in respect to handling large file upload/download when using WSRP portlets (apart from by-passing WebCenter all-together for these use-cases, that is). We continue to get OutOfMemoryErrors and TimeoutExceptions as soon as the file being transfered becomes larger than a few hundred megabytes. The portlet is happily streaming the file as part of its javax.portlet.ResourceServingPortlet.serveResource(ResourceRequest, ResourceResponse) implementation, so the problem must somehow lie within WebCenter itself.
Thanks in advance,
Chris

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

Large file with fstream with WS6U2

I can't read large file (>2GB) with the stl fstream. Can anyone do this or is this a limitation of WS6U2 fstream classes. I can read large files with low level function C functions

I thought that WS6U2 meant Forte 6 Update 2. As to more information, the OS is SunOS 5.8, the file system is NFS mounted from an HP-UX 11.00 box and it's largefile enabled. my belief is that fstream does not implement access to large files, but I can't be sure.
Anyway, I'm not sure by what you mean by the access to the OS support for largefiles by the compiler, but as I mentioned before, I can read more then 2GB with open() and read(). My problem is with fstream. My belief is that fstream must be largefile enabled. Any idea?

Reading large file with JCA Adapter in OSB

Hello,
We are searching for a solution how to read large file (>50M) from network drive and deliver it to queue via OSB 11gR4 (10.3.4). The problem is when reading the file with JCA File Adapter. It seems that it cannot handle as large files as we have. The documentation provides a way to bypass file size limitation by using Chunk Read but it seems to require BPEL Process execution which is not possible in our environment. Does anyone know if there are ways to implement this without having BPEL Process?
Our usecase:
read file from network drive -> transfer with OSB -> deliver MQ
Other options than JCA File Adapter can be considered, if anyone can advice...

If it's a plain routing use case and no message processing is required then you may simply use OSB's FILE transport instead of JCA adapter. Create a messaging type proxy service and select request message type as "binary". Also enable the content streaming (Disk buffer, compression).
From OSB Dev guide -
Oracle JCA Adapter for FTP and Files – Attachments (large payload support), pre- and post-processing of files, using a re-entrant valve for processing ZIP files, content streaming, and file chunked read are not supported with Oracle Service Bus.
http://download.oracle.com/docs/cd/E17904_01/doc.1111/e15866/jca.htm#BABBICIA
You may also refer -
Reading huge flat file in OSB 11gR1
Regards,
Anuj

How can I set-up to receive a file with FTP?

I want to receive a .pdf file from another, but the file is rejected by my email service provider for size. What are the requirements and steps for arranging to receive this large file using File Transfer Protocol (FTP)? I'm using OS Snow Leopard 10.6.8 with a DSL Internet connection nominally at 10 MBs download.

Daniel Lichtenwald wrote:
What are the requirements and steps for arranging to receive this large file using File Transfer Protocol (FTP)?
Usually, we don't speak of "receive" when using FTP, since the file is transferred from server to client, so it's more of a case of "download".
At your end, it's simple. You use an FTP client; under SL, that includes Finder and Safari, so you don't even need to get any additional software.
At the other end, it's more complicated; the 'sender' must set up an FTP server on his machine.
Alternatively, you can set up your own Mac as an FTP server, and have the 'sender' connect to you with an FTP client and upload the file; but, if your Mac lives behind a router, then you have more work to do with the router settings.
That's why it's much easier to use the file sharing services mentioned above -- if they are available in both sender's and receiver's locations. (Keep in mind that some countries block access to all those mentioned -- except perhaps <www.transfer.ro>, of which I know absolutely nothing.)

How to handle flat file with variable delimiters in the file sender adapter

Hi friends,
I have some flat files in the ftp server and hope to poll them into XI, but before processing in XI, I hope to do some content conversion in the file sender adapter, according to the general solution, I just need to specify the field names, field seperator, end seperator, etc. But the questions is:
The fileds in the test data may have different amount of delimiters (,), for example:
ORD01,,,Z4XS,6100001746,,,,,2,1
OBJ01,,,,,,,,,,4,3
Some fileds only have 1 ',' as the delimiter, but some of them have multiple ','.
How can I handle it in the content conversion?
Regards,
Bean

Hi Bing,
Plz do refer the following blogs u will get an idea:
File content conversion Blogs
/people/venkat.donela/blog/2005/03/02/introduction-to-simplefile-xi-filescenario-and-complete-walk-through-for-starterspart1
/people/venkat.donela/blog/2005/03/03/introduction-to-simple-file-xi-filescenario-and-complete-walk-through-for-starterspart2
/people/arpit.seth/blog/2005/06/02/file-receiver-with-content-conversion
/people/anish.abraham2/blog/2005/06/08/content-conversion-patternrandom-content-in-input-file
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
/people/venkat.donela/blog/2005/03/02/introduction-to-simplefile-xi-filescenario-and-complete-walk-through-for-starterspart1
/people/venkat.donela/blog/2005/03/03/introduction-to-simple-file-xi-filescenario-and-complete-walk-through-for-starterspart2
/people/venkat.donela/blog/2005/06/08/how-to-send-a-flat-file-with-various-field-lengths-and-variable-substructures-to-xi-30
/people/anish.abraham2/blog/2005/06/08/content-conversion-patternrandom-content-in-input-file
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
/people/jeyakumar.muthu2/blog/2005/11/29/file-content-conversion-for-unequal-number-of-columns
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/michal.krawczyk2/blog/2004/12/15/how-to-send-a-flat-file-with-fixed-lengths-to-xi-30-using-a-central-file-adapter
/people/arpit.seth/blog/2005/06/02/file-receiver-with-content-conversion
http://help.sap.com/saphelp_nw04/helpdata/en/d2/bab440c97f3716e10000000a155106/content.htm
Regards,
Vinod.

Can´t print large files with 10.5.7

Hi,
we finally have updated our G5s, 2x2GHz, 4GB RAM, 500GB system disk. Since then we are unable to print documents from CS3 + CS4 which result in cups print files > 1 Gb. Doesn´t work with GMG rips, HP Laserwriters and Adobe PDF printers. The jobs always show up as "stopped" in the queue. Error log of cups says:
I [11/Jun/2009:16:42:09 +0200] [Job 55] Adding start banner page "none".
I [11/Jun/2009:16:42:09 +0200] [Job 55] Adding end banner page "none".
I [11/Jun/2009:16:42:09 +0200] [Job 55] File of type application/pictwps queued by "dididi".
I [11/Jun/2009:16:42:09 +0200] [Job 55] Queued on "EpsonISOcoated_39L02" by "dididi".
I [11/Jun/2009:16:42:09 +0200] [Job 55] Started filter /usr/libexec/cups/filter/pictwpstops (PID 1363)
I [11/Jun/2009:16:42:09 +0200] [Job 55] Started backend /usr/libexec/cups/backend/pap (PID 1364)
E [11/Jun/2009:16:42:10 +0200] [Job 55] can't open PSStream for stdout
E [11/Jun/2009:16:42:10 +0200] [Job 55] pictwpstops - got an error closing the PSStream = -50
E [11/Jun/2009:16:42:10 +0200] PID 1363 (/usr/libexec/cups/filter/pictwpstops) stopped with status 1!
I [11/Jun/2009:16:42:10 +0200] Hint: Try setting the LogLevel to "debug" to find out more.
E [11/Jun/2009:16:42:12 +0200] [Job 55] Job stopped due to filter errors.
We have tested a catalog file with 150 pages which didn´t print. If you broke it up into 6 pieces (print page 1-25, 26-50...) it did work. Unfortunally we have jobs with single pages exceeding this limit. All together we have tested around 10 differend files.
We have tried to reset the printer queue, installed the drivers and deleted and created all printers, but nothing helps. Looks like cups or one of it´s helper can´t handle files larger than 1 GB?
Any help is welcome.

Hi there,
Does you printer/RIP support protocols other than AppleTalk? If it does, then I would change it to something like HP Jetdirect-Socket.
PaHu

Handling Large files in PI scenarios?

Hello,
We have lot of scenarios (almost 50) where we deal with file interfaces atleast in receiver or sender side. Some of them are just file transfers where we use AAE and some are where we have to do message mapping (sometimes very complex ones).
the interfaces work perfectly fine will a normal file which dont have much records but recently we started testing big files with over 1000 records and its taking a lot of time to process. It is also causing other messages which gets lined up in the same queue to wait in the queue for the amount of time it takes for the first message to process.
This must be a very practical scenario where PI has to process large files specially files coming from banks. What is the best way to handle its processing? Apart from having a better system hardware (we are currently in the test environment. Production environment will definetely be better) is there any technique which might help us improve the processing of large files without data loss and without interrupting other message?
Thanks,
Yash

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

Hardware Question – Handling large files in Photoshop

I'm working with some big TIFF files (~1GB) for large-scale hi-res printing (60" x 90", 10718 x 14451), and my system is lagging hard like never before (Retina MacBook Pro 2012 2.6GHz i7 /8 GB RAM/ 512GB HD).
So far I've tried:
1) converting to .psd and .psb
2) changing the scratch disk to an external Thunderbolt SSD
3) allocating all available memory to the program within photoshop preferences
4) closing all other applications
In general I'm being told that I don't have enough RAM. So what are the minimum recommended system requirements to handle this file size more comfortably? Newest Retina Pro with 16GB RAM? Or switch to iMac w/ 32? Mac Pro?
Thanks so much!

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

Large Files from FTP to Azure Storage Account.

We required to transfer files from FTP to Azure Storage Account.
Our FTP contains Almost 4 TB data and in future it will be growing.
We require some automate process which will transfer data from FTP to Windows Azure Storage.
Currently Implemented Solution: We have created
a windows service which will download files from FTP and converting to file stream and with Blob Storage object it will be uploaded to
Storage account.
With the mentioned approach bandwidth is utilized at 2 places
For downloading file from FTP
For Uploading to Azure Storage
Can you please provide any solution/suggestion which will reduce the consumption of bandwidth and large data will transfer in less time

Hi,
Please have a look at below article, it is talking about Blob Transfer Utility tool. Blob Transfer Utility is a GUI tool to upload and download thousands of small/large files to/from Windows Azure Blob Storage.
#https://blobtransferutility.codeplex.com/
Disclaimer: This is not a Microsoft official tool and it is not supported by Microsoft. It's just a (very useful) sample.
Best Regards,
Jambor
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Put File of a large file through FTP not working correctly

I put a lot of large files to my web server through Dreamweaver, and when I get files larger than about 45 meg, the file is put fine to the server. But then Dreamweaver waits for a server responce to make sure it got the file. and after about 30 seconds or so, it says server timed out, and it starts over with the FTP, and the first thing it does on the start over is to delete the file.
The only wai I have been able to put my large files to the server, is to sit and wait for it to finish, and then cancel the check for if the file is there or not.
I have my FVT server timeout set to 300 seconds, but that does not help eihter. it still times out after about 30 seconds.
Is there a fix for this so I don't have to sit and watch my files being transfered to the server?

I changed it to passive PTF, but that did not help, sending a 90 Meg file did the same thing. What the problem is, is that the Dreamweaver FTP wants to verify that the file got there. and when you send that large of a file, the server wants to make sure it is a clean file, so it takes a while to put it to the correct location. And while the server is doing this, Dreamweaver says that there is no responce from the server, and causes a reconnect. and on the reconnect it starts over, and the first thing it does is to make sure the file is deleted. What it should do it verify that the file is there and the same date as the one you sent up there. That would eliminate this problem. I was hoping there was a setting in Dreamweaver that I could tell the FTP not to verify the send.
Are there any Dreamweaver developers that visit this forum. that could tell me how to bypas the verify on put. It seams silly to have a tool that costs this much that can not do a task that a free pice of software can do.

Handling Large File

Hi all,
We need to handle a large file 88omb .is there any provision at the adapter level to break the file in to smaller chunks.
Can we avoid using the shell scripts ansd OS level commands.
Thanks,
Srinivas

Hi Srinivas,
if it is a text file then you could break up the file into multiple recordsets,
e.g.,
[Converting Text Format in the Sender File/FTP Adapter to XML |http://help.sap.com/saphelp_nwpi711/helpdata/en/44/658ac3344a4de0e10000000a1553f7/frameset.htm]
and
[#821267 File Adapter FAQ|http://service.sap.com/sap/support/notes/821267]
14. Memory Requirements
Regards
Kenny

[iPhone] Can't download file with FTP

Dear everybody.
I want to download a text file at : ftp://username:[email protected]
Current, I use below code:
NSString *s = [NSString stringWithContentsOfURL:[NSURL URLWithString:@"ftp://username:[email protected]"] encoding:NSUTF8StringEncoding error:pError];
and I try:
NSData *tmpdata = [NSData dataWithContentsOfURL:[NSURL URLWithString:SERVER_NAME]];
But I can't download content of text.
I think NSURL can not get content from ftp protocol
So, please help me download file use FTP.
Thank you very much!

Yes, if I try access with specify the file name as:
@"ftp://username:[email protected]/file.txt"
then I can get content of file.
But I want list and show all file in path: @"ftp://username:[email protected]".
Pls help me

Handling large files with FTP in XI

Similar Messages

Maybe you are looking for