Large file processing in file adapter

Hi,
We are trying to process a large file of ~280 MB file size and we are getting timeout errors. I followed all the required tunings for memory and heap sizes and still the problem exists. I want to know if installation of decentral adapter engine just for this large file processing might solve the problem which I doubt.
Based on my personal experience there might be a limitation of file size processing in XI may upto 100 MB with minimul mapping and no BPM.
Any comments on this would be appreciated.
Thanks
Steve

Dear Steve,
This might help you,
Topic #3.42
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/70ada5ef-0201-0010-1f8b-c935e444b0ad#search=%22XI%20sizing%20guide%22
/people/sap.user72/blog/2004/11/28/how-robust-is-sap-exchange-infrastructure-xi
This sizing guide & the memory calculations it will be usefull for you to deal further on this issue.
http://help.sap.com/bp_bpmv130/Documentation/Planning/XISizingGuide.pdf#search=%22Message%20size%20in%20SAP%20XI%22
File Adpater: Size of your processed messages
Regards
Agasthuri Doss

Similar Messages

File processing in FIle adapter

Hi,
i have configured Sender File Adapter (FTP) in my scenario which picks the file from FTP folder and sends to JDBC.
now i have 2 flat files in my FTP (yesterdays and todays). file names are test_07062010 and test_08062010.
now my requirement is while processing the files, my file adapter has to process first file as test_07062010 and then test_08062010. but now it is processing first test_08062010 and then test_07062010. due to this yesterdays data is overwriting on todays data in Database.
i know we have some option in file sender adapter (NFS)...but i need to do with FTP file sender...please let me know how to proceed further.
Thanks in Advance.

Hi Praveen...thanks for reply
my communication channel is active all the days. but sometimes due to PI server down or some maintance we will stop channels.
that time file will not pick by file sender and it will be in FTP.then next day one more file will generate in FTP.
So my proble is while processing these files my sender file adapetr is processing todays file first and then yesterdays.due to this yesterdays data is overwriting on todays data.
can you tell me how to handle that in file adapter.

Alert if no file process :Same file adapter with same communication Channel

Hi XI Guru,
I have encountered 1 requirement from my client where, sender will send different data file with different file name. And there want XI to alert the user if no file to process for the schedule date. Please take note that the communication channel is common to all the sender.
Example:
Communication Channel : CC_FILE_SND
FILE_10.TXT ---> Every month on the 10, user have to send FILE_10.TXT to process
FILE_20.TXT ---> Every month on the 20, user have to send FILE_20.TXT to process
If user fail or forget to send the FILE_10.TXT and FILE_20.TXT on special schedule date to process, XI should alert user.
Many thanks

Hi Lord
<<< there want XI to alert the user if no file to process for the schedule date.
Can you tell me how you are planning to configure this schedule date in your sender file adapter. I mean this date will be fixed or it is going to be change according to the requirement.
Now for the second part in order to alert the user if file is not there, one solution is to use BPM with wait step ( specify the time in this step ), but then again it takes me back to my previous question, the time seems to be different everytime according to the requirement.

Row By Row File Processing using File Adapter

Hi,
Can any one explain that how can a read a file row by row using File adapter in BPEL?
Like ...When One row is executed completely only then second row starts........
Regards,
En
Edited by: Enceladus on Apr 16, 2013 11:55 AM

For This ...you need to make you service Synchronus by adding a reply in interface WSDL of BPEL process.
Regards,
En

File process - some files missing in processing

Hi,
I am using the below code to create drives on network.
This exe file is used by ILM as part batch job. The job will pick input files from local drive.
Query: While executing this job some input files are missing so for them drive is not created. Please let me know what could be the reason and how to resolve? let me know how to avoid file missing without processing?
System.Diagnostics.Process cmd = new Process();
cmd.StartInfo.UserName = "ad_user";
cmd.StartInfo.Domain = "domain";
cmd.StartInfo.Password = "password";
cmd.StartInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
cmd.StartInfo.FileName = @"d:\miis\createdrive.cmd";
cmd.StartInfo.WorkingDirectory = @"d:\miis\";
cmd.StartInfo.UseShellExecute = false;
string these_args = string.Format("{0} {1} {2}", domain, ocgsamaccountname, ocgpathtocreate);
cmd.StartInfo.Arguments = these_args;
try
Console.WriteLine(cmd.Start());
if (!cmd.WaitForExit(15000))
skip_delete = true;
jaya

Found the issue:
At first glance my itunes music folder looks normal, but after some careful reading of the location data...it turns out all of my music is also stored in a subfolder "MUSIC" in with all the artist names. I found this after discussing with a genius today while having my battery replaced. Together we found the issue.
How? iTunes at some point started making a MUSIC and MOVIES and maybe some other folders in the iTunes folder. When I redid my library some time ago, it must have changed the directory and copied all my music to the user/music/iTunes/iTunes Music/Music folder. This would also explain why I found that sometimes my library took up more data than I had expected. iTunes would say 45 GB but the folder would say 90GB.

File Process in File 2 Proxy in PI 7.1

Hi
I am working on a File to Proxy scenario where File will have multiple records, When I execute the scenario whole content of the file is coming as single record,
I want to process the each record at a time from the file, Can anyone let me know how to process the each single record at a time from the file
Thanks
PR

PR,
Try to read the file as record by record. Proabably there might be some error in reading the file with your content conversion. If you can give the content conversion parameters you used and one sample file somebody can help you out PR.
Else if you want to read the whole file as one record which you think is the correct process then you may need to write either java mapping or abap mapping.
Regards,
---Satish

File processing issue -File- PI- RFC

Dear Experts,
I have struck with this source file while processing the scenerio FTP(File) ->PI->RFC
I need to take the data from CSV file,
Just push the first row data to Char field and second row data to Mean field in RFC.
FCC structure:
The source data is simple but i struck with some logic,Kindly guide me how to process this data by using FCC or we can handle by mapping.Also check the source structure whether i followed correctly.
Best Regards,
Monikandan

Hi Amit,
Exactly i need to send the entire 1st row data to CHAR Description field and 2nd row data to mean field.
While i am testing in message mapping using the source file will getting the below output
Exact output i need is Process -001,Date row data will go to CHAR field and 10000,20-02-05 row data will go to MEAN field in RFC.
How to split it.kindly guide.
Best Regards,
Monikandan.

Process Multiple Files in PSE 7

I'm running PSE7.
I'd like to take a group of images (jpeg's) and do the following to them:
Run QuickFix on them.
Save them as a new file with the name based on the original name plus some extra information.
Resize the jpeg's to a size suitable for the web. Or possibly a little larger.
It would really be great if:
I could use a 'tag' to select the images I want to process.
The tags on the new images are the same as the originals.
The new files are written into the same folder as the original. Even if multiple folders are needed due to
the tags bringing in images from multiple folders.
The new images would become part of a version set or stack along with the originals.
Well, that's what's I'm interested in doing.
Any thoughts would be greatly appreciatted!

The Process Multiple Files command applies settings to a folder of files. If you have a digital camera or a scanner with a document feeder, you can also import and process multiple images. (Your scanner or digital camera may need an acquire plug‑in module that supports actions.)
When processing files, you can leave all the files open, close and save the changes to the original files, or save modified versions of the files to a new location (leaving the originals unchanged). If you are saving the processed files to a new location, you may want to create a new folder for the processed files before starting the batch.
Note: The Process Multiple Files command does not work on multiple page files.
Choose File > Process Multiple Files.
Choose the files to process from the Process Files From pop‑up menu:
Folder
Processes files in a folder you specify. Click Browse to locate and select the folder.
Import
Processes images from a digital camera or scanner.
Opened Files
Processes all open files.
Select Include All Subfolders if you want to process files in subdirectories of the specified folder.
For Destination, click Browse and select a folder location for the processed files.
If you chose Folder as the destination, specify a file-naming convention and select file compatibility options for the processed files:
For Rename Files, select elements from the pop‑up menus or enter text into the fields to be combined into the default names for all files. The fields let you change the order and formatting of the components of the filename. You must include at least one field that is unique for every file (for example, filename, serial number, or serial letter) to prevent files from overwriting each other. Starting Serial Number specifies the starting number for any serial number fields. If you select Serial Letter from the pop-up menu, serial letter fields always start with the letter “A” for the first file.
For Compatibility, choose Windows, Mac OS, and UNIX® to make filenames compatible with the Windows, Mac OS, and UNIX operating systems.
Under Image Size, select Resize Images if you want each processed file resized to a uniform size. Then type in a width and height for the photos, and choose an option from the Resolution menu. Select Constrain Proportions to keep the width and height proportional.
To apply an automatic adjustment to the images, select an option from the Quick Fix panel.
To attach a label to the images, choose an option from the Labels menu, then customize the text, text position, font, size, opacity, and color. (To change the text color, click the color swatch and choose a new color from the Color Picker.)
Select Log Errors That Result From Processing Files to record each error in a file without stopping the process. If errors are logged to a file, a message appears after processing. To review the error file, open with a text editor after the Batch command has run.
Click OK to process and save the files.

Process Multiple Files - Not Renaming RAW to JPG

Im trying to process a few RAW files and convert them to JPG.
The Process raw dialog is set up correct, but it seems after the first image is processed, I am prompted for a location to save the file, and its wanting to be saved as a .psd file and in the same location of the .RAW files...dispite the fact that I have a seperate destination location specified, and despite the fact that I have the settings to convert to >JPG Max Quality.
Any ideas why this may be happening? I am new to elements so im not sure if I am doing something wrong. I modified a RAW file and set my edits as the Camera Raw Default, and I just with to apply these settings to all my RAW files and convert to jpg.

There is a bit of a sequence to follow but this seems to work best.
To convert to jpeg select your raw files in Organizer and click on the Fix tab and choose Full Photo Edit. Wait for Camera Raw to open and click the button Select All (top left) then apply your saved default settings to all the selected images. Keep them all selected and click Open Images (towards bottom right - and select 8 bit as Barbara suggests before clicking that button) to send the selected images into the Editor.
After the Editor opens go to the top menu bar click File >Process Multiple Files.
Set the dropdown for your source to Opened Files then for the destination, check same as source.
At the bottom of the dialog under File Type choose the jpeg quality from the dropdown e.g. Max, High, Medium or Low. Choose your naming convention.
Click OK and leave the program to run (this can take a bit of time as raw files are usually large). When finished close the Editor and return to your folder to see your Jpegs next to your raw files. Or go to your new destination folder if preferred.

PSE6 for Mac - Process Multiple Files

I'm using PSE 6 for Mac on an iMac running Leopard. I shoot in RAW on a Canon DSLR and use Bridge and PSE6 for everything photographic as I can't stand iPhoto's hidden filing system which prevents me from being able to organise my files as I choose. I want to use "Process Multiple Files" to make jpg files for emailing and easy sharing. I'm following the on-screen dialogue box instructions to the letter (and those in Elements 6, the Missing Manual) and keep getting one or both of the following messages. Either "your source file is empty", which it plainly isn't, and/or "you must select a source/destination folder", which, according to the dialogue box, I have done. I really don't want to have to convert and 'save as' 30-40 files individually every time I've been out for a session, when there should be a perfectly simple way of doing it as a 'batch' action. Any thoughts, anyone?

Try using open files (File à Open) then select your batch of raw files using Shift+click or cmd+click to select individual files. Wait for Camera Raw to open and click the button Select All (top left) then apply your saved default settings to all the selected images. Keep them all selected and click Open Images (towards bottom right) to send the selected images into the Editor.
After the Editor opens go to the top menu bar click File >Process Multiple Files.
Set the dropdown for your source to Opened Files then for the destination, check same as source or a new folder.
At the bottom of the dialog under File Type choose the jpeg quality from the dropdown e.g. Max, High, Medium or Low.
Click OK and leave the program to run (this can take a bit of time as raw files are usually large).
When finished close the Editor and return to your folder to see your Jpegs next to your raw files. Or go to your new destination folder if preferred.

Multiple file processing

I have a large number of jpeg files that I would like to process at one time, applying the Enhance/Auto Smart Fix tool, and then saving and overwriting the original files in the original folder.
Is this possible? If so, how do I do it.
Thanks for help
Bill

bcselltime wrote:
I have a large number of jpeg files that I would like to process at one time, applying the Enhance/Auto Smart Fix tool, and then saving and overwriting the original files in the original folder.
Is this possible? If so, how do I do it.
Thanks for help
Bill
In Expert mode, use the menu 'File / process multiple files'.
You have the option to overwrite the original files (a very bad idea in my opinion) and you can specify which automatic enhancements you want.

Optimal workflow for batch processing LR files with Nik plug ins

I have a series of 40 or so landscape images that I would like to efficiently apply a couple of Nik filters to using Color Efex. I'd appreciate any suggestions re: the most efficient workflow to accomplish this task in a batch processing approach, as my current approach of taking each image into Photoshop, while preserving Smart Filter benefits, is terribly inefficient. Thanks in advance.

Hi,
Sorry but File->Process Multiple Files is the only way and functionalities with which you can batch process your images in PSE. However you can add action in Window->Action Player and apply to your images opened in workspace -
http://tv.adobe.com/watch/learn-photoshop-elements-11/adding-actions/
http://forums.adobe.com/message/4736578
The functionality is available in full functionality product Photoshop via File->Automate->Batch where you can load actions in action panel and they would appear in this dialog box but this is not available in Elements.
The best way to do here is to load actions in Window->Action Player and open all your images and click on play button one by one and then use Process Multiple Files to save all these in one go as per your preferences of file type, size, name etc.
Thanks,
Garry

Changing process.max-file-descriptor in non global zone

Hello Folks,
I have non global zone.
i wanted to change process.max-file-descriptor to 8192 so i issued the below command
projmod -s -K 'process.max-file-descriptor=(basic,8192,deny)' default
i have rebooted zone, after reboot system is not showing the value as 8192.
can u someone help me to find out the missed

# id -p
uid=0(root) gid=0(root) projid=1(user.root)
# prctl -P $$ | grep file
process.max-file-descriptor basic 256 - deny 19452
process.max-file-descriptor privileged 65536 - deny -
process.max-file-descriptor system 2147483647 max deny -
process.max-file-size privileged 9223372036854775807 max deny,signal=XFSZ -
process.max-file-size system 9223372036854775807 max deny -
# ulimit -n
256
# cat /etc/project | grep file
default:3::::process.max-file-descriptor=(basic,8192,deny)
#

Process Multiple Files to Reduce View Size within Safe Margin

I use Photoshop Elements and Premier Elements. I am looking for a way to multiple process 400 pictures to Premiere so photos are not cropped because of TV 4x3 image cropping some safe margin. In photoshop I could increase the canvas size to give a black boarder which gives the effect I want. The Photoshop File/Process Multiple Files... does not allow canvas size change. In Premier I can resize using effects for safe margin individually. I am looking for a procedure to multiple (bulk) process. I would even prefer some black letterboxing rather than losing the edges of the images by the TV and Premier cropping.
This must be a common need. - Thanks for any help.

I know of no way you can batch increase the canvas size of your photos, John. Sorry. It is not a feature of Process Multiple Files.
For what it's worth, the area that's lost around the edges of your video frame isn't due to any cropping by any of the programs. It's something your TV does, and it's called overscan.
But it varies greatly from TV to TV -- so one TV may cut off up to 10% while another may not cut off any of the edge of your video frame at all and you'd just be stuck with your black box around the outside of your video frame. Is that really what you want?
Also, CRTs (the old style of TV) tend to cut off much more of the frame edge than LCDs and plasma screens, which cut off almost none. So, depending on who you're doing this video for, it may not be worth the effort.

Process multiple files prompts for "save" at every image

Hi, I am trying to use Elements 10 to convert Panasonic RAW files to JPEGs for web use. The batch runs, but prompts for a "save" location at every file. I have reinstalled Elements, reinstalled the Camera RAW update, and have ensured I have ownership of source and destination folders (in Windows 7).
I have tried several folders, set up test folders etc. and nothing seems to work. Does anyone know what I am doing wrong? Its been ayear or so since I did this, but it used to work. TIA.

Try selecting all thumbnails in camera raw (left hand thumbnail strip) then click the open image button (bottom right) to open them in PSE Editor. When the images appear in the project bin use:
File >> Process Multiple Files
Use the source dropdown for Opened files
Choose a separate folder for destination
Then set your jpeg quality/size.

Large file processing in file adapter

Similar Messages

Maybe you are looking for