How to process large input CSV file with File adapter

Hi,
could someone recommend me the right BPEL way to process the large input CSV file (4MB or more with at least 5000 rows) with File Adapter?
My idea is to receive data from file (poll the UX directory for new input file), transform it and then export to one output CSV file (input for other system).
I developed my process that consists of:
- File adapter partnerlink for read data
- Receive activity with checked box to create instance
- Transform activity
- Invoke activity for writing to output CSV.
I tried this with small input file and everything was OK, but now when I try to use the complete input file, the process doesn't start and automatically goes to OFF state in BPEL console.
Could I use the MaxTransactionSize parameter as in DB adapter, should I batch the input file or other way could help me?
Any hint from you? I've to solve this problem till this thursday.
Thanks,
Milan K.

This is a known issue. Martin Kleinman has posted several issues on the forum here, with a similar scenario using ESB. This can only be solved by completely tuning the BPEL application itself, and throwing in big hardware.
Also switching to the latest 10.1.3.3 version of the SOA Suite (assuming you didn't already) will show some improvements.
HTH,
Bas

Similar Messages

How to Create an Input Schedule Comparison with Data Using EVDRE

Hello,
I try implement the scenario described in "How to Create an Input Schedule Comparison with Data Using EVDRE".
Once I am using the "Insert Function" from the panel and selecting the EVTIM function Excel crashes (see page 8 How to paper).
Systems:
BPC 7.0 NW SP02
Office 2007
BPCADminClient and BPCOfficeClient up to date
Have anyone a solution?
Thanks
Oktay

Hi Oktay -
This function works in my BPC70NW SP02 system. Your issue might be that you are trying to access a TIME member that does not exist. Please make sure the offset value is a valid dimension member.
I can confirm that EVTIM does allow the offset for base members (such as 2009.MAY) as well as parent nodes (such as 2009.Q1 or 2008.TOTAL)... BUT...the offset result of the EVTIM function needs to be a valid dimension member!
Regards,
Sheldon

SQL Developper Version 4.1.0.19: cannot open sql files with file type .pck

Hi
I can't open sql files with file type .pck. They are opened as a package. The icon is a package too. (Worked fine in 4.0)
My settings:
Thanks for any help

Thanks, but that doesn't help
To specify it more precisly: I can open such a file, but it is not opened as a sql file. The icons you see above left are the icons of the files (not of the packages).
Greetings
Ovi

How to process large data files in XI ? 100 MB files ?

Hi All
At present we have a scenario as follows
It is File to IDoc ....Problem is the size of the file
We need to transfer 100mb file to SAP R/3 system ? So this huge data how to
process ?
Adv thanx and regards
Rakesh

Hi,
In general, an extra sizing for XI memory consumption is not required. The total memory of the SAP Web Application Server should be sufficient except in the case of large messages (>1MB).
To determine the memory consumption for processing large messages, you can use the following rules of thumb:
Allocate 3 MB per process (for example, the number of parallel messages per second may be an indicator)
Allocate 4 kB per 1kB of message size in the asynchronous case or 9 kB per 1kB message size in the synchronous case
Example: asynchronous concurrent processing of 10 messages with a size of 1MB requires 70 MB of memory
(3MB + 4 * 1MB) * 10 = 70 MB With mapping or content-based routing where an internal representation of the message payload may be necessary, the memory requirements can be much higher (possibly exceeding 20 kBytes per 1kByte
message, depending on the type of mapping).
The size of the largest message thus depends mainly on the size of the available main memory. On a normal 32Bit operating system, there is an upper boundary of approximately 1.5 to 2 GByte per process, limiting the respective largest message size.
please check these links..
/community [original link is broken]:///people/michal.krawczyk2/blog/2006/06/08/xi-timeouts-timeouts-timeouts
Input Flat File Size Determination
/people/shabarish.vijayakumar/blog/2006/04/03/xi-in-the-role-of-a-ftp
data packet size - load from flat file
How to upload a file of very huge size on to server.
Please let me know , your problem is solved or not..
Regards
Chilla..

How to process Large Image Files (JP2 220MB+)?

All,
I'm relatively new to Java Advanced Imaging, so I need a little help. I've been working on a thesis that involves converting digital terrain data into X3D scenes for future use in military training and applications. Part of this work involves processing large imagery data to texture the previously mentioned terrain data. I have an image slicer that can handle rather large files (200MB+ jpeg files). But it can't seem to process jpeg 2000 data. Below is an excerpt from my code.
public void testSlicer(){
String fname = "file.jp2";
Iterator readers = ImageIO.getImageReadersByFormatName("jpeg2000");
ImageReader imageReader = (ImageReader) readers.next();
try {
ImageInputStream imageInputStream = ImageIO.createImageInputStream(new File(fname));
imageReader.setInput(imageInputStream, true);
} catch (IOException ex) {
System.out.println("Error: " + ex);
ImageReadParam imageReadParam = imageReader.getDefaultReadParam();
BufferedImage destBImage = new BufferedImage(256, 256, BufferedImage.TYPE_INT_RGB);
Rectangle rect = new Rectangle(0, 0, 1000, 1000);
//Only reading a portion of the file
imageReadParam.setSourceRegion(rect);
//Used to subsampling every 4th pixel
imageReadParam.setSourceSubsampling(4, 4, 0, 0);
try {
destBImage = imageReader.read(0, imageReadParam);
} catch (IOException ex) {
System.out.println("IO Exception: " + ex);
The images I am trying to read are in excess of 30000 pixels by 30000 pixels (15m resolution at 5 degrees latitude and 6 degrees longitude). I continually get an OutOfMemoryError, though I am pumping up the heap size to 16000MB when using the command line.
Any help would be greatly appreciated.

Hi,
In general, an extra sizing for XI memory consumption is not required. The total memory of the SAP Web Application Server should be sufficient except in the case of large messages (>1MB).
To determine the memory consumption for processing large messages, you can use the following rules of thumb:
Allocate 3 MB per process (for example, the number of parallel messages per second may be an indicator)
Allocate 4 kB per 1kB of message size in the asynchronous case or 9 kB per 1kB message size in the synchronous case
Example: asynchronous concurrent processing of 10 messages with a size of 1MB requires 70 MB of memory
(3MB + 4 * 1MB) * 10 = 70 MB With mapping or content-based routing where an internal representation of the message payload may be necessary, the memory requirements can be much higher (possibly exceeding 20 kBytes per 1kByte
message, depending on the type of mapping).
The size of the largest message thus depends mainly on the size of the available main memory. On a normal 32Bit operating system, there is an upper boundary of approximately 1.5 to 2 GByte per process, limiting the respective largest message size.
please check these links..
/community [original link is broken]:///people/michal.krawczyk2/blog/2006/06/08/xi-timeouts-timeouts-timeouts
Input Flat File Size Determination
/people/shabarish.vijayakumar/blog/2006/04/03/xi-in-the-role-of-a-ftp
data packet size - load from flat file
How to upload a file of very huge size on to server.
Please let me know , your problem is solved or not..
Regards
Chilla..

Reading large file with JCA Adapter in OSB

Hello,
We are searching for a solution how to read large file (>50M) from network drive and deliver it to queue via OSB 11gR4 (10.3.4). The problem is when reading the file with JCA File Adapter. It seems that it cannot handle as large files as we have. The documentation provides a way to bypass file size limitation by using Chunk Read but it seems to require BPEL Process execution which is not possible in our environment. Does anyone know if there are ways to implement this without having BPEL Process?
Our usecase:
read file from network drive -> transfer with OSB -> deliver MQ
Other options than JCA File Adapter can be considered, if anyone can advice...

If it's a plain routing use case and no message processing is required then you may simply use OSB's FILE transport instead of JCA adapter. Create a messaging type proxy service and select request message type as "binary". Also enable the content streaming (Disk buffer, compression).
From OSB Dev guide -
Oracle JCA Adapter for FTP and Files – Attachments (large payload support), pre- and post-processing of files, using a re-entrant valve for processing ZIP files, content streaming, and file chunked read are not supported with Oracle Service Bus.
http://download.oracle.com/docs/cd/E17904_01/doc.1111/e15866/jca.htm#BABBICIA
You may also refer -
Reading huge flat file in OSB 11gR1
Regards,
Anuj

How to zip a folder containing subfolders with files and download the zip file in silverlght 4

private void hbtnDownloadReceipt_Click(object sender, RoutedEventArgs e)
try
string Docpath = "ClaimsDetails/" + "20tech" + "/" + PaymentAdviceUNo;
string dd = "ClaimsDetails/";
Uri uri = new Uri(Application.Current.Host.Source, "/" + Docpath);
Uri uri1 = new Uri(Application.Current.Host.Source, "/" + dd);
string path = uri.AbsoluteUri.ToString();
path1 = uri1.AbsoluteUri.ToString();
//HtmlPage.Window.Eval("window.open('" + path + "')");
catch (Exception ex)
ex.Data.Clear();
1) Here "path" is D:\\TL 24-12-2014 \\OfficeConnect_17_10\\OfficeConnect.Web\\ClaimsDetails\\20tech\\PaymentAdviceUNo.
2) In path "PaymentAdviceUNo" is the folder containing subfolders which i want to zip.
Thanks in Advance.

Hi,
You can download a zip file like any files with the Webclient class, look at the details and examples in the msdn documentation for downloading content on demand it even talks about how to download and get a specific file from a zip archive.
http://msdn.microsoft.com/en-us/library/cc189021(VS.95).aspx
Besides, if you want to list the files,you could check articles below:
http://blogs.msdn.com/b/blemmon/archive/2009/11/25/reading-zip-files-from-silverlight.aspx
Best Regards,
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

How to process overly compressed high volume audio files

I used a process in scientific filters in audition 3 to process overly compressed high volume audio files normalized to 0 db. It was like magic.
How would you process this sort of file in Audition 6.

Below was the process used in Adobe Audition 3. The sound was much improved. Try the steps on a file you have to hear the difference.
Effects- Amplitude - Normalize - uncheck the decibles format box - normalize to and normalize L/R boxes checked, enter 48 into percentage box.
After normalization was complete
Effects - Filters - Scientific filters - select Chebychev 1 tab - band pass button and select the following parameters.
Cutoff =4, high cutoff = 22050, order = 18 all other settings at defaults values (0). Delay radio button sellected, check the extended range box, press OK.
Increase to appropriate volume after processing if necessary.

Importing text file (with file names) into Automator.. is it possible?

Hello all,
I have been working with Windows Batch files for my line of work. I have a couple of file names in a text file (a column), which I want to copy from one folder of one hdd to another folder on a different hdd. I have been trying to do this kind of work with a Mac. I already know how you copy and rename files in automator (which isn't difficult, of course) but you have to 'select' the files in the finder first (with get specified items).
But the only way i see that you can specify items is by selecting them... is there a way to import a text file with all the file names instead of selecting all the file names manually?
or is there an AppleScript alternative which I can use to import the text file (or just copy into applescript) and run before the query's of copying and renaming the files? I am kind of new to Apple programming.
The text file looks like this:
image1.jpg
image2.jpg
etc..
so there has to be a command to: 'goto' a specific folder as well.
Thanks in advance!

You can import text files, but if they are just names you will need an additional action to add the source folder path. A *Run AppleScript* action can be used, for example:
Tested workflow:
1) *Ask for Finder Items* {Type: files } -- choose the text file containing the names
2) *Combine Text Files* -- this gets the text file contents
3) *Filter Paragraphs* { return paragraphs that are not empty } -- skip blank lines
4) *Run AppleScript* -- copy and paste the following script:
<pre style="
font-family: Monaco, 'Courier New', Courier, monospace;
font-size: 10px;
font-weight: normal;
margin: 0px;
padding: 5px;
border: 1px solid #000000;
width: 680; height: 340px;
color: #000000;
background-color: #FFEE80;
overflow: auto;"
title="this text can be pasted into an Automator 'Run AppleScript' action">
on run {input, parameters} -- add folder path
add the specified folder path to a list of file names
input: a list of text items (the file names)
output: a list of file paths (aliases)
set output to {}
set SkippedItems to {} -- this will be a list of skipped items (errors)
set SourceFolder to (choose folder with prompt "Choose the folder containing the file names") as text -- this is the folder containing the names
repeat with AnItem in the input -- step through each name in the input
try
set AnItem to SourceFolder & AnItem -- add the prefix
set the end of the output to (AnItem as alias) -- test
on error number ErrorNumber -- oops
set ErrorNumber to (" (" & ErrorNumber as text) & ")" -- add the specific error number
set the end of SkippedItems to (AnItem as text) & ErrorNumber
end try
end repeat
ShowSkippedAlert for SkippedItems
return the output -- pass the result(s) to the next action
end run
to ShowSkippedAlert for SkippedItems
show an alert dialog for any items skipped, with the option to cancel the workflow
parameters - SkippedItems [list]: the items skipped
returns nothing
if SkippedItems is not {} then
set {AlertText, TheCount} to {"Error with AppleScript action", count SkippedItems}
if TheCount is greater than 1 then
set theMessage to (TheCount as text) & space & " items were skipped:"
else
set theMessage to "1 " & " item was skipped:"
end if
set {TempTID, AppleScript's text item delimiters} to {AppleScript's text item delimiters, return}
set {SkippedItems, AppleScript's text item delimiters} to {SkippedItems as text, TempTID}
if button returned of (display alert AlertText message (theMessage & return & SkippedItems) ¬
alternate button "Cancel" default button "OK") is "Cancel" then error number -128
end if
return
end ShowSkippedAlert
</pre>
5) *Copy Finder Items* { To: _your external drive_ }

Pick up a specific file with File Sender Adapter.

Hi guys,
I would like to know how I can pick a specific file in a file pool (folder)? I would like to choose this file by name, like FileA or FileB, etc.
Im asking this because I have an asynchronous file scenario (BPM) with a receiver adapter that put the File with a specific name (variable substitution) in a folder.
And I would like to do something like this:
In another asynchronous scenario (BPM), a File sender adapter picks up this specific file (using the name). The correlation is made trough an IDOC that XI receives before pick up the file, this IDOC has payload field with the name of file to be picked.
Is it possible receives the IDOC, read the field with the name of the file to be picked and choose this specific file? In a Sender File Adapter how I can do something like variable substitution like receiver adapter does.
Thanks in advance,
Ricardo.

Hi,
<i>Is it possible receives the IDOC, read the field with the name of the file to be picked and choose this specific file? In a Sender File Adapter how I can do something like variable substitution like receiver adapter does.</i>
No this is not possible. The only dynamic thing you can do is use wild card characters like *.
So, maybe you can pick a file like . or AA. and so on...
Regards,
Bhavesh

One to many xml files with file adaptor

Hi,
I have a scenario HCM-ABAPProxy--XI-File for one structure I need to generate multiple xml files with 100 records per file.
this is my input messag
MT_in
Node
     PositionIDs
        descrption
        job
        IsActive
what I was doing on the ABAP side for every node I have 100 PositionID's sub nodes. so each node should be a sperate file.
my output structure is
PositionIDs
    PositionID
      pid
      description
      job
so there should be one PositionIDs per file which contains 100 PositionID.
I've multi message mapping without BPM that did not work out just wondering if any one came across the same scenario.
thanks,
Joe

You can not create multiple files without BPM. You can pretty much perform multi mapping to achieve your split but to write it to a file, you will have to call the file adapter for each split which you can not do without using BPM. In BPM, for each split that you perform, you can use a send step in for-each loop which will give you the functionality you require.
Award if helpful,
Sarath.

Reading fixed position files with File Adapter

Hi !
I'm trying to use the file adapter to read a file with data in fixed positions .
I cannot get it to work, I'm getting :
[Line=3, Col=133] Expected "${eol}" at the specified position in the native data, while trying to read the data for "element with name Hours", using "style" as "array" and "cellSeparatedBy" as "${eol}", but not found.
Ensure that "${eol}", exists at the specified position in the native data.
and don't understand what the problem is.

Sorry I didn't reply sooner.
If you are only interested in the first 133 characters then as you stated you need to put in a dummy filler at the end.
The file adapter is not as smart as you think. What is does is reads the string in a stream, so it gets its command and reads the stream until it hits it, then it gets the next command.
In your situation you said read 1 - 133 thinking it would ignore the other characters in the line. In reality what happens is that it reads the first 133 characters. Assuming that it wasn't looking for a end of line it would then read the next 133 characters. This would include the character you want to ignore, so your pattern would fail.
Good way to think about it is to apply the rules as if you are reading the file your self. In your mind you say read these characters then I want to ignore all the others in the row. You have to tell the file adapter to do the same.
Hope this explains.
cheers
James

Idoc to file with file name as ABCD timestamp and count

Hi,
Greetings!
I have an requirement like iDoc to File and the output file name should be ABCDYYMMDDXX (e.g ABCD14091701) in PI 7.1
where,
Default = ABCD
YY = year
MM = month
DD = date
XX = sequence no. on the same day; e.g. first batch on same day = 01, second batch on the dame day = 02
I tried checking many blogs but its not meeting this requirement.
Kindly please help me out in completing this interface.
Regards,
Vinoth

Hi Vinoth,
Below is the code. Use it in-conjunction with your dynamic config code.
String inputFile = "D:\\Receiver"; //This is your target receiver folder
String dateStr = DateFormat.getDateInstance().format(new Date());
dateStr = dateStr.substring(0, 2)+ dateStr.substring(3, 5)+ dateStr.substring(8,10);
final String fileNamePattern = "DEFAULT"+dateStr;
File file = new File(inputFile);
File[] listFiles = file.listFiles(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
if(name.contains(fileNamePattern))
return true;
return false;
//System.out.println("Number of files in the directory "+inputFile+" : Next Counter"+ (listFiles.length+1));
int counter = listFiles.length+1;
String fileName = "";
if(counter < 10)
fileName = fileNamePattern+0+counter;
else
fileName = fileNamePattern+counter;
System.out.println("Name of the file will be:"+fileName);
Hope it helps!
Best Regards,
Anand Patil

How to ignore columns in CSV File with File Sender Adapter

Hi,
I have a CSV File that I need to load with XI. The File contains 10 columns but I only need the data from 3 columns, let's say column 1,4 and 7. Can I configure the XI File Sender Adapter in a way that it only loads the data for the required columns and ignores the rest?
Thanks in advance.
Alex

Alex,
Dont think so. Why not create Dummy fieldName for these columns you want to ignore.?
Regards
Bhavesh

How to control (the input and output) EXE file after I call it using exec?

Hi,
I knew that I can use runtime.exec() to call one EXE file, and this works. But this EXE has two characteristics:
1. After this exe starts, it asks user to input number such as 1 or 2 onto computer screen, then press return. Then the exe will start the calculation.
2. after it starts calculation, it prints 3 columns of numbers onto the screen.
My two questions are:
1. How to use java to input the number such as 1 or 2 automatically? this EXE can not work like this in DOS command line:
C:> file.exe parameter
The parameter is the number such as 1 or 2 that I wanna input.
2. how to redirect the 3 columns of numbers from computer screen to txt file?
My colleague can solve these two questions using Mathematica. So I know that definitely there is at least one solution for it. I just can not do it using Java. This wierd exe file bothered me a lot and I really wish that I can get help from someone in java community.
Thank you!
Tony

When you call Runtime.exec, you get a Process object. (I presume something similar happens when you use ProcessBuilder.) Process has methods with names getOutput, getInput, and getError. These correspond to the standard input, standard output, and standard error streams of the spawned process.
You can read and write to the process on the streams corresponding to input and output that the process writes to the console.
[add]
In fact, you should be grabbing and reading the output/error streams anyway, because of the points raised by the Traps article. Google "Java Runtime exec traps" and you'll probably get a link to this JavaWorld article, which describes common Runtime.exec problems and how to solve them.
Edited by: paulcw on Jun 15, 2010 4:09 PM

How to process large input CSV file with File adapter

Similar Messages

Maybe you are looking for