How to handle large data in file adapter

We have a scenario Proxy -> PI -> File Sever using File adapter.
File adapter is using FCC for conversion.
recently we had wave 2 products live and suddenly for this interface we have increase in volume of messages, due to which File adapter is not performing well, PI goes slow or frequent disconnect from file server problem. Due to which either we will have duplicate records in file or file format created is wrong.
File size is somewhere around 4.07 GB which I also think quite high for PI to handle.
Can anybody suggest how we can handle such large data.
Regards,
Vikrant

Check this Blog for Huge File Processing:
Night Mare-Processing huge files in SAP XI
However, you can take a look also to this Blog, about High Volume Messages:
Step-by-Step Guide in Processing High-Volume Messages Using PI 7.1's Message Packaging
PI Performance Tuning Best Practice:
http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/2016a0b1-1780-2b10-97bd-be3ac62214c7?QuickLink=index&overridelayout=true&45896020746271

Similar Messages

  • How to handle blank line in File adapter

    Hi,
    I am using a file adapter with CSV as input file. If I have a blank line in CSV file, BPEL is failing read it, no instance is created and file is archieved.
    Please let me know how to handle this issue.
    Thanks

    Hi Yatan,
    I am already using rejectedMessageHandlers, I am on 10.1.3.3.
    Here's my bpel.xml
    ============
    <?xml version = '1.0' encoding = 'UTF-8'?>
    <BPELSuitcase>
    <BPELProcess id="EmpAddressUpdABCSImpl" src="EmpAddressUpdABCSImpl.bpel">
    <partnerLinkBindings>
    <partnerLinkBinding name="ReadFile">
    <property name="wsdlLocation">ReadFile.wsdl</property>
    <property name="rejectedMessageHandlers">file://E:/HRARCHIVE/BadData</property>
    </partnerLinkBinding>
    <partnerLinkBinding name="InsertAddressUpd">
    <property name="wsdlLocation">InsertAddressUpd.wsdl</property>
    <property name="retryInterval">60</property>
    </partnerLinkBinding>
    </partnerLinkBindings>
    <configurations>
    <property name="sensorLocation">sensor.xml</property>
    <property name="sensorActionLocation">sensorAction.xml</property>
    </configurations>
    <activationAgents>
    <activationAgent className="oracle.tip.adapter.fw.agent.jca.JCAActivationAgent" partnerLink="ReadFile">
    <property name="portType">Read_ptt</property>
    <property name="rejectedMessageHandlers">file://E:/HRARCHIVE/BadData</property>
    </activationAgent>
    </activationAgents>
    </BPELProcess>
    </BPELSuitcase>
    Someone told me that this can be achieved by changing the native schema, I tried many options but no result. Please suggest me.
    -- Shirish

  • How to handle large data sets?

    Hello All,
    I am working on a editable form document. It is using a flowing subform with a table. The table may contain up to 50k rows and the generated pdf may even take up to 2-4 Gigs of memory, in some cases adobe reader fails and "gives up" opening these large data sets.
    Any suggestions? 

    On 25.04.2012 01:10, Alan McMorran wrote:
    > How large are you talking about? I've found QVTo scales pretty well as
    > the dataset size increases but we're using at most maybe 3-4 million
    > objects as the input and maybe 1-2 million on the output. They can be
    > pretty complex models though so we're seeing 8GB heap spaces in some
    > cases to accomodate the full transformation process.
    Ok, that is good to know. We will be working in roughly the same order
    of magnitude. The final application will run on a well equipped server,
    unfortunately my development machine is not as powerful so I can't
    really test that.
    > The big challenges we've had to overcome is that our model is
    > essentially flat with no containment in it so there are parts of the
    We have a very hierarchical model. I still wonder to what extent EMF and
    QVTo at least try to let go of objects which are not needed anymore and
    allow them to be garbage collected?
    > Is the GC overhead limit not tied to the heap space limits of the JVM?
    Apparently not, quoting
    http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html:
    "The concurrent collector will throw an OutOfMemoryError if too much
    time is being spent in garbage collection: if more than 98% of the total
    time is spent in garbage collection and less than 2% of the heap is
    recovered, an OutOfMemoryError will be thrown. This feature is designed
    to prevent applications from running for an extended period of time
    while making little or no progress because the heap is too small. If
    necessary, this feature can be disabled by adding the option
    -XX:-UseGCOverheadLimit to the command line."
    I will experiment a little bit with different GC's, namely the parallel GC.
    Regards
    Marius

  • How to handling Binary data and File operation?

    Hello Everyone,
    I think this question might have been asked a lot of time but I was unable to find one solution so, please help needed in this from all you guys..
    I am creating a byte[] of the media files of mp4, and jpg, using the below code,,
    File ff = new File(filename);
            fos = new FileOutputStream(ff);
            int b;
            byte[] f = this.getMediaFile();
            for (b = 0; b < f.length; b++) {
                if (f[b] != -1) {
                    fos.write(f);
    fos.flush();
    fos.close();
    After i convert it i have to store it in the MySql Database as BLOB object, for retrieving byte[] from the database is simple i can do that also successfully with this code.java.sq.Blob obj = (java.sql.Blob) rs.getBlob("file");
    InputStream is = null;
    ByteArrayOutputStream bc = null;
    is = obj.getBinaryStream();
    bc = new ByteArrayOutputStream();
    int b;
    while ((b = is.read()) != -1) {
    System.out.print(b + " ");
    bc.write(b);
    Now the real problem is that when i want to recreate a mp4 audio or video file i am unable to do so, the file created is smaller in size and also it doesn't get played in windows default player nor VLC. I am able to get the image file of jpg format using the BufferedImage & ImageIO.writer();
    How do i do for mp4 file, have no clue for that.
    Pleases help out with this.
    Thanks..                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

    if (f[b] != -1) {I don't get that line. What if you try to output the data without that condition, like this:
    fos = new FileOutputStream(ff);
            byte[] f = this.getMediaFile();
            for (int b = 0; b < f.length; b++) {
                 fos.write(f);
    fos.close();

  • How to handle rejected messages in file adapter

    Hi....! every one.........
    in file adapter read operation it will pick the files even the file contains unstructure message but that file wont be processed.
    where the rejected message goes.
    i have requirement like that rejected message i have to store in another file
    please provide me answer..........

    Hi,
    You can add the parameter 'PhysicalErrorArchiveDirectory' in the JCA file (of the File Adapter).
    Put there the required directory where you want the rejected file to be sent.
    You can add this property in 2 ways:
    1. In your JDeveloper open the JCA file and add: <property name="PhysicalErrorArchiveDirectory" value="/../.."/>
    2. In EM, right click the composite->Service/Reference Properties->Choose your file->On the Properties tab press Add-> Choose 'PhysicalErrorArchiveDirectory' and put the directory-> press Apply.
    Arik

  • How To split large message using File Adapter

    Hello everyone,
    Here is my scenario FTP > XI > BI.
    I got 2 questions:
    (1) I am getting large message around 70 MB file.
        How we can split the message into multiple files 10 MB each before processing the file into XI?
    (2) Is there is any way we can find out size of file which is on FTP  without contacting FTP admin?    
        through XI before processing the file?
    Thanks
    Vick

    hi vick,
    check the blog
    Zip or Unzip your Payload with the new PayloadZipBean module of the XI Adapter Framework                                   
    Working with the PayloadZipBean module of the XI Adapter Framework
    SAP XI acting as a (huge) file mover                                   
    The specified item was not found.                                   
    Managing bulky flat messages with SAP XI (tunneling once again) - UPDATED                                   
    The specified item was not found.                                   
    regards
    kummari

  • How to handle large data while acquisition? BNC 2110

    I want to acquire data using  BNC 2110, I am writing a software in VB 6. We will use 3 channels. We are supposed to scan about 10000 points before AcquiredData is triggered. in all we will need to scan 10000 * 1000 * 1000 before data is put into a binary fall. Can anybody let me know, how to hande this large number points

    Hello Vjuno,
    In order to acquire 10,000,000,000 points you are going to have to be streaming this data to your hard drive as you go.  To do this you'll need to write the data you read to a file each loop iteration.  In general it is a good practice to make your "samples to read" at least 10% of your sample rate in seconds to avoid overflowing buffers, however, depending on your computer you may be able to go faster.  I made an example program in LabVIEW and was able to read 10,000 points at a time from each of 3 analog inputs at 333MHz and write the values to file without overflowing a buffer.  However, even opening a web browser while the code was running was enough to delay the VI long enough for the buffer to overflow.
    You can use the DAQmx Configure Input Buffer call to increase the buffer size and account for spikes in CPU usage from other processes, and you should also monitor the "Available Samples Per Channel" property to make sure you aren't steadily gaining samples in your buffer.  Since you want to acquire 10 billion samples at 1MHz this acquisition will take several hours; if you're not able to keep the buffer empty then it will become apparent before the end of your acquisition.  By monitoring the samples in the buffer you can tell if you're pulling the samples out fast enough, if you find that this number is steadily increasing then you should either reduce the sample rate or increase the number of samples to read each time you call the DAQmx Read.
    In my example program I used a write to TDMS (binary) file and a PCI-6251.
    I hope this helps, and have a good night.
    Cheers,
    Brooks

  • How to choose data in file adapter?

    How to choose data in file adapter?

    create a xml file in source folder in local source location and pick it up using file adapter in BPEL and deliver it to Target location.

  • How to handle errors in a file at sender side?

    Hi
    I have done a file to proxy scenario.
    I know how to handle errors on proxy.
    But on sender side when picking the file if one the record have worng fomat its throwing mapping error and its not processing any record..
    I wanted to process the records which have right format and data and  all remaining recrods which have wromg format should be send back to the sender as file.
    How to do this.
    How to handle error in sender file.
    Regards
    Sowmya

    Hello Sowmya,
    In your scenario Three ways you can validate the data.
    1) Before the data reaches into SAP system ie in XI system during Mapping or before mapping i,e in Adapter Module in the Sender side
    2) this option, is in the receiver applications side. ie. Validations will be taken care in the SAP system i.e in ABAP server proxy code.
    3)Through BPM, If Mapping Exception Occures then through exception Branch you can send bad formate file to sender again.
    Generally, it is prefer to more business critical validations in the Application System ie Receiver Application System (ABAP Server Proxy)
    In this, you can have more flexibility of the validations as you are validating some of the SAP payroll informations as Personal ID etc.
    Based on the complexity and flexibility of the requirement, you can either do this in the XI (if XI, ie Sender Adapter Module or Mapping) or in the ABAP proxy
    Thanks'
    Sunil Singh

  • How to handle large result set of a SQL query

    Hi,
    I have a question about how to handle large result set of a SQL query.
    My query returns more than a million records. However, the Query Template has a "row count" parameter. If I don't specify it, it by default returns only 100 lines of records in the query result. If I specify it, then it's limited to a specific number.
    Is there any way to get around of this row count issue? I don't want any restriction on the number of records returned by a query.
    Thanks a lot!

    No human can manage that much data...in a grid, a chart, or a direct-connected link to the brain. 
    What you want to implement (much like other customers with similar requirements) is a drill-in and filtering model that helps the user identify and zoom in on data of relevance, not forcing them to scroll through thousands or millions of records.
    You can also use a time-based paging model so that you only deal with a time "slice" at one request (e.g. an hour, day, etc...) and provide a scrolling window.  This is commonly how large datasets are also dealt with in applications.
    I would suggest describing your application in more detail, and we can offer design recommendations and ideas.
    - Rick

  • How to handle time&date

    can anyone tell me how to handle time&date correctly?using Calendar,GregorianCalendar,TimeZone,Locale
    thank you very much

    This is too large a topic to discuss in depth here. Here is a link to a tutorial on times and dates, and a search link that references many documents on the subject.
    http://java.sun.com/docs/books/tutorial/i18n/format/dateintro.html
    http://onesearch.sun.com/search/developers/index.jsp?and=calendar+&nh=100&phr=how+to&qt=&not=&field=&since=&col=javatecharticles&col=javatutorials&col=devall&rf=0&Search.x=20&Search.y=7
    When you have specific questions, just ask.

  • In  BDC how you handled header data and item data

    In  BDC how you handled header data and item data

    Raja,
    Can you be more clear ?
    Usually you load the header data one and then loop at the item data and then load the item data.
    This example should help you.
    http://www.sap-img.com/abap/bdc-example-using-table-control-in-bdc.htm
    Regards,
    Ravi
    Note - Please mark all the helpful answers

  • How to handle the java.policy file ?

    Can somebody tell me how to handle the java.policy file?
    I always get java.net.SocketExceptions and java.security.AccessControlExceptions while connecting to an appserver from an applet.
    What do I have to write in the java.policy file, where do I have to place it and do I have to call it in some way form my applet?
    Thanks in advance.
    don call

    The java.policy file goes in your jre installation directory in .../jre/lib/security (there should be one there already).
    I used it to allow otherwise restricted permissions for an applet using javax.comm. Add something like the following to the file:
    grant codeBase "URL:http://yourDomainName/rootDirectoryOfYourApp/*" {
         permission java.security.AllPermission;
    This will give the applet downloaded from your site all permissions. You might want to give only certain permissions, I don't know.
    Teri

  • How to have Secure Connection for File Adapter.Pls help

    Hi All,
    For File to File Scenario
    How to have Secure Connection for File Adapter.
    Pls send links/blogs explaining this scenario.
    Regards

    hi rich
    go through these links
    FTPs connection failed - error ".. certificate rejected by ChainVerifier"
    Re: What is SFTP, FTI channels
    http://help.sap.com/saphelp_erp2005/helpdata/en/e3/94007075cae04f930cc4c034e411e1/frameset.htm
    http://help.sap.com/saphelp_erp2005/helpdata/en/bc/bb79d6061007419a081e58cbeaaf28/frameset.htm
    FTPS implementation question.
    http://help.sap.com/saphelp_nw04s/helpdata/en/43/0e16bfd7b021aee10000000a1553f6/frameset.htm
    Server certificate rejected by ChainVerifier:FTPS server(Points Guaranteed)
    /people/krishna.moorthyp/blog/2007/07/31/sftp-vs-ftps-in-sap-pi
    File adapter
    thanks
    Kunaal

  • Can express vi handle large data

    Hello,
    I'm facing problem in handling large data using express vi's. The input to express vi is a large data of 2M samples waveform & i am using 4 such express vi's each with 2M samples connected in parallel. To process these data the express vi's are taking too much of time compared to other general vi's or subvi's. Can anybody give the reason why its taking too much time in processing. As per my understanding since displaying large data in labview is not efficient & since the express vi's have an internal display in the form of configure dialog box. Hence i feel most of the processing time is taken to plot the data on the graph of configure dailog box. If this is correct then Is there any solution to overcome this.
    waiting for reply
    Thanks in advance

    Hi sayaf,
    I don't understand your reasoning for not using the "Open Front Panel"
    option to convert the Express VI to a standard VI. When converting the
    Express VI to a VI, you can save it with a new name and still use the
    Express VI in the same VI.
    By the way, have you heard about the NI LabVIEW Express VI Development Toolkit? That is the choice if you want to be able to create your own Express VIs.
    NB: Not all Express VIs can be edited with the toolkit - you should mainly use the toolkit to develop your own Express VIs.
    Have fun!
    - Philip Courtois, Thinkbot Solutions

Maybe you are looking for