Splitting a huge file in chunks.

We have a requirement to split file containing 23 million records in chunks of 5000 records.I have used "Recordsets per message" option but it splits just 2.25 lakh records and from that records there will not be any idocs to be posted.Does anyone has idea on this?

Hi Sravya
Have a look at these threads,
Processing huge file loads through XI
Handling Large Files in XI
cheers
Sameer

Similar Messages

Processing Large Files using Chunk Mode with ICO

Hi All,
I am trying to process Large files using ICO. I am on PI 7.3 and I am using new feature of PI 7.3, to split the input file into chunks.
And I know that we can not use mapping while using Chunk Mode.
While trying I noticed below points:
1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
2) I used Dummy Interface in my scenario and it worked Fine.
So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
Thanks in Advance,
- Pooja.

Hello,
While trying I noticed below points:
1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
2) I used Dummy Interface in my scenario and it worked Fine.
So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
According to this blog:
File/FTP Adapter - Large File Transfer (Chunk Mode)
The following limitations apply to the chunk mode in File Adapter
As per the above screenshots, the split never cosiders the payload. It's just a binary split. So the following limitations would apply
Only for File Sender to File Receiver
No Mapping
No Content Based Routing
No Content Conversion
No Custom Modules
Probably you are doing content conversion that is why it is not working.
Hope this helps,
Mark
Edited by: Mark Dihiansan on Mar 5, 2012 12:58 PM

How to split a big file into several small files using XI

Hi,
Is there any way to split a huge file into small files using XI.
Thanks
Mukesh

There is a how to guide for using file adapters, an "sdn search" will get you the document.
Based on that , read a file into XI, Use strings to store the file content and do the split
here is some code to get you started
===========
Pseudocode:
===========
// This can be passed in - however many output files to which the source is split
numOutputFiles = 5;
// Create the same number of filehandles as numOutputFiles specifies
open file1, file2, file3, file4, file5;
// Create an Array to hold the references to those filehandles
Array[5] fileHandles = {file1, file2, file3, file5, file5};
// Initialize a loop counter
int loopCounter = 0;
// a temporary holder to "point" the output stream to the correct output file
File currentOutputFile = null;
// loop through your input file
while (sourceFile.nextLine != null)
loopCounter++;
if (loopCounter == (numOutputFiles +1) // you've reached 5, loop back
loopCounter = 1;
currentOutputFile = fileHandles[loopCounter]; // gets the output file at that index of the array
currentOutputFile.write(sourceFile.nextLine);
regards
krishna

Split a ZIP File in smaller Chunks / payloads on ECC 6.0 (NOT PI)

I call a external web service directly from ECC6.0. In one of the proxy class's methods I need to send a zip file. I can generate ZIP file using SAP supplied class. The file size of this zip file 150 MB and I want to break this file in smaller chunks say 2 MB chunks. Is there a FM / Class / some way to break this file into smaller chunks?
Appreciate your inputs
Thanks,
Vikram

Hi Vikram,
You can use the FM DX_SPLIT_FILE to split the zip file.
Regards,
Ashvin

I have a huge file which is in GB and I want to split the video into clip and export each clip individually. Can you please help me how to split and export the videos to computer? It will be of great help!!

I have a huge file which is in GB and I want to split the video into clip and export each clip individually. Can you please help me how to split and export the videos to computer? It will be of great help!!

video
What version of Premiere Elements do you have and on what computer operating system is it running?
Please review the following workflow.
ATR Premiere Elements Troubleshooting: PE11: Project Assets Organization for Scene and Highlight Grabs from Collection o…
But please also determine if your project goal is supported by
a. format of your source
and
b. computer resources
More later based on details that you will post.
ATR

Issue with file to file in PI 7.3 (Splitting huge files)

Hi All,
Need your help in fixing the issue with file splitting
We are doing some sample scenarios(file to file) on PI 7.3 server.
We are trying to split a 10MB file by using the 'Advanced Mode' option in the sender file adapter. We gave max split file size as 2MB. The file got split into 5 chunks and was successfully sent to receiver file adapter. In receiver adapter we are able to see that. But in the target folder only 1 file was seen with size 2MB. All other chunks were missing. We need to have the whole data sent from source to target.
How to fix this issue? please provide your inputs.
Thanks and Regards,
Lakshmi Narayana

PI 7.3 has capable of processing larger size files.
Questions:
Have you picked EOIO quality of service? Hope you dont do mapping or content conversion for this file?
have you seen this link
http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/a06d79f3-d094-2e10-1a81-f4d802d0bcf1?QuickLink=index&overridelayout=true
http://help.sap.com/saphelp_nw73/helpdata/en/44/682BCD7F2A6D12E10000000A1553F6/frameset.htm

Splitting large video files into small chunks and combining them

Hi All,
Thank you for viewing my thread. I have searched for hours but cannot find any information on this topic.
I need to upload very large video files. The files can be something like 10GB in size, currently .mpeg format but they may be .mp4 format later on too.
Since the files will be uploaded remotely using using internet from the clients mobile device (3G or 4G in UK) I need to split the files into smaller chunks. This way if there is a connection issue (very likely as the files are large) I can resume the upload without doing the whole 10gb again.
My question is, what can I use to split a large file like 10Gb into smaller chunks like 200mb, upload the files and then combine them again? Is there a jar file that I can use. i cant install stuff like ffmpeg or JMF on the clients laptops.
Thank you for your time.
Kind regards,
Imran

There is a Unix command called split that does just that splits a file into chunks of a size you specify. When you want to put the bits bact together you use the command cat.
If you are comfortable in the terminal you can look at the man page for split for more information.
Both commands are data blind and work on binary as well as text files so I would think this should work for video but of course check it out to see if the restored file still works as video.
regards

Handle huge file size in file receiver

I have file(NfS) to file(Ftp) scenario the receiver file failing because of my huge data(nearly 20KB) i am using Message mapping functionality i cant use file check machanisam to split my file .could you share some ideas how can i achieve this requirement .
file ---> PI (7.4 java)--->File(Ftp) 20 to 25 KB)
how can i handle huge data in FIle receiver side ?

Hi Raman,
First you need to identify which step is taking more time. Adapter or transformation. Please share more details about your scenario.
If your scenario is pass through (with no transformation), then you can use the chunk mode. Please refer the below blog
File/FTP Adapter - Large File Transfer (Chunk Mode)
PI/XI: PI 7.3 processing of large files - teaser
If your file is having multiple recordset then you can pick the fix no of record set. Please refer the below blog
Night Mare-Processing huge files in SAP XI
regards,
Harish

Processing file in chunks

I have a file structure like Header-Transaction Records(100K)-Trailer with Transaction Records of 100k.
I want to process the this file through file-xi-file scenario. I am using the "Recordsets per message"
(5000 records per each chunk)parameter in the sender file communicaiton channel to process in chunks. But I didn't observed anything differce in processing by XI. The file is processed same way as it before without any "Recordsets per message" parameter set. How can I monitor that this parameter is took into effect and the system performance is improved.
If I got to write a operating sytstem command/script to split , how to write it for the file structure that i am
talking abt above ??
thank
kumar

Hi Palnati
This works for FCC and NFS else its not having any effect.
I replied your question in other thread.
handling huge files
Thanks
Gaurav

Split an XML File and invoke a Web Serivce for each splits using an XSLT

Hi,
We Have a requirement to Split an incoming XML File into chunks based on a child element and hit the target Web Service for each and every split ( i.e no of splits = no of hits on WebService).
Currently, the incoming XML file is getting splitted and gets appended to the SAME Payload and the WebService is hitted only once with the entire Payload.
Is it possible to invoke a WebService within a XSLT ?
Please find below the XSLT code used to split the file in chunks of 3
<xsl:param name="pItemsNumber" select="3"/>
<xsl:template match="@*|node()">
<xsl:choose>
<xsl:when test="count(Batch/Item) > 4">
<ItemMaintenance>
<xsl:for-each-group select="Batch/Item"
group-adjacent="(position()-1) idiv $pItemsNumber">
<xsl:result-document href="ItemsMatch\{current-grouping-key()}.xml">
<ItemMaintenance>
<xsl:copy-of select="current-group()"/>
</ItemMaintenance>
</xsl:result-document>
</xsl:for-each-group>
</ItemMaintenance>
</xsl:when>
<xsl:otherwise>
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:otherwise>
</xsl:choose>
</xsl:template>

Hello,
It is possible to invoke a webservice from XSLT. To achieve this you must create a custom XSLT function that does the actual webservice call. You need to write an implementation in Java for this. See my blogpost http://blog.melvinvdkuijl.nl/2010/02/custom-xslt-functions-in-oracle-soa-suite-11g/ for a simple example.
Regards,
Melvin

File Splitting for Large File processing in XI using EOIO QoS.

Hi
I am currently working on a scenario to split a large file (700MB) using sender file adapter "Recordset Structure" property (eg; Row, 5000). As the files are split and mapped, they are, appended to a destination file. In an example scenario a file of 700MB comes in (say with 20000 records) the destination file should have 20000 records.
To ensure no records are missed during the process through XI, EOIO, QoS is used. A trigger record is appended to the incoming file (trigger record structure is the same as the main payload recordset) using UNIX shellscript before it is read by the Sender file adapter.
XPATH conditions are evaluated in the receiver determination to eighther append the record to the main destination file or create a trigger file with only the trigger record in it.
Problem that we are faced is that the "Recordset Structure" (eg; Row, 5000) splits in the chunks of 5000 and when the remaining records of the main payload are less than 5000 (say 1300) those remaining 1300 lines get grouped up with the trigger record and written to the trigger file instead of the actual destination file.
For the sake of this forum I have a listed a sample scenario xml file representing the inbound file with the last record wih duns = "9999" as the trigger record that will be used to mark the end of the file after splitting and appending.
<?xml version="1.0" encoding="utf-8"?>
<ns:File xmlns:ns="somenamespace">
<Data>
     <Row>
          <Duns>"001001924"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001925"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001926"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001927"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001928"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001929"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"9999"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
</Data>
</ns:File>
In the sender file adapter I have for test purpose changed the "Recordset structure" set as "Row,5" for this sample xml inbound file above.
I have two XPATH expressions in the receiver determination to take the last record set with the Duns = "9999" and send it to the receiver (coominication channel) to create the trigger file.
In my test case the first 5 records get appended to the correct destination file. But the last two records (6th and 7th record get sent to the receiver channel that is only supposed to take the trigger record (last record with Duns = "9999").
Destination file: (This is were all the records with "Duns NE "9999") are supposed to get appended)
<?xml version="1.0" encoding="UTF-8"?>
<R3File>
     <R3Row>
          <Duns>"001001924"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
     <R3Row>
          <Duns>"001001925"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
     <R3Row>
          <Duns>"001001926"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</xtract_Code>
     </R3Row>
          <R3Row>
          <Duns>"001001927"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
          <R3Row>
          <Duns>"001001928"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
</R3File>
Trigger File:
<?xml version="1.0" encoding="UTF-8"?>
<R3File>
          <R3Row>
          <Duns>"001001929"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Ccr_Extract_Code>"A"</Ccr_Extract_Code>
     </R3Row>
          <R3Row>
          <Duns>"9999"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Ccr_Extract_Code>"A"</Ccr_Extract_Code>
     </R3Row>
</R3File>
I ve tested the XPATH condition in XML Spy and that works fine. My doubts are on the property "Recordset structure" set as "Row,5".
Any suggestions on this will be very helpful.
Thanks,
Mujtaba

Hi Debnilay,
We do have 64 bit architecture and still we have the file processing problem. Currently we are splitting the file into smaller chuncks and processsing. But we want to process as a whole file.
Thanks
Steve

The amount of memory used for data is a lot larger than the saved file size why is this and can I get the memory usage down without splitting up the file?

I end up having to take a lot of high sample rate data for relativily long periods of time. When I save the data it is usually over 100 MB. When I load the data for post-processing though the amount of memory used is excessively higher than the file size. This causes my computer to crash because 1.5 GB is not enough. Is there a way to stop this from happening withoput splitting up the file into smaller files.

LabVIEW can efficiently handle large files, far beyond 100Mb, provided that care is taken in the coding of the loading/processing routines. Here are several suggestions:
1) Check out the resources National Instruments has put together (NI Developer Zone > Development Library > Measurement and Automation Software > LabVIEW > Development System > Optimizing Applications > Managing Memory), specifically the article entitled "Managing Large Data Sets in LabVIEW".
2) Load and process the data in chunks if possible.
3) Avoid sending the data to front panel indicators, using local/global variables for data storage, or changing data types unless absolutely necessary.
4) If using LabVIEW 7.1, use the "show buffer" tool to determine when LabVIEW is creating extra
copies of data in memory.

Handling huge files

I have a file structure like Header-Transaction Records(100K)-Trailer with Transaction Records of 100k.
I want to process the this file through file-xi-file scenario.
If I got to write a operating sytstem command/script to split the file in chunks , how to write it for the file structure that i am talking abt above ??
Regards
kumar

Hi Palnati
Check my reply to this thread
File Adapter Not handling High Volume data!
You can try using this Unix bash script
# Input: lines containing thre fields separated by white spaces
TAR_DIR=/tmp/data # The path name of the target directory
# Create the target directory if necessary
[ -d $TAR_DIR ] || mkdir $TAR_DIR
# Clear the contents of the target directory
rm -rf $TAR_DIR/*
# Do it
while read N E1 E2; do
echo $N>>$TAR_DIR/$E1$E2 # Create or append
done <<!
condition1
condition2
make changes as per your need
Thanks
Gaurav

Split a huge photo library in aperture 3 ?

Is it possible tu split a huge library in Aperture 3 ?

I actually find this very helpful because my Aperture (version 3.4.2) is extremely slow and crashes very often.
This can also be an indication that your Aperture Library needs repairing, or you may have imported corrupted media - image files or incompatible videos.
I'd try the Aperture lIbrary First Aid Tools to repair the database and permissions, if necessary to rebuild the Library, see:
Repairing and Rebuilding Your Aperture Library: Aperture 3 User Manual
After a crash it is always a good precaution to repair the library, since a crash may interrupt a database transaction and leave the Aperture library in an inconsistent state.
If repairing/rebuilding does not fix the slowness, check your recent imports, if they contain incompatible videos or images that cannot be adjusted and cause Aperture to crash.

Huge Files Processed in XI

Hi Experts,
Scenario: File ---> XI -
>JMS
I have a requirement where XI will processed above 5MB files uing content conversion. Is there any performance issue in XI to proceesed huge files.
Does XI support above 5MB files? If XI processed huge file then what would be the limitation? what would be the best approach to achive the sceario.
Would apreciate your inputs.
Regards
Sameer,

Thanks Udo for your reply.
Since I am not using BPM in this scenario. As the file contains above 22,000 records so can we split the file at every 5000th record and process it by using Recordsets per Message in sender FCC.
I am using Graphical mapping (1:1) with FCC both the side i,e FILE --- XI --- JMS
Is there any performance issue in XI while doing this?
Sameer!

Splitting a huge file in chunks.

Similar Messages

Maybe you are looking for