Split/join large files

I have a need to split and eventually re-join a large file. Are there recommended utilities for doing this?
Thanks,
George E.

Resolved. Found the answer in the archives. Stuffitt.

Similar Messages

File Splitting for Large File processing in XI using EOIO QoS.

Hi
I am currently working on a scenario to split a large file (700MB) using sender file adapter "Recordset Structure" property (eg; Row, 5000). As the files are split and mapped, they are, appended to a destination file. In an example scenario a file of 700MB comes in (say with 20000 records) the destination file should have 20000 records.
To ensure no records are missed during the process through XI, EOIO, QoS is used. A trigger record is appended to the incoming file (trigger record structure is the same as the main payload recordset) using UNIX shellscript before it is read by the Sender file adapter.
XPATH conditions are evaluated in the receiver determination to eighther append the record to the main destination file or create a trigger file with only the trigger record in it.
Problem that we are faced is that the "Recordset Structure" (eg; Row, 5000) splits in the chunks of 5000 and when the remaining records of the main payload are less than 5000 (say 1300) those remaining 1300 lines get grouped up with the trigger record and written to the trigger file instead of the actual destination file.
For the sake of this forum I have a listed a sample scenario xml file representing the inbound file with the last record wih duns = "9999" as the trigger record that will be used to mark the end of the file after splitting and appending.
<?xml version="1.0" encoding="utf-8"?>
<ns:File xmlns:ns="somenamespace">
<Data>
     <Row>
          <Duns>"001001924"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001925"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001926"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001927"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001928"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001929"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"9999"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
</Data>
</ns:File>
In the sender file adapter I have for test purpose changed the "Recordset structure" set as "Row,5" for this sample xml inbound file above.
I have two XPATH expressions in the receiver determination to take the last record set with the Duns = "9999" and send it to the receiver (coominication channel) to create the trigger file.
In my test case the first 5 records get appended to the correct destination file. But the last two records (6th and 7th record get sent to the receiver channel that is only supposed to take the trigger record (last record with Duns = "9999").
Destination file: (This is were all the records with "Duns NE "9999") are supposed to get appended)
<?xml version="1.0" encoding="UTF-8"?>
<R3File>
     <R3Row>
          <Duns>"001001924"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
     <R3Row>
          <Duns>"001001925"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
     <R3Row>
          <Duns>"001001926"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</xtract_Code>
     </R3Row>
          <R3Row>
          <Duns>"001001927"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
          <R3Row>
          <Duns>"001001928"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
</R3File>
Trigger File:
<?xml version="1.0" encoding="UTF-8"?>
<R3File>
          <R3Row>
          <Duns>"001001929"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Ccr_Extract_Code>"A"</Ccr_Extract_Code>
     </R3Row>
          <R3Row>
          <Duns>"9999"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Ccr_Extract_Code>"A"</Ccr_Extract_Code>
     </R3Row>
</R3File>
I ve tested the XPATH condition in XML Spy and that works fine. My doubts are on the property "Recordset structure" set as "Row,5".
Any suggestions on this will be very helpful.
Thanks,
Mujtaba

Hi Debnilay,
We do have 64 bit architecture and still we have the file processing problem. Currently we are splitting the file into smaller chuncks and processsing. But we want to process as a whole file.
Thanks
Steve

Problem using Category to split a large file

I want to split a large class file by putting methods used only internally into a separate source file. The original file is "Calculator.m"; the new file is "CalculatorP2.m" which has been added to the project. In the main file I have added the statement:
#import "CalculatorP2.m"
just ahead of "@implementation Calculator".
In "CalculatorP2.m I have placed the statements:
#import "Calculator.h"
@interface Calculator ( CalculatorP2 )
@end
@implementation Calculator ( CalculatorP2 )
ahead of the source code. The compiler is happy with this but the linker fails, saying:
/usr/bin/ld: multiple definitions of symbol .objccategory_name_CalculatorCalculatorP2
Any idea as to how to fix this? I am using XCode 2.4.1.

SecondViewControllerP2.h:
#import <UIKit/UIKit.h>
#import "SecondViewController.h"
@interface Secondviewcontroller(SecondviewcontrollerP2)
- (void)method1;
@end
SecondViewControllerP2.m:
#import "SecondViewControllerP2.h"
@implementation Secondviewcontroller(SecondviewcontrollerP2)
- (void)method1 {
NSLog(@"method1");
@end
SecondViewController.m:
#import "SecondViewController.h"
#import "SecondViewControllerP2.h"
@implementation Secondviewcontroller
@end

Processing/Splitting of large file

Hi,
Some time ago I did post some related question.
Is it possible in file2file(RFC) scenario to send an XML - file containing 1000 records to XI and split it so that there will be 1000 single output-messages written to a network destination. All of them containing one record. As output it shall be written to File (or RFC)?
Please let me know if you have some ideas.
Thanks,
Sebastian

Hi Wojciech,
thanks again for your help so far.
Queue is okay.
Source files:
<?xml version="1.0" encoding="ISO-8859-1"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://MyTutorial/SHeinz07" targetNamespace="http://MyTutorial/SHeinz07">
<xsd:element name="MT_Werkteil" type="DT_Werkteil" />
<xsd:complexType name="DT_Werkteil">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
13289f7070c111db8d5e00508b691bcc
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="Row" minOccurs="0" maxOccurs="unbounded">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
96d1db50592111dbc096cb2fc0a864a8
</xsd:appinfo>
</xsd:annotation>
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Nummer" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
96d1db51592111db8ca0cb2fc0a864a8
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="Name" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
96d1db52592111db8741cb2fc0a864a8
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="Preis" type="xsd:integer">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
96d1db53592111db8dd6cb2fc0a864a8
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:schema>
Target file:
<?xml version="1.0" encoding="ISO-8859-1"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://MyTutorial/SHeinz07" targetNamespace="http://MyTutorial/SHeinz07">
<xsd:element name="MT_Werkteil_single" type="DT_Werkteil_single" />
<xsd:complexType name="DT_Werkteil_single">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
13a6f73070c111dbc23600508b691bcc
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="Nummer" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
e53f2c90700d11dbadfdc229c0a864a8
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="Name" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
e53f2c91700d11dbb999c229c0a864a8
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="Preis" type="xsd:integer">
<xsd:annotation>
<xsd:appinfo source="http://sap.com/xi/TextID">
e53f2c92700d11dbb2edc229c0a864a8
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:schema>
Currently my output in Mapping Test looks like that:
<?xml version="1.0" encoding="UTF-8"?>
<ns0:Messages xmlns:ns0="http://sap.com/xi/XI/SplitAndMerge">
   <ns0:Message1>
      <ns1:MT_Werkteil_single xmlns:ns1="http://MyTutorial/SHeinz07">
         <Nummer>1000</Nummer>
         <Name>Schraube</Name>
         <Preis>12</Preis>
      </ns1:MT_Werkteil_single>
   </ns0:Message1>
</ns0:Messages>
Though I did some researches on contexts and splitbyvalue I am right now just guessing instead of understanding why there is no correct output.
do you have any idea?
Regards,
Sebastian

Splitting a Large File

I record a 5 hour radio program that I like to listen to on my iPod. My previous recorder automatically split the program at pre-determined times. My Mac recording software is one big file. Is there an AppleScript way to split the track or add chapter bookmarks? It has to be easy since I will be doing this daily. Thanks.

afernandes wrote:
Thanks Michel,
I will try this out.
Do you know if this will create a new file(s) ?
http://discussions.apple.com/thread.jspa?threadID=2460925&tstart=0
What I want to do is to break up my 2 hours video into smaller chunks then burn the good chunks as raw footage ( AVI/MOV) onto backup data DVDs. Then export all the chunks into compresssed files (MPEG-4?) and save these on another data DVD.
Avoid to compress. Save as quicktime movie.
Michel Boissonneault

How to split a large file when disk space is limited

I have a long (200 minutes) DV clip which uses up most [40 GB] of my external hard drive. The last 90 minutes of the clip is blank, and I need to delete the corresponding [18 GB] filespace to free-up space on my hard drive for editing.
How do I delete a portion of the file without having the 22 GB of free space Quicktime seems to need to temporarily store the new (smaller file).
Thanks,
David

I modified the program as follow:
ZipEntry entry = new ZipEntry(file.getName());
entry.setTime(file.lastModified());
zip.putNextEntry(entry);
byte[] bytes=new byte[1024];
int len;
while ((len=in.read(bytes))>0) {
zip.write(bytes,0,len);
And now it is working. I can zip a file with 125MB size.
Thank you for your help.
Michelle

Oracle Service Bus 10.3 - split join how to set header for invoked service

In Oracle Service Bus 10.3 how can I set the header for the service being invoked. We make parallel calls to 2 different business services. One of the invoked service requires header to be set and I am unable to figure out how it can be done. Below are snippet of the code and WSDL.
--- Login Operation start ---
<operation name="login">
<wsdlsoap:operation soapAction="blah"/>
<input name="loginRequest">
<wsdlsoap:body use="literal"/>
<wsdlsoap:header message="passport:loginHeader" part="hppwsHeaderElement" use="literal"/>
</input>
<output name="loginResponse">
<wsdlsoap:body use="literal"/>
</output>
<fault name="genericFaultException">
<wsdlsoap:fault name="genericFaultException" use="literal"/>
</fault>
</operation>
--- Login Operation end ---
--- hppwsHeader Element --
<complexType name="hppwsHeaderElement">
<sequence>
<element maxOccurs="1" minOccurs="0" name="version" type="tns1:version"/>
<element maxOccurs="1" minOccurs="0" name="tx-id" type="tns1:tx-id"/>
<element maxOccurs="1" minOccurs="1" name="systemLangCode" type="xsd:string"/>
</sequence>
</complexType>
--- LoginRequest Element ---
<xs:element name="loginRequestElement" type="s0:loginRequestType"/>
<xs:complexType name="loginRequestType">
<xs:sequence>
<xs:element name="userId" minOccurs="1" maxOccurs="1" type="xs:string"/>
<xs:element name="password" minOccurs="1" maxOccurs="1" type="xs:string"/>
</xs:sequence>
</xs:complexType>
--- LoginResponse Element ---
<xs:element name="loginResponseElement" type="s0:loginResultType"/>
<xs:complexType name="loginResultType">
<xs:sequence>
<xs:element name="sessionToken" minOccurs="1" maxOccurs="1" type="xs:string"/>
</xs:sequence>
</xs:complexType>
In Oracle Workshop for WebLogic when I create a split-join .flow file while adding the invoke service call to login operation I am prompted to create the loginRequest and loginResponse variables. I can assign the appropriate expression to loginRequest.loginRequestElement prior to making the invoke call but then the backend business call fails because "systemLangCode" is not defined and set. Actual fault is
<ns1:genericFault xmlns:ns1="http://hppcbl.globalops.company.com/webservice">
<systemLangCode>en</systemLangCode>
<fault>
<ruleNumber>589</ruleNumber>
<fieldName>systemLangCode</fieldName>
<code>field.empty</code>
<desc>System language code is required.</desc>
<ftype>2</ftype>
</fault>
</ns1:genericFault>
I would really appreciate if someone could guide me on how to resolve this
Regards,
-Nilay

There is this $header variable which you should populate before calling the BS. What is the type of BS? If it is WSDL BS , Any Soap BS then above solution will work. I assume that yours is WSDL based BS and setting $header should resolve this issue.
AnyXML and Messaging type BS services requires that you create proper pay load (setting $header will not work).
There is also other actions Set Transport Headers {outbound request/inbound response} which can be used to set transport related headers but not soap headers.
Manoj

Transferring a large file

Hi,
I have to transfer a huge file. I am zipping the file and passing the byteArray. but the problem is the size. I am not able to transfer file of size 2-3 MB.
how do I split the file or send it in small parts.

thank you for the reply but FileInputStream is not serializable i guess. I tried it.
let me give a clear picture of what i am trying to do.
i have to transfer a file from client to server. I am doin this using RMI.
so i will call the remote method by passing the byteArray.
for this i need to split the file. be it FileInputStream or byteArray.
So how do i do it. how do i split the large file.
do i have to use some buffer and keep appending to it.
if you can give an example or sample code it will be great.
hope u can help me out
thanks

Splitting a large video file into smaller segments/clips using FCE

Is there a way to split a large FCE / iMovie Captured event into smaller segments/clips.
Am attempting to convert my home videos ( Analog: 8mm) into iMovie clips.
Used a ADS Pyro that converts my composite signal ( RCA: Yellow+Red/White) to a digital signal via Firewire using the iMovie capture. However, this creates a single 2 hour file that takes approx 26Gb because there is now DV information.
I would like to (a) Break this into smaller segments/clips then (b) Change the date on these.

afernandes wrote:
Thanks Michel,
I will try this out.
Do you know if this will create a new file(s) ?
http://discussions.apple.com/thread.jspa?threadID=2460925&tstart=0
What I want to do is to break up my 2 hours video into smaller chunks then burn the good chunks as raw footage ( AVI/MOV) onto backup data DVDs. Then export all the chunks into compresssed files (MPEG-4?) and save these on another data DVD.
Avoid to compress. Save as quicktime movie.
Michel Boissonneault

The amount of memory used for data is a lot larger than the saved file size why is this and can I get the memory usage down without splitting up the file?

I end up having to take a lot of high sample rate data for relativily long periods of time. When I save the data it is usually over 100 MB. When I load the data for post-processing though the amount of memory used is excessively higher than the file size. This causes my computer to crash because 1.5 GB is not enough. Is there a way to stop this from happening withoput splitting up the file into smaller files.

LabVIEW can efficiently handle large files, far beyond 100Mb, provided that care is taken in the coding of the loading/processing routines. Here are several suggestions:
1) Check out the resources National Instruments has put together (NI Developer Zone > Development Library > Measurement and Automation Software > LabVIEW > Development System > Optimizing Applications > Managing Memory), specifically the article entitled "Managing Large Data Sets in LabVIEW".
2) Load and process the data in chunks if possible.
3) Avoid sending the data to front panel indicators, using local/global variables for data storage, or changing data types unless absolutely necessary.
4) If using LabVIEW 7.1, use the "show buffer" tool to determine when LabVIEW is creating extra
copies of data in memory.

A pdf file failed to convert to word, presumably because of size. how do i split a large pdf file into manageable secrtions?

I'm running Abode Reader XI version 11.0.7. Repeated attempts to convert a large (439 page) file, a dissertation, failed. How do I split a large pdf file like this into manageable sections for conversion?

Hi Mike,
Your 11MB file is well within the file-size limits for ExportPDF, but depending on the number of pages, complexity of the file (and yours doesn't sound complex), and your connection speed, it is possible that the service is simply timing out before it can finish processing. These steps can help:
If the file already contains editable text (that is, it isn't a scanned document), try disabling OCR as outlined in this this document: How to disable Optical Character Recognition (OCR) when converting PDF to Word or Excel.
Clear the browser cache and try again.
Try a different browser.
Let's start there. If you still can't export the file to Word, let me know and we'll take it from there.
Best,
Sara

How to split/copy/move a large file by exactly line number?

Hi everyone,
I already know the exactly line numbers of a large file. For example,
I want to split or copy or move Infile from 1~66666 to outfile1;
split or copy or move Infile from 66667~166666 to outfile2;
split or copy or move Infile from 166667~266666 to outfile3....
After check and try some commands, still not solving... Can any person help me?
Thanks

head -n 66666 Infile > outfile1
sed "1,66666d" Infile |head -n `expr 166666 - 66667` > outfile2
sed "1,166667d" Infile |head -n `expr 266666 - 166667` > outfile3

Large file doesn't work in a message-splitting scenario

Hello,
I'm trying to measure the XI performance in a message-splitting scenario by having the file adapter to pull the xml file below and send it to XI to perform a simple "message split w/o BPM" and generate xml files for each record in another folder.   I tried 100, 500, 1000 records and they all turned out fine. Once I tried the 2000-records case, the status of that message is WAITING in RWB and I checked sxmb_moni, the message is in "recorded for outbound messaging", I couldn't find any error.
Is there some kinda of threshold that can be adjusted for large files? Thank you.
<?xml version="1.0" encoding="UTF-8"?>
<ns0:MultiXML xmlns:ns0="urn:himax:b2bi:poc:multiMapTest">
   <Record>
      <ID>1</ID>
      <FirstName>1</FirstName>
      <LastName>1</LastName>
   </Record>
</ns0:MultiXML>

Hello,
The Queue ID is "XBTO1__0000". I double-clicked it, it took me to the "qRFC Monitor (Inbound Queue) screen, and showed a XBT01_0000 queue with 3 Entries.
I double-clicked on that, and it took me to another screen that showed a queue with same same and "Running" in Status.
I double-clicked on that, and i saw three entries. For the latest entry, its StatusText says "Transaction recorded".
I double-clicked on that, it took me to Function Builder:Display SXMS_ASYNC_EXEC.
What do I from here?

Splitting large file in XI

can we split the incoming file in XI, we are getting a large file of size 80MB , wanted to cut down to 40MB each
Sender system is sending 80MB file at single shot, they cannot change it.
It has become mandatory for me to break it in XI. (scenario is File to Proxy)

Hi Viswanath,
Handling large files say anything above 100MB is always a problem with File adapter as the data has to be moved from Adapter Engine integration Engine and vice versa.
Third party tools are generally used for that. Conversion Agent by Itemfield is one of the best approaches.
Also, on the Advanced tab of the file sender adapter, select the check box next to Advanced Mode. There you can specify Maximum File Size (Bytes) option.
Huge processing of files
Night Mare-Processing huge files in SAP XI
Step-by-Step Guide in Processing High-Volume Messages Using PI 7.1's Message Packaging
Step-by-Step Guide in Processing High-Volume Messages Using PI 7.1's Message Packaging
SAP XI acting as a (huge) file mover
The specified item was not found.
Managing bulky flat messages with SAP XI (tunneling once again) - UPDATED
The specified item was not found.
Regards,
Vinod.

How do I split a pdf file when the file size is too large?

How do I split a pdf file when the file size is too large? Thanks!

With Adobe Acrobat. It can also optimize your document to make the size smaller.

Split/join large files

Similar Messages

Maybe you are looking for