Unicode huge file conversion problem

I use the command in Unix "<b>sapiconv -f 4110 -t 8600 -o <source file name> <destination file name></b>" calling by ABAP command "CALL 'SYSTEM' ID 'COMMAND' " to convert non-unicode source file to unicode destination file in code page 8600 (Thai language). For the small size source file, It has no problem, but for the huge size source file (27MB - the file was extracted and created by OPEN DATASET from asset master data about 95000 records), it has problem. There are only about 1300 Records (about 200k) in destination file and <b>the remain records are missing!?!?</b> And every time I run the program, It still has certain result.
What can I do for this case? or the command 'sapiconv' has the size limitation for unicode file conversion? or which additional parameter should I use?
Thank you in advance.
David

Hello,
You can use this function module
CALL FUNCTION 'ARCHIVOBJECT_GET_CLIENT'
          EXPORTING
            ARCHIV_DOC_ID               =   V_ARCDOCID
            ARCHIV_ID                        =   V_ARCHIVID
            PATH                                 =   V_PATH
*            DOC_TYPE                       =   'TXT '
*         IMPORTING
*            REPATH                           =
*            COUNT                             =
*            FILE_NAMES                    =
         EXCEPTIONS
           ERROR_ARCHIV                = 1
           ERROR_COMMUNICATIONTABLE       = 2
           ERROR_KERNEL                = 3
           ERROR_HTTP                     = 4
           OTHERS                             = 5.
I think it will help to you,
Anil.

Similar Messages

Remote huge file load problem

---Following is the code to load a huge image file (50-100M)
InputStream in = new FileInputStream(readFile);
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
while((i = in.read(buffer)) != -1){
out.write(buffer, 0, i);
in.close();
byte[] result = out.toByteArray();
---It will crash (or hang) on toByteArray() when file size exceed 18mb.
Is there any way to load large files?
Thanks

1)I am new to the java forum. May I know what is
documentation for java BUT?http://java.sun.com/docs/index.html
2)Also may I know where I can find topics about
processing the file without loading the whole of it
into memory?
3) Actuall we are loading this file into bytearray in
java at client side(PC or MAC) and use rpc to
transfer to server(UNIX) - the C code in server side
will take this byte buffer and use fwrite to create
this huge file. Is there a better way than this?a) Break the file into smaller chunks and send the chunks together with a chunk identifer to the server. The server will then re-construct the file from the chunks.
b) Install an ftp server on the server machine and user FTP to sent the file.

File conversion Problem

Hi Experts,
we are facing a problem when converting one text file by using FTP adapter.
the flat txt file is as below
AT1230001
"AT" - Segment Name
"123" - Field One
"0001" - Field Two
we are using FTP Adapter conversion for the same
the result of XML is as below
- <AT>
<F1>AT</F1>
<F2>123</F2>
</AT>
for the field one it is taking the value of Segment Name, as we are giving KeyFieldValue in Configuration "AT".
please help, it is urgent,
Regards,
Study SAP

Hi Chirag,
Thanks for your reply,
what we are facing here is that in our flat file data, it is EDI file
AT is segment name value
*(estric) is separator
123 is first field value
0001 is second field value
in XML conversion the Segment Name is going into the first field name value and first field name is going in second field value and second field we are not able to capture as we have only two fields for capturing.
As in XML format
<AT>
<F1>AT</F1> "it is taking the value of Segment Name, but it should capture first field value which is "123"
<F2>123<F2> "it is taking the value of First field, it should capture value "0001"
</AT>
How we can ignore Segment value when transferring data ?
Regards,
Study SAP

Idoc to flat file conversion: problem with mapping

Dear experts,
I have the following scenario: SAP (idoc) -> XI -> 3rd party
The idoc has s structure like this:
. ZSCX001 (idoc name)
... IDOC
..... ORDERHEADER
....... ORDERLINE
......... DELIVERY LINE
......... DELIVERY LINE
..... ORDERHEADER
....... ORDERLINE
......... DELIVERY LINE
......... DELIVERY LINE
etc
The flat file has this structure:
. FILENAME
... RECORD
..... FIELD1
..... FIELD2
... RECORD
..... FIELD1
..... FIELD2
etc
Now, per IDOC I need to create 1 file. For each delivery-line in the IDOC I need to create 1 record in the outputfile (flat file).
In message mappings, I did the following:
- messages-tab: Occurrence of the Idoc and inbound message type for flatfile: both 1
- design-tab: I linked IDOC to the FILENAME (so for each Idoc I create 1 flat file)
- design-tab: I linked the tag "Delivery line" to "RECORD" (I expected 1 record per delivery line)
When I test it, I only get 1 resulting record in the flat file, even if I test with 3 ORDERHEADER->ORDERLINE->DELIVERY LINE segments.
Can anybody tell me what I am doing wrong?
Thanks in advance,
William

Hi,
First make sure occurence of Record is 0..unbounded.
Second use the blow mapping rule..
Delivery line --->RemoveContext ---> RECORD
Regards,
Sarvesh

Batch file conversion problem

I'm trying to batch convert a folder of tif files into jpg files using the file>process mulitple dialogue.
I select the folder and the select convert to jpeg hi quality
The process starts but stops at each image with the default file type as .psd I change it and continue and it stops at each image in turn.
What am I doing wrong?
Cheers
Garin

Are your tif files 16 bits/channel? Open one of the tif files and look at the top
of the tab.
MTSTUNER

How to debug file content conversion problems?

Hi,
I'm trying to debug a file content conversion problem. I'm mapping a few nodes in an IDOC to a file of fixed length fields. I'm using the "<Node A>.fieldFixedLengths", "<Node B>.fieldFixedLengths", "<Node C>.fieldFixedLengths", etc. parameters to specify the fixed length records.
However, a certain node (for e.g. Node B) is causing a problem and if it is present in the IDOC, the output file does not get created. Upon checking the XI monitor, I notice that the file gets mapped correctly and thus the problem lies when the file adapter does the file content conversion. How do I debug this because there is no descriptive error in the XI log? If this node is not present, the file gets generated fine.
Thanks,
Basant Gupta

Hi,
If your SXMB_MONI shows, success status, then go to RWB->Message Monitoring->Message display tool and then check Audit log for the analysis,
So it wil help you debug the situation.
If there is no error, then check RWB->Component Monitoring->Adapter Monitoring for you file communciation channel..
/people/michal.krawczyk2/blog/2005/01/02/simple-adapter-and-message-monitoring
Regards,
Moorthy

File Adapter-Problem Reading Huge Files

Hi,
Here is the issue that i am facing
When reading huge file(csv file upto 6MB-8MB) the communication channel configured as File Adapter with a polling interval of 7 min(420 sec) is inconsistent in reading the complete file.Sometimes it reads the the complete file of 6 MB and sometimes it reads a part of the file say 3MB/6MB.Can this inconsistent behaviour be resolved.??
Your suggestions highly appreciated.
Regards
Pradeep

Hi Pradeep !
8mb is not a huge file for XI, I think it is a small one. Maybe your problem is not the size..please check if XI is not starting to read the file before it is completely written to the source folder. If you are creating that csv file from another application directly to the poll source directory of the XI scenario specified in the file adapter, and your poll interval is small, XI could start reading the file while you are still writing it. If this is the case, try to put the file with a different extension or filename than the specified in file adapter comm channel and when the file is completely written, rename it to its final filename and check if you are still having that misbehavior.
You can write the file to a temp directory and the move it to the XI directory once finished.
Regards,
Matias.

LSMW: Codepage conversion error with a Unicode data file

Hi all,
I am currently developing a LSMW upload program which has to use a Unicode data file. The underlying system/target system is NOT a Unicode system. The data file also contains non-Latin2 characters.
In the step "Specify Files", I have specified my Unicode data file and specified the codepage type "4110 - Unicode UTF-8".
In the step "Read Data", then I get the runtime error "CONVT_CODEPAGE", exception "CX_SY_CONVERSION_CODEPAGE".
I would expect that all non-Unicode characters are automatically transformed to "#", but the conversion progam breaks. The character transformation to "#" would be fine.
I am really wondering why, at first, I am able to specify the Unicode codepage type, but then, the file cannot be converted correctly.
What do I make wrong, what can I do to avoid the error?
Thanks a lot in advance for helping me out...
Regards,
Klaus

Hello,
You need convert the file with the format UTF-8. In notepad you can choose this option.
Regards,
Oscar.

Large File Processing Problem

HI Group,
I am facing problem in XI while processing 48 MB File through File adapter,I have used Content Conversion in the design.
I am using normal 64Bit operating system with Max of 2GB heap size,still I am facing the problem,Can any body tell me how much Heap size I required to process 48 MB size file through XI?

Hi,
Refer following SAP note-for this go to www.service.sap.com/notes
File adapter faq- 821267
Java Heap - 862405
for java settings- 722787
This blog may give some insight-/people/sap.user72/blog/2004/11/28/how-robust-is-sap-exchange-infrastructure-xi
/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi
btw, if the error tells that, trailer is missing.. where are you getting this error ?
Regards,
Moorthy

How to load unicode data files with fixed records lengths?

Hi!
To load unicode data files with fixed records lengths (in terms of charachters and not of bytes!) using SQL*Loader manually, I found two ways:
Alternative 1: one record per row
SQL*Loader control file example (without POSITION, since POSITION always refers to bytes!)<br>
LOAD DATA
CHARACTERSET UTF8
LENGTH SEMANTICS CHAR
INFILE unicode.dat
INTO TABLE STG_UNICODE
TRUNCATE
A CHAR(2) ,
B CHAR(6) ,
C CHAR(2) ,
D CHAR(1) ,
E CHAR(4)
) Datafile:
001111112234444
01NormalDExZWEI
02ÄÜÖßêÊûÛxöööö
03ÄÜÖßêÊûÛxöööö
04üüüüüüÖÄxµôÔµ Alternative2: variable length records
LOAD DATA
CHARACTERSET UTF8
LENGTH SEMANTICS CHAR
INFILE unicode_var.dat "VAR 4"
INTO TABLE STG_UNICODE
TRUNCATE
A CHAR(2) ,
B CHAR(6) ,
C CHAR(2) ,
D CHAR(1) ,
E CHAR(4)
) Datafile:
001501NormalDExZWEI002702ÄÜÖßêÊûÛxöööö002604üuüüüüÖÄxµôÔµ Problems
Implementing these two alternatives in OWB, I encounter the following problems:
* How to specify LENGTH SEMANTICS CHAR?
* How to suppress the POSITION definition?
* How to define a flat file with variable length and how to specify the number of bytes containing the length definition?
Or is there another way that can be implemented using OWB?
Any help is appreciated!
Thanks,
Carsten.

Hi Carsten
If you need to support the LENGTH SEMANTICS CHAR clause in an external table then one option is to use the unbound external table and capture the access parameters manually. To create an unbound external table you can skip the selection of a base file in the external table wizard. Then when the external table is edited you will get an Access Parameters tab where you can define the parameters. In 11gR2 the File to Oracle external table can also add this clause via an option.
Cheers
David

Upload huge file size: "The page cannot be displayed" browser

What are precautions to be take for uploading a huge file
I have an upload file operation in the web application. The web server is OC4J.
//UploadForm.jsp
<FORM NAME="InputForm" ACTION="UploadAction.jsp" METHOD="POST" enctype=multipart/form-data>
<input type="file" name="fileName">
</FORM>
After I deploy the application to the web server, if I upload a small file size, it is fine.
But if i upload huge data ,data is uploading but if any action button is clicked
After 30 seconds, it has error "The page cannot be displayed" shown on the web browser.

Hi All,
I have a similar time out problem, it ll be great if some one provide a solution.
The problem I'm facing is becasue of the time out setting in the proxy server which is 1.5 mins. The app server is sending Response properly but after some 2 mins or so because the processing is a bit complicated (3 API calls done intermittently for different validations).
I cant request for change in proxy server time out setting. So I have to handle this with in my application.
It will be greatful if anyone helps us out of this.
Thanks
Noufal

Sender File adapter File Conversion with key fields

Please help me guys, I already spent way too much time on this!
I checked a couple of BLOGs and I think I'm doing the right thing except I'm not getting the result I would expect.
I'm trying to create an IDoc using the from a flat file.
I get the following error:
During the application mapping com/sap/xi/tf/_MM_TEST_ a com.sap.aii.utilxi.misc.api.BaseRuntimeException was thrown: RuntimeException in Message-Mapping transformatio~
The Source file is this:
H
L0015
The file adapter does create the XML below (looks fine):
<ns:MT_SINGLE xmlns:ns="http://PPSC_TO_BO_INVOICE">
<HEADER>
     <ID>H</ID>
</HEADER>
<LINE>
     <ID>L</ID>
     <SITE>0015</SITE>
</LINE>
</ns:MT_SINGLE>
When I test the map in IR, with the XML that this file adapter created (above) it works fine!
Also if I take away the HEADER from the source structure everything works fine and I can create the IDoc, therefore I think something must be wrong in the file conversion, the mapping is so simple there is nothing to fail there.
The file conversion parameters that I have:
Document Name: MT_SINGLE
Document Namespace: http://PPSC_TO_BO_INVOICE
Document Offset:
Recordset Name:
Recordset Namespace:
Recordset Structure: HEADER,1,LINE,*
Recordset Sequence: Ascending
Recordset per Message:
Key Field name: ID
Key Field Type: String (Case-Sensitive)
LINE.fieldNames: ID,SITE
LINE.fieldFixedLengths: 1,4
LINE.keyFieldValue: L
HEADER.fieldNames: ID
HEADER.fieldFixedLengths: 1
HEADER.keyFieldValue: H
What am I doing wrong???
Any help is much appreciated.
Thanks,
Viktor Varga

Hi Viktor,
Basic rules:
1. Down load XML payload(output From Adapter) and test it in IR mapping program. If it is working there is no reason why it will fail in Runtime. Also Check Cache is up to date.
Generally the problem lies with namespace.
Hope this helps
Regards,
Satish

File conversion and append file in receiver file adapter

Hi
I have a batch IDoc -> XI(3.0) -> file scenario with file conversion to create a flat CSV-like file. The problem is that because of the amount of data sent from R/3 (~20000 employees), the idoc is split into several idocs. In my receiver file adapter I want to append all the idocs contained in this data transfer to one flat file, before transmitting to FTP server.
For the next batch transfer (the next day) XI should start all over and create a new file and append all idocs contained in that transfer before transmitting the flat file to FTP server.
Can this be done using the 'Use Temporary File' and 'Append' options in the FTP connection Parameters in the comm-channel?
It is not an option to append the file in the FTP-server, because the files in the destination folder are moved immediately after arrival, so the file will be moved before file adapter can append all the files.
Does anyone have any suggestions to how to solve this?
Br
Kenneth

Hi Kenneth
Did you ever get a solution to this problem?
Regards
Russel Irvine
[email protected]

Huge file size reduced by unlocking

I ended up with a 2.7GB file that consisted of 8 locked tracks that had been locked and unlocked lots of times. I knew that was about ten times the real size. The only way to get rid of whatever was causing the huge file size was to unlock the tracks and Save As Archive which reduced the size to 400MB.
Why did this happen?
Also, is there a detailed manual available that tells you all the tricks about GB? Things like: exactly what does locking a track do (Help is not very informative); and how do Save, Save As and Save As Archive differ in their treatment of purple, orange and other regions. I have had problems saving everything when I backup to an external drive, so I'd like to find out exactly how these Saves work.

Christoph,
Yes I just did that. I saved a copy of the song and then trashed the freeze files that should not have been there. The song played fine, no problem. THe file went from 1.6G to 176M.
I have found this situation in a few other files also.
These are my saving habits when I'm working on a song. I work for a while using "Save". Then after a while I start to save many major changes as "Save as Archive". I always end a session this way. Usually I'm not concerned about disc space. I use "Save as archive" all the time because I would rather have a file with everything I'm using contained within that file. Last year I accidently trashed 10,000 loops and I have since found that "save as archive" is a helpful backup when I'm acting stupid around the trash.
I may be one of the few folks who uses "Save as archive " so often and somehow it seems that every once in a while a hiccup occurs and the freeze files do not erase when the tracks are unlocked.
Since this discovery I have gone thru many other files and found this to be the case every now and then. Some of the files didnt hint at being over stuffed but had a few errant freeze files. Normally I would never go and check a files contents but that one file at 1.6G grabbed my attention
Just thought I would post to see if anyone has this problem.

XML file conversion after sender file content conversion

Hi,
I have issue refarding file content conversion.
My input structure is
<MT_RCICrecords>
<TRNH>
<RCIC>
<RECH>
<RECL>
<RCIC>
<TRNH>
Afetr sender File content conversion (csv to xml) it produces xml file as below (since file conversion does not support 3rd level of hierarchy)
<TRNH>
<RECH>
</RECH>
<RECL>
</RECL>
</TRNH>
It does not recognize RCIC.
Now i am trying to map this to IDOC and getting error as
'MT_RCICRecords tag found instead of IDOC BEGIN ='.
CAn anyone suggest me how to chaage this xml output after File content conversion to add RCIC tag in xml file?
I am new to XI so please give me some sample code to.
Thanks.
Yashpal
Its urgent!

My problem is xml generated from content conversion is like below
<MT_RCICrecords>
<TRNH></TRNH>
<RECH></RECH>
<RECL></RECL>
<TRLR></TRLR>
</MT_RCICrecords>
and i want it to be
<MT_RCICrecords>
<TRNH></TRNH>
<RCIC>
<RECH></RECH>
<RECL></RECL>
</RCIC>
<TRLR></TRLR>
</MT_RCICrecords>
which is not happening
My input message structure is
MT_RCICrecords
TRNH
RCIC
RECH
RECL
i hope it is clear now
TRLR

Unicode huge file conversion problem

Similar Messages

Maybe you are looking for