Corrupt XML file encoding utf-8 special chars (IDOC - File scenario)

Dear experts,
I have a problem with the XML output files of XI and could not find the answer in one of the current posts.
I'm sending Master Data from R/3 with IDOCs through XI to a FTP directory. These files include characters as Á, Ê, etc.
The XI server includes the utf-8 encoding in the output XML message. However, when opening these files I receive errors (tried it in multiple programs). It tells me that Á is not utf-8.
It will not accepts Á. I was under the impression that utf-8 included extended Latin and thus would accept these characters. Thus implying that the message was created wrong. Also importing these files into MDM import manager gives errors.
All rfc destinations are on Unicode.
By the way, we experience the same problem when syndicating files from the MDM server.
Any suggestions?
Cheers.
* Will reward points for helpful answers.

Hi,
Check out this guide..
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/502991a2-45d9-2910-d99f-8aba5d79fb42
make use of the messagetransformbean - http://help.sap.com/saphelp_nw04/helpdata/en/57/0b2c4142aef623e10000000a155106/content.htm
Also for further ref: go thru this thread - Change encoding from utf-8 to iso-8859-1 in JMS receiver!
regards
sasi.........
<b>Reward if useful</b>

Similar Messages

IDOC_AAE sender adapter XML : missing encoding="UTF-8"?

Hi All,
We have a IDOC to JMS scenario using IDOC_AAE as sender channel.
Symptom
The IDOC whihc we have received from ECC should add XML version and encoding information to the created IDOC XML file.
Other Terms
IT SHOULD BE AS
<?xml version="1.0" encoding="UTF-8"?>
But we are getting as <?xml version="1.0" >
Can you please share your views on this ?
Regards.,
Siva

Shiva,
Add XML AnonymizerBean in target channel module to replace/Add UTF encoding.
AF_Modules/XMLAnonymizerBean
Regards
Aashish Sinha

Receiver file adapter error for special char.

I am using MTB in my receiver file adapter. I am getting a special character from source system at end of the string. How to resolve this type of error u2013 please advice.
This error is not coming for every message.
Error message:
Column value xxxx Â too long - must stop, probably configuration error in file adapter
Thanks
Vick

Hi,
Itu2019s basically IDOC to file scenario.
This error is coming for name field.
What I am thinking at this stage to do changes in MTB by define parameters like (NameA.enclosureConversion)

Problem with reading special char from file

Hello Oracle community,
Got a problem when reading from a file. I am using a croatian keyboard and trying to read special charachters (ČĆŽŠĐ) from a file.
declare
l_file utl_file.file_type;
s varchar2(200);
begin
l_file := utl_file.fopen('test_dir', 'test.txt', 'R');
loop
    utl_file.get_line(l_file, s);
    dbms_output.put_line(s);
end loop;
exception
when no_data_found then
    utl_file.fclose(l_file);
end;But I keep getting this in dbms_output: ČĆĐ. For some reason it keeps skipping 2 chars Š and Ž. If I try to insert or update data the values are show correctly. What could be the cause of such a problem?
Best regards,
Igor

Hi Igor,
Looks like a NLS_LANGUAGE issue. Check the following threads:
UTL_FILE and NLS_LANG setting
Re: Arabic characters not displaying in Forms
Regards,
Sujoy

Rename Files while uploading, having special characters in File Name.

Hello Forum,
In our SharePoint environment, we are having many document library. Our users regularly uploads files to the documents library. As SharePoint doesn't allow to use "&" character in file name, it'll throw an exception regarding the file
name.
Now what I needed is :
Is there any way to remove "&" character while uploading files? Can we replace "&" with any other character while uploading files.
If is there any paid solution available, leave comment.
Thanks in Advance.

You have to change the folder/file names before hand then upload them to Sharepoint.
here is the script which will scan the files/folder and find the special charaters and fix for you.
http://get-spscripts.com/2011/11/use-powershell-to-check-for-illegal.html
Courtesy :Waqas
Sarwar(MCSE 2013)
Please 'propose as answer' if it helped you, also 'vote helpful' if you like this reply.

Append messages with out header tag " ?xml version="1.0" encoding="UTF-8"?

Hi all,
I am doing file to file scenario.When I use APPEND in the File adapter it is also adding <?xml version="1.0" encoding="UTF-8" ?>
I need to send a file for every 10 minutes consolidate all files and send at the end of the day.
<?xml version="1.0" encoding="UTF-8" ?>
<ID>31154</ID>
The next time when i send the file with different< ID>31155</ID>
it should append ignoring <?xml version="1.0" encoding="UTF-8" ?>
The consolidated file must look like this...
<?xml version="1.0" encoding="UTF-8" ?>
<ID>31154</ID>
< ID>31155</ID>
Thanks ,
Srinivas

Hey
as pointed out by everyone else,there is no straight way for this,one thing u can do is that create two separate scenarios.
in first scenario use content conversion on receiver side and keep on appending the text for 10 mins (guess this is ur pooling interval),now since ur using FCC u wont get <?xml version="1.0" encoding="UTF-8" ?>,you will get a flat file on receiver side.
after 10 mins u can have one more scenario which picks up this flat file and this time use FCC on sender side so that it will convert this flat file to XML,in this way you will get <?xml version="1.0" encoding="UTF-8" ?> only once.
hope this solves ur problem.
just make sure that you specify correct polling intervals for both the scenarios.
thanx
ahmad

How to Set "file.encoding" System Property to default "UTF-8"

When i execute my code some special character are not being display correct so by programming approach i am trying to set "file.encoding" system property to "UTF-8", using command System.setProperty( "file.encoding", "UTF-8" ); and it is not working.
If i run my jar using command java -Dfile.encoding=UTF-8 -jar myprog.jar . It is working and my special characters are also looking in right way.
Can i set this defalut encoding by programming approach.
Thanks
Ashish Pancholi

Hello,
I have the same problem. I have a java prog that is started with "-Dfile.encoding=ISO-8859-1". Now in this program I want to print some characters using the UTF-8 encoding because I know that the terminal I will be printing on has this encoding. I tried using InputStramReader without success:
    InputStreamReader isr = new InputStreamReader(new ByteArrayInputStream("Müller".getBytes()), "UTF-8");
    BufferedReader br = new BufferedReader(isr);
    String line = null;
    while ((line = br.readLine()) != null) {
        System.out.println(line);
    }EDIT:
the above example is to read something into my java program. If I want to write something from my java class to an output it goes like this:
Writer out = new BufferedWriter(new OutputStreamWriter(System.out, "UTF8"));
out.write("Müller\n");
out.flush();... in that case I get the correct encoding.
Thanks,
T

Handling special char at BizTalk end?

I am receiving message as flat file which may contain special char like <> ' "....... etc
Can any one tell me how can i handle it at BizTalk end.
If i am converting it into xml due to data, it is not a valid xml.
<FirstName> <John miller</FirstName>
And single quotes ' can create problem in sql query .
Can any one let me know how could i handle it.

Hi Phill,
Let’s be clear when you mean by “special character” in flat-file schema.
If the special character which your referring are not the delimiters used in your schema or any escape characters (about escape characters I’ll refer later), then as la
Cour said, XML shall handle them without any issues. If you have these characters, as you highlighted, you will have XML file/element like “<FirstName>
<John miller</FirstName>”. It’s still a valid XML file.
But if you don’t want this characters to be saved into SQL database i.e for the above XML element if don’t want the character “<” saved along with “John
miller” as “<John miller”, then you need to have a custom pipeline component to remove those characters.
Escape character:
If you know that a character which could come in an element, then you can use “Escape Character” property for the element where you can specify the character you want to
ignore. i.e of you expect character “<” to come with FirstName element, then you can specify the escape character property as “<” in the “FirstName” element.
More special character to ignore:
As said XML can parse these characters without issue. But if you don’t want to send those characters to your destination system (SQL), then after parsing the flat-file into
XML, use a custom pipeline component to replace all the special character from the parsed XML.
Following article discuss about this concept:
How
to remove invalid character in incoming XML message using custom pipeline component
If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.

HOW SPECIFY FILE.ENCODING=ANSI FORMAT IN J2SE ADAPTER.

Hi All,
we are using j2se plain adapter, we need the outputdata in ANSI FORMAT.
Default file.encoding=UTF-8
how to achive this.
thanks in advance.
Regards,
Mohamed Asif KP

File adapter would behave in a similar fashion on J2ee. Providing u the link to ongoing discussion
is ANSI ENCODING possible using file/j2see adapter
Regards,
Prateek

CSV File in UTF-8

Hi all,
I want some clarification about CSV file. we using CSV file to upload data from us front end site to db(oracle db). First we read data from CSV file via java CSV reader, the problem is starts from here. The CSV file may be contain some foreign character (i.e. chines, Spanish, Japanese, hindi, arabic), We need to create the CSV file as UTF-8 format.So,
Step - 1 :we create a excel file and save as CSV format
Step - 2 the same file in open Edit+ and change the utf-8
Mean the file has been read in java CSV reader.
Otherwise suppose i changed the second step that is the file open in notepad and convert utf-8 mean this same file not recognized to read in java CSV reader
Please give some idea how to create CSV file in UTF-8 charset

If your input file is in csv format, try importing it directly into the database using SQLDeveloper. If you have a table created already, riight click on the table in the Connections navigator and select import data. On the first page of the wizard, select the correct encoding from the Encoding list. You should see the characters in the file displayed correctly in the bottom of the page. Select the other options like format, delimiiters and line terminators. When these options are specified correctly, you should see the file displayed as rows and columns in the bottom of the screen. Continue with the import using the wizard. If the table is not already created, you can right click on the table folder in the Connections navigator. The second page of the wizard will allow you to enter a new table name, the wizard will automatically create a column in the table for each column in the file and you can refine the column definitions using step 4, the Column Definition panel.
Joyce Scapicchio
SQLDeveloper Team

File.encoding in windows influence by the locale

How can I set the file.encoding in windows platform that will not be influence by the locale.
For example, in the Control Panel->Regional Options the locale is set to Russian
and what I get is that I use file.encoding Cp1251 even though I pass the parameter in the command line
-Dfile.encoding=Cp1252 (I want to keep Cp1251,
Cp1251 is US westen default and Cp1252 is Windows Cyrillic)
I run java program to see what encoding I use
D:\ProgramFiles\jdk1.3.1\bin\java -Dfile.encoding= Cp1252 TestEncodingThe locale in my pc is Russian and the result is:
System.getProperty("file.encoding") == Cp1252
Default ByteToChar Class == sun.io.ByteToCharCp1251
Default CharToByte Class == sun.io.CharToByteCp1251
Default CharacterEncoding == Cp1251
OutputStreamWriter encoding == Cp1251
InputStreamReader encoding == Cp1251
TestEncoding.java
import java.io.PrintStream;
import java.io.ByteArrayOutputStream;
import java.io.OutputStreamWriter;
import java.io.InputStream;
import java.io.ByteArrayInputStream;
import java.io.InputStreamReader;
class TestEncoding{
public static void main(String[] args) {
String encProperty = System.getProperty("file.encoding");
System.out.println("System.getProperty(\"file.encoding\") == " + encProperty);
String byteToCharClass = sun.io.ByteToCharConverter.getDefault().getClass().getName();
System.out.println("Default ByteToChar Class == " + byteToCharClass);
String charToByteClass = sun.io.CharToByteConverter.getDefault().getClass().getName();
System.out.println("Default CharToByte Class == " + charToByteClass);
String defaultCharset = sun.io.ByteToCharConverter.getDefault().getCharacterEncoding();
System.out.println("Default CharacterEncoding == " + defaultCharset);
ByteArrayOutputStream buf = new ByteArrayOutputStream(10);
OutputStreamWriter writer = new OutputStreamWriter(buf);
System.out.println("OutputStreamWriter encoding == " + writer.getEncoding());
byte[] byteArray = new byte[10];
InputStream inputStream = new ByteArrayInputStream(byteArray);
InputStreamReader reader = new InputStreamReader(inputStream);
System.out.println("InputStreamReader encoding == " + reader.getEncoding());

What are you really trying to accomplish? Applications should avoid relying on undocumented or implementation dependent features, such as the file.encoding property and sun.* classes (see http://java.sun.com/products/jdk/faq/faq-sun-packages.html).
On the other hand, there's plenty of documented public API that lets you work with specific character encodings. For example, you can specify the character encoding for conversion between byte arrays and String objects (see the String class specification) or when reading or writing files (see the InputStreamReader and OutputStreamWriter classes in java.io).
The default encoding is needed by the Java runtime when accessing the Windows file system, for example file names, so changing it would likely result in erroneous behavior.
Norbert Lindenberg

Remove ?xml version="1.0" encoding="UTF-8"? from xml file

I have generated an XML file using sax paraser. In the XML file thats generated I have the version and the encoding line
<?xml version="1.0" encoding="UTF-8"?>which is automatically generated in my XML file. Is there any way that I can avoid that from the XML file thats generated.

try
                        FileWriter fr = new FileWriter(new File(path, fileName));
                        Document docNode = docNodeMap.get(name);
                        XMLOutputter outputter = new XMLOutputter();
                        outputter.output(docNode, fr);
                        fr.close();
                catch (IOException e)
                        e.printStackTrace();
                }this the code generating xml file.

"encoding = UTF-8" missing while writing XML file using file Adapter

Hi,
We are facing an unique problem writing xml file using file adapter. The file is coming without the encoding part in the header of xml. An excerpt of the file that is getting generated:
<?xml version="1.0" ?>
<customerSet>
<user>
<externalID>51017</externalID>
<userInfo>
<employeeID>51017</employeeID>
<employeeType>Contractor</employeeType>
<userName/>
<firstName>Gail</firstName>
<lastName>Mikasa</lastName>
<email>[email protected]</email>
<costCenter>8506</costCenter>
<departmentCode/>
<departmentName>1200 Corp IT Exec 8506</departmentName>
<businessUnit>1200</businessUnit>
<jobTitle>HR Analyst 4</jobTitle>
<managerID>49541</managerID>
<division>290</division>
<companyName>HQ-Milpitas, US</companyName>
<workphone>
<number/>
</workphone>
<mobilePhone>
<number/>
</customerSet>
</user>
So if you see the header the "encoding=UTF-8" is missing after "version-1.0".
Do we need to configure any properties in File Adapter?? Or is it the standard way of rendering by the adapter.
Please advice.
Thanks in advance!!!

System.out.println(nodeList.item(0).getFirstChild().getNodeValue());

Outbound XML file requires attribute encoding="utf-8" in header tag

Hi All-
I am creating xml file as an outbound transaction.
BPEL process creating the XML file with header
<?xml version="1.0" ?>
Is it possible that I can add the attribute encoding="utf-8" in it as below:
<?xml version="1.0" encoding="utf-8" ?>
Any idea, please help.
Thanks and Regards,
Sreejit

Hi,
I need the same , any help on this?
--Khaleel

OUTBOUND XML FILE CONTAINS SPECIAL CHARS AFTER SP08 UPGRADE

Experts,
Recently MDM server upgraded to SP08 from SP05.
In our outbound interface scenario we have 1 hierarchy field. The hierarchy field mapped as complete path(parent to Child)
ex: 1,Parent > 11,Child > 111,Child > 1111,Child.
In SP05 XML files are generating correctly & PI also dont have any issues while passing this data to ECC.
After SP08 Server pack upgrade, xml file generating like below:
1??, ??Parent??, ???>?11??, ??Child??, ???>?111??, ??Child??, ???>?1111??, ??Child??, ???
If open the xml file in non ASCC editor I could able to see these special chars. Due to this ISSUE PI could not processing xml files.
All XML files are failing& blocking in PI.
Could you let me know what needs to be done at MDM or PI level.
Appreciate your inputs.
Thanks
Audinarayana

Hello,
Please check the Destination preview in the syndicator.
So an syndication on the local machine, open the XML in an Browser, to see check for the special characters.
If every things is ok, then just place the file in the outbound ready folder.
If not then, raise an OSS message with SAP, as this a latest release, error resolution would be best provided by SAP
Regards,
Abhishek

Corrupt XML file encoding utf-8 special chars (IDOC - File scenario)

Similar Messages

Maybe you are looking for