Convert from ISO 8859-1 encoding to UTF-8

Hi
My Os name is 'SunOS ut51109 5.10 Generic_144500-19 sun4v sparc SUNW,T5440'.
I want to change the encoding of the existing .bcp file from ISO 8859-1 to UTF-8 with out using any temp files as these .bcp file will be pointed by an external table.
here is the command I issued in the script(ksh file)
iconv -f ISO8859-1 -t UTF-8 file1.bcp > file1.bcp
After the script got executed file1 is empty(showing 0 bytes).
Please correct me or let me know the syntax to be followed to write the data into the same file in UTF-8 format.
Thanks
kartheek

You cannot do conversion from ISO 8859-1 to UTF-8 in-place because the UTF-8 version will generally be longer (unless you convert a pure ASCII file, which does not need conversion in the first place). Therefore, you would have to overwrite what you have not read yet. Instead, convert to a new file with a temporary name, drop the original and rename the temporary back to original. This is not that complicated.
If the problem is that you want to overwrite a file already open by the database, then rename the incoming file first and then convert copying to the target.
-- Sergiusz

Similar Messages

[svn:fx-trunk] 7661: Change from charset=iso-8859-1" to charset=utf-8" and save file with utf-8 encoding.

Revision: 7661
Author:   [email protected]
Date:     2009-06-08 17:50:12 -0700 (Mon, 08 Jun 2009)
Log Message:
Change from charset=iso-8859-1" to charset=utf-8" and save file with utf-8 encoding.
QA Notes:
Doc Notes:
Bugs: SDK-21636
Reviewers: Corey
Ticket Links:
    http://bugs.adobe.com/jira/browse/iso-8859
    http://bugs.adobe.com/jira/browse/utf-8
    http://bugs.adobe.com/jira/browse/utf-8
    http://bugs.adobe.com/jira/browse/SDK-21636
Modified Paths:
    flex/sdk/trunk/templates/swfobject/index.template.html

same problem here with wl8.1
have you sold it and if yes, how?
thanks

How to store UTF-8 characters in an iso-8859-1 encoded oracle database?

How can we store UTF-8 characters in an iso-8859-1 encoded oracle database? We can NOT change the database encoding but need to store e.g. Polish or Russian characters besides other European languages.
Is there any stable sollution with good performance?
We use Oracle 8.1.6 with iso-8859-1 encoding, Bea WebLogic 7.0, JDK 1.3.1 and the following thin driver: "Oracle JDBC Driver version - 9.0.2.0.0".

There are a couple of unsupported options, but I wouldn't consider using them on a production database running other critical applications. I would also strongly discourage their use unless you understand in detail how Oracle National Language Support (NLS) works, otherwise you could end up with corrupt data or worse.
In a sense, you've been asked to do the impossible. The existing databas echaracter sets do not support encoding the data you've been asked to store.
Can you create a new database with an appropriate database character set and deploy your application there? That's probably the easiest solution.
If that isn't an option, and you really need to store data in this database, you could use one of the binary data types (RAW and BLOB), but that would mean that it would be exceptionally difficult for applications other than yours to extract the data. You would have to ensure that the data was always encoded in the same character set, otherwise you wouldn't be able to properly decode it later. This would also add a lot of complexity to your application, since you couldn't send or recieve string data from the database.
Unfortunately, I suspect you will have to choose from a list of bad options.
Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC

Cannot change charset from 'ISO-8859-1' to 'UTF-8' ?

I've programed a web application with Facelets. As I start the first page "index.html" I got the following exception:
"java.lang.IllegalStateException: Attempt to change ContentType after calling getWriter() (cannot change charset from 'ISO-8859-1' to 'UTF-8')"
The header of the "index.html" looks as follow:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<meta http-equiv="refresh" content="0; url=./guess.jsf" />
<title>Untitled Document</title>
</head>
What's wrong? Is it a bug of weblogic?

same problem here with wl8.1
have you sold it and if yes, how?
thanks

WSDL Generating in ISO-8859-1 encoding want in UTF - 8

Hi XI Geeks,
       When I am using "Define Web Service" tool in Integration Directory, the WSDL file is being created with encoding <b>ISO-8859-1</b>, <i><b>but we want in "UTF-8"</b></i>.
        Is there any setting we need to change in WAS which will change this default.
Any ideas are welcome and thanks in advance.
Regards
Sujan

Hi Vishnu,
         We already did that for the time being. But the client wants it be consistent.
         In another client engagement the WSDL is created in "UTF-8" format we double checked twice.
         We are running on SP12.
If any of the SP12 guys can generated a WSDL and let us know that it is created in "ISO-8859-1" encoding.
Regards
Sujan

File adapter ISO-8859-1 encoding problems in XI 3.0

We are using the XI 3.0 file adapter and are experiencing some XML encoding troubles.
A SAP R/3 system is delivering an IDoc outbound. XI picks up the IDoc and converts it to an external defined .xml file. The .xml file is send to a connected ftp-server. At the remote FTP server the file is generating an error, as it is expected to arrive in ISO-8859-1 encoding. The Transfer Mode is set to Binary, File Type Text, and Encoding ISO-8859-1.
The .xml file is encoded correctly in ISO-8859-1, but the problem is that the XML encoding declaration has the wrong value 'UTF-8'.
Does anybody know of a work around, to change the encoding declaration to ISO-8859-1 in the message mapping program?

An example of the XSL code might be as follow:
<?xml version='1.0'?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method='xml' encoding='ISO-8859-1' />
<xsl:template match="/">
<xsl:copy-of select="*" />
</xsl:template>
</xsl:stylesheet>

Java App on Linux : Unable to read iso-8859-1 encoded file correctly.

I have a file which is encoded as iso-8859-1, and contains characters such as ô .
I am reading this file with java code, something like:
File in = new File("myfile.csv");
InputStream fr = new FileInputStream(in);
byte[] buffer = new byte[4096];
while (true) {
int byteCount = fr.read(buffer, 0, buffer.length);
if (byteCount <= 0) {
break;
String s = new String(buffer, 0, byteCount,"ISO-8859-1");
System.out.println(s);
However the ô character is always garbled, usually printing as a ? .
I am running this on a Linux machine. It works fine on my XP machine.
I have verified that I can see the correct characters when I cat the file on the terminal.
(Interestingly, but I think maybe only by co-incidence, it works when I run with the -Dfile.encoding=UTF16 option, but not with UTF8, although this appears a hack rather than a fix since this option was not intended for developer use by sun - but I thought mentioning it may provide some clues as to what is going on)

I think your main probelm is with the console. When you send text to the console, it's sent in the system default encoding. On an English-locale system that might be ASCII, ISO-8859-1, windows-1252, UTF-8, MacRoman, and probably several other possibilities. Then the console decodes the the bytes using whatever encoding it feels like using--on my WinXP machine, it uses cp437 by default (just for laughs, as far as I can tell). If the text happens to be pure, seven-bit ASCII, there's no problem, since all those encodings are identical in that range.
But if you need to output anything other than ASCII characters, avoid the console. Send the output to a file and specify an encoding that you know will be able to handle your characters--UTF-8 can handle anything. Then open the file with an editor that can read that encoding; most of them can handle UTF-8 these days, and many will even detect it automatically. You also need to be using a font that can display your characters.
However, you're also going about the reading part wrong. Instead of reading the text in as bytes and passing them to a String constructor, you should use an InputStreamReader and read it as text from the beginning: BufferedReader br = new BufferedReader(
new InputStreamReader(
new FileInputStream("myfile.csv"), "ISO-8859-1"));I am curious about your statement that "it works" when you run with the -Dfile.encoding=UTF16 option. I wouldn't be surprised to see it output the correct characters (ASCII characters, anyway), but I would expect to see the characters interspersed with blank spaces or rectangles.

Setting ISO-8859-1 Encoding in AXIS RPC call

As per my understanding UTF-8 is default for webservice call in AXIS implementation I may be wrong.
Is there any way to set ISO-8859-1 encoding in AXIS RPC call?
Below is the WSDL file details.
Thanks for you help.
Regards,
Hemen
<Code>
<?xml version="1.0" encoding="ISO-8859-1"?>
<definitions name="TestService_Service"
targetNamespace="http://webservice.de.rt.wsdl.WebContent/TestService_Service/"
xmlns="http://schemas.xmlsoap.org/wsdl/"
xmlns:format="http://schemas.xmlsoap.org/wsdl/formatbinding/"
xmlns:java="http://schemas.xmlsoap.org/wsdl/java/"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns:tns="http://webservice.de.rt.wsdl.WebContent/TestService_Service/" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<message name="getResponseRequest">
<part name="xml" type="xsd:string"/>
<part name="product" type="xsd:string"/>
<part name="source" type="xsd:string"/>
<part name="tran" type="xsd:string"/>
<part name="content" type="xsd:string"/>
</message>
<message name="getResponseResponse">
<part name="return" type="xsd:string"/>
</message>
<portType name="TestService">
<operation name="getResponse" parameterOrder="xml product source tran content">
<input message="tns:getResponseRequest" name="getResponseRequest"/>
<output message="tns:getResponseResponse" name="getResponseResponse"/>
</operation>
</portType>
<binding name="TestServiceJavaBinding" type="tns:TestService">
<java:binding/>
<format:typeMapping encoding="Java" style="Java">
<format:typeMap formatType="java.lang.String" typeName="xsd:string"/>
</format:typeMapping>
<operation name="getResponse">
<java:operation methodName="getResponse"
parameterOrder="xml product source tran content" returnPart="return"/>
<input name="getResponseRequest"/>
<output name="getResponseResponse"/>
</operation>
</binding>
<binding name="TestServiceBinding" type="tns:TestService">
<soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
<operation name="getResponse">
<soap:operation soapAction="" style="rpc"/>
<input name="getResponseRequest">
<soap:body
encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
namespace="distributionengine.com"
parts="xml product source tran content" use="encoded"/>
</input>
<output name="getResponseResponse">
<soap:body
encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
namespace="distributionengine.com" use="encoded"/>
</output>
</operation>
</binding>
<service name="TestServiceService">
<port binding="tns:TestServiceJavaBinding" name="TestServiceJavaPort">
<java:address className="rt.de.webservice.TestService"/>
</port>
</service>
<service name="TestService_Service">
<port binding="tns:TestServiceBinding" name="TestServicePort">
<soap:address location="http://localhost:9080/servlet/rpcrouter"/>
</port>
</service>
</definitions>
</code>

try to put the below code at the top of the JSP page
<%@ page language="java" pageEncoding="UTF-8"%>

IllegalStateException: (cannot change charset from 'ISO-8859-1' to 'null')

          My index.jsp page has the directive <%@ page contentType="text/html;charset=ISO-8859-1"
          %>.
          It uses my templating tag-library, which uses a requestDispatcher to include my
          template.
          My template doesn't have this contentType directive.
          Then I get an exception :
          java.lang.IllegalStateException: Attempt to change ContentType after calling getWriter()
          (cannot change charset from 'ISO-8859-1' to 'null')
          It actually works perfectly on WLS6.0.
          Here is the complete stacktrace which shows the path page -> tag -> include
          java.lang.IllegalStateException: Attempt to change ContentType after calling getWriter()
          (cannot change charset from 'ISO-8859-1' to 'null')
               at weblogic.servlet.internal.ServletResponseImpl.setEncoding(ServletResponseImpl.java:695)
               at weblogic.servlet.internal.ServletResponseImpl.setHeader(ServletResponseImpl.java:566)
               at weblogic.servlet.internal.ServletResponseImpl.setContentType(ServletResponseImpl.java:236)
               at weblogic.servlet.internal.ServletRequestImpl.reportJSPFailure(ServletRequestImpl.java:162)
               at weblogic.servlet.internal.ServletRequestImpl.reportJSPTranslationFailure(ServletRequestImpl.java:168)
               at weblogic.servlet.jsp.JspStub.reportTranslationFailure(JspStub.java:503)
               at weblogic.servlet.jsp.JspStub.compilePage(JspStub.java:317)
               at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:181)
               at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:149)
               at weblogic.servlet.internal.ServletStubImpl.getServlet(ServletStubImpl.java:344)
               at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:240)
               at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:200)
               at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:327)
               at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:207)
               at bv2.template.TemplateRequest.doStartTemplate(TemplateRequest.java:56)
               at bv2.template.BodyTag.doStartTag(BodyTag.java:31)
               at jsp_servlet._index._jspService(_index.java:173)
               at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
               at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:245)
               at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:200)
               at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2279)
               at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:1923)
               at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
               at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)


[att1.html]

How to convert back from UTF8 to ISO-8859-1 encoding?

hi,
I have a bunch of XML files which were wrongly encoded, and we lost all our accent characters.
ie: é become Ã©
so how can I recover my XML files using powershell?
so I want to change all the UTF8 ecoded characters back to the original ISO accent character
Ã© -> é
I try this:
1")
$utf8 = [System.text.Encoding]::UTF8
$utfBytes = $utf8.GetBytes("Ã©")
$isoBytes = [System.text.Encoding]::Convert($utf8, $iso, $utfBytes)
$iso.GetString($isoBytes)
but doesnt works.
so is there a way to do this in powershell?
I have to scan hundreds of files...
thanks.

You can't. UTF-8 strips all of the information from the characters so you cannot know which characters are which. If you know which characters you need to fix (requires knowing the spelling of the words) you could possible develop an matrix of
replacements. There is no simple one line method.
¯\_(ツ)_/¯

XML data encoding iso-8859-1 . Currently utf-16 is default encoding

Hello ABAP Gurus ,
Need a help from you .
Scenario : We have SAP4.7 enterprise version which we have now converted to Unicode system . There is a BSP application which talks to an external web application (Non Unicode) thru HTTP protocol and sends data thru XML.
Problem : Problem is at the time when BSP application prepares the XML While preparing XML data , before converting to Unicode environment the encoding was "iso-8859-1" . But now after Unicode conversion , the encoding is "UTF-16".
The XML data looks like
<?xml version="1.0" encoding="utf-16"?><DATA><ACTION>CREATE_TICKET</ACTION><CRI
I have tried replacing "utf-16" by "iso-8859-1 " . The interface works . But at the recieving end , ie external web application , the German umlauts appear as some garbage values .
I know we need to enforce encoding . I have tried with the following code but could not suceed . The encoding appears as "utf-16".
Following is the section of code which I have written in BSP application.
Daten in DOM-Baum wandeln
    CALL FUNCTION 'SDIXML_DATA_TO_DOM'
      EXPORTING
        name        = 'DATA'
        dataobject = ls_cr_xml
      IMPORTING
        data_as_dom = if_dom
      CHANGING
        document    = if_document
      EXCEPTIONS
        OTHERS      = 1.
    IF sy-subrc NE 0.
      error_out text-f47 text-f48 space space.
    ENDIF.
DOM-Baum in Character-Stream wandeln
    if_pixml = cl_ixml=>create( ).
    IF if_pixml IS INITIAL.
      error_out text-f50 text-f53 space space.
    ENDIF.
    if_pstreamfact = if_pixml->create_stream_factory( ).
    IF if_pstreamfact IS INITIAL.
      error_out text-f51 text-f53 space space.
    ENDIF.
    if_postream = if_pstreamfact->create_ostream_cstring( string = xml_doc ).
    IF if_pstreamfact IS INITIAL.
      error_out text-f52 text-f53 space space.
    ENDIF.
--Encoding--
data: gv_str type string.
data: gv_l_xml_encoding type ref to if_ixml_encoding.
data gv_l_resultb type boolean.
gv_str = 'ISO-8859-1' .
clear gv_l_xml_encoding.
      call method if_pixml->create_encoding
        EXPORTING
          byte_order    = 0
          character_set = gv_str
        RECEIVING
          rval          = gv_l_xml_encoding.
      clear gv_l_resultb.
      call method gv_l_xml_encoding->set_character_set
        EXPORTING
          charset = gv_str
        RECEIVING
          rval    = gv_l_resultb.
        call method if_document->set_encoding
        EXPORTING
          encoding = gv_l_xml_encoding.
----Append child -
CALL METHOD if_document->append_child
      EXPORTING
        new_child = if_dom
      RECEIVING
        rval      = lv_return.
    IF lv_return NE 0.
      error_out text-f47 text-f49 space space.
    ENDIF.
--Render the XML data to output stream--
    CALL METHOD if_document->render
      EXPORTING
        ostream = if_postream.
After this section of code is executed , the variable xml_doc gets filled up with XML data .
<?xml version="1.0" encoding="utf-16"?><DATA><ACTION>CREATE_TICKET</ACTION><CRI
Can anybody help how to enforce encoding ?
Regards,
Laxman Nayak.

Hi Aslam Riaz,
Did you find any solution..?
Kindly help me on this issue.
Thanks and Regards,
Shailaja Chityala

UTF-8 encoding vs ISO 8859-1 encoding

The iTunes tech specs call for UTF-8 encoding of the XML feed file; a friend of mine uses feed generator software through his blog that uses ISO 8859 encoding. Is there a way to convert the latter to UTF-8 so that iTunes tags may be successfully added?
When I tried editing his XML file, I got error messages when I submitted the file to RSS feed validator sites (such as http://feedvalidator.org/. Any help or knowledge is appreciated because I am not the least bit expert in this coding arena.

You don't need to convert iso 8859-1 (us-ascii) to utf-8 unless you have nonstandard characters. Basically, ascii is a subset of utf-8 and for English it will serve you just fine. You can have iTunes tags in the xml file even if the file itself is encoded in iso 8859-1.
The error you see at feedvalidator.org is most likely a warning.
Hope this helps!
- Andy Kim
Potion Factory
http://www.potionfactory.com

How to switch DB string storage format from/to UTF-8 to/from ISO 8859-1 ?

As far as I understand strings in tables are stored by default in an Oracle DB in ISO 8859-1 format.
How can I switch the storage to UTF-8 format?
Do I have to change just a parameter (which ?) or do I have to setup/install the whole DB again?
If just a parameter switching is necessary:
How can I change already existing strings from one format to another?
Does it take place automatically or do I have to issue and explicite convert command (which ?).
Peter

Please refer to
http://download.oracle.com/docs/cd/B19306_01/server.102/b14225/ch2charset.htm#sthref157
And
http://download.oracle.com/docs/cd/B19306_01/server.102/b14225/ch11charsetmig.htm#sthref1476
Sybrand Bakker
Senior Oracle DBA

Encoding autodetection chooses ISO-8859-5 instead of UTF-8

A certain site producing HTML in UTF-8 without meta tags. The Firefox detects the encoding as ISO-8859-5, an absolutely dead and useless 'standard'. How to exclude it from the autodetection and maki it proper unicode?

It's a strange guess, if it's a guess. Are you sure the server isn't sending a header specifying that encoding? To view the content-type header, you can use an add-on (or if it's not a secure page, an external proxy):
* Live HTTP Headers extension: https://addons.mozilla.org/en-us/firefox/addon/live-http-headers/
* Firebug extension: https://addons.mozilla.org/en-US/firefox/addon/firebug/
* Fiddler debugging proxy: http://www.fiddler2.com/fiddler2/

Problems with reading XML files with ISO-8859-1 encoding

Hi!
I try to read a RSS file. The script below works with XML files with UTF-8 encoding but not ISO-8859-1. How to fix so it work with booth?
Here's the code:
import java.io.File;
import javax.xml.parsers.*;
import org.w3c.dom.*;
import java.net.*;
* @author gustav
public class RSSDocument {
    /** Creates a new instance of RSSDocument */
    public RSSDocument(String inurl) {
        String url = new String(inurl);
        try{
            DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
            Document doc = builder.parse(url);
            NodeList nodes = doc.getElementsByTagName("item");
            for (int i = 0; i < nodes.getLength(); i++) {
                Element element = (Element) nodes.item(i);
                NodeList title = element.getElementsByTagName("title");
                Element line = (Element) title.item(0);
                System.out.println("Title: " + getCharacterDataFromElement(line));
                NodeList des = element.getElementsByTagName("description");
                line = (Element) des.item(0);
                System.out.println("Des: " + getCharacterDataFromElement(line));
        } catch (Exception e) {
            e.printStackTrace();
    public String getCharacterDataFromElement(Element e) {
        Node child = e.getFirstChild();
        if (child instanceof CharacterData) {
            CharacterData cd = (CharacterData) child;
            return cd.getData();
        return "?";
}And here's the error message:
org.xml.sax.SAXParseException: Teckenkonverteringsfel: "Malformed UTF-8 char -- is an XML encoding declaration missing?" (radnumret kan vara f�r l�gt).
    at org.apache.crimson.parser.InputEntity.fatal(InputEntity.java:1100)
    at org.apache.crimson.parser.InputEntity.fillbuf(InputEntity.java:1072)
    at org.apache.crimson.parser.InputEntity.isXmlDeclOrTextDeclPrefix(InputEntity.java:914)
    at org.apache.crimson.parser.Parser2.maybeXmlDecl(Parser2.java:1183)
    at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:653)
    at org.apache.crimson.parser.Parser2.parse(Parser2.java:337)
    at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:448)
    at org.apache.crimson.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:185)
    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:124)
    at getrss.RSSDocument.<init>(RSSDocument.java:25)
    at getrss.Main.main(Main.java:25)

I read files from the web, but there is a XML tag
with the encoding attribute in the RSS file.If you are quite sure that you have an encoding attribute set to ISO-8859-1 then I expect that your RSS file has non-ISO-8859-1 character though I thought all bytes -128 to 127 were valid ISO-8859-1 characters!
Many years ago I had a problem with an XML file with invalid characters. I wrote a simple filter (using FilterInputStream) that made sure that all the byes it processed were ASCII. My problem turned out to be characters with value zero which the Microsoft XML parser failed to process. It put the parser in an infinite loop!
In the filter, as each byte is read you could write out the Hex value. That way you should be able to find the offending character(s).

Convert from ISO 8859-1 encoding to UTF-8

Similar Messages

Maybe you are looking for