ODBC and UTF8 charset

Win7 x32
Oracle XE 11g
How to set a UTF8 charset?
My script is connecting via PDO ODBC: Driver={Oracle in XE};Dbq=mydb;Uid=user;Pwd=psw;
The registry NLS_LANG is set to AMERICAN_AMERICA.AL32UTF8
Windows locale is set to US.
When I insert word "тест" it looks like "С‚РµСЃС‚" in DB.

Looks like you have a presentation problem. Version of SQL Developer used?I use v3.1.07 but actually there isn't problem with the SQL developer tool because when I use it for seeing, inserting/updating the data it works correctly. There is a problem when I try to insert or update (select ok) from a php script. I update a table from php script (which is utf8 encoded) using PDO ODBC driver but it saves the data in windows 1251. If I encode the php script to windows 1251 than it saves it in utf8. Is it possible that ODBC driver converts the charset while inserting or updating method?
PS: Before all I migrated all data from MySQL DB (which was utf8 encoded) to Oracle DB using SQL developer tool. I suppose it was migrated without converting the charset.

Similar Messages

SQL Loader UTF8 Charset issue

Hi,
My database charset is UTF8, NLS_LANG parameter on mid tier (unix) is UTF8. I have created a spool file (Korean language) in UTF8 charset. Also, verified the dat file using a unicode text editor. When I am trying to upload data back to the database using SQL loader data is not getting uploaded properly. It uploads after doing some internal encoding/decoding. I have tried with characterset parament in ctrl file as UTF8.
Could anyone pls help me identifying the issue and respective resolution?
Thanks in advance.
Sanyog

Also useful for you (NLS_LANG and sqlloader) may be this thread:
sqllder especial carector issue

Discussion Forum Portlet - Problems with JAVA and UTF8?

Hi
I installed the Discussion Forum Portlet successfully. It also seems that almost everything works fine. There's only a problem if I have new posts that include special German characters (Umlaute) like ä, ö, ü or special French characters like é, è or ç. They are saved correctly in the table but if you view the post the characters are not displayed correctly.
Example
input: ça va?
result: Ã§a va?
I know that there are problems with Java and UTF8 Database. Is there a possibility to change this (bug)?
Regards
Mark

Here's what I got. I don't see anything that helps but I'm kinda new to using SQL and java together.
D:\javatemp\viddb>java videodb
Problem with SQL java.sql.SQLException: [Microsoft][ODBC Microsoft Access Driver
] Syntax error in CREATE TABLE statement.
Driver Error Number-3551
java.sql.SQLException: [Microsoft][ODBC Microsoft Access Driver] Syntax error in
CREATE TABLE statement.
at sun.jdbc.odbc.JdbcOdbc.createSQLException(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.standardError(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.SQLExecDirect(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcStatement.execute(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcStatement.executeUpdate(Unknown Source)
at videodb.main(videodb.java:31)
D:\javatemp\viddb>

AL32UTF8 and UTF8

On Oracle 8i I was using the utf-8 charset, when I'm installing Oracle 9.2 I can find AL32UTF8 and utf8 without 4 byte chars
Which one should I use, what is the equivalent to the 8i utf-8
Tal Olier (otal_mercury.co.il)

The UTF-8 character set is available in both 8i & 9i. It's the same character set in both.
UTF-8 does not have 4 byte characters-- characters in UTF-8 require between 1 and 3 bytes of storage. UTF-16 is, IIRC, a new character set in 9i where characters are stored in either 2 or 4 bytes.
Justin

ODBC and XA

Hi all.
We are thinking about using ODBC in our tuxedo services to make easy the migration
between one database and other of a different vendor. We also need to use XA.
Have some of you information about using ODBC and XA, (conception, problems, ...)
and how many DB ODBC drivers support it?
Thx in advance.

I suspect you're not going to be able to retrieve Spatial data via ODBC. The ODBC protocol doesn't have the capability to request an Object type, so it can't retrieve Spatial. Unfortunately, the ODBC standard doesn't provide any mechanism to reasonably ask for an object or for the ODBC driver to return one.
Justin

How to detect encoding file in ANSI, UTF8 and UTF8 without BOM

Hi all,
I am having a problem with detecting a .txt/.csv file encoding. I need to detect a file in ANSI, UTF8 and UTF8 without BOM but the problem is the encoding of ANSI and UTF8 without BOM are the same. I checked the function below and saw that ANSI and UTF8
without BOM have the same encoding. so, How can I detect UTF8 without BOM encoding file? because I need to handle for this case in my code.
Thanks.
public Encoding GetFileEncoding(string srcFile)
// *** Use Default of Encoding.Default (Ansi CodePage)
Encoding enc = Encoding.Default;
// *** Detect byte order mark if any - otherwise assume default
byte[] buffer = new byte[10];
FileStream file = new FileStream(srcFile, FileMode.Open);
file.Read(buffer, 0, 10);
file.Close();
if (buffer[0] == 0xef && buffer[1] == 0xbb && buffer[2] == 0xbf)
enc = Encoding.UTF8;
else if (buffer[0] == 0xfe && buffer[1] == 0xff)
enc = Encoding.Unicode;
else if (buffer[0] == 0 && buffer[1] == 0 && buffer[2] == 0xfe && buffer[3] == 0xff)
enc = Encoding.UTF32;
else if (buffer[0] == 0x2b && buffer[1] == 0x2f && buffer[2] == 0x76)
enc = Encoding.UTF7;
else if (buffer[0] == 0xFE && buffer[1] == 0xFF)
// 1201 unicodeFFFE Unicode (Big-Endian)
enc = Encoding.GetEncoding(1201);
else if (buffer[0] == 0xFF && buffer[1] == 0xFE)
// 1200 utf-16 Unicode
enc = Encoding.GetEncoding(1200);
return enc;

what you want is to get the encoding utf-8 without bom which can only be detected if the file has special characters, so do the following:
public Encoding GetFileEncoding(string srcFile)
// *** Use Default of Encoding.Default (Ansi CodePage)
Encoding enc = Encoding.Default;
// *** Detect byte order mark if any - otherwise assume default
byte[] buffer = new byte[10];
FileStream file = new FileStream(srcFile, FileMode.Open);
file.Read(buffer, 0, 10);
file.Close();
if (buffer[0] == 0xef && buffer[1] == 0xbb && buffer[2]
== 0xbf)
enc = Encoding.UTF8;
else if (buffer[0] == 0xfe && buffer[1] == 0xff)
enc = Encoding.Unicode;
else if (buffer[0] == 0 && buffer[1] == 0 && buffer[2]
== 0xfe && buffer[3] == 0xff)
enc = Encoding.UTF32;
else if (buffer[0] == 0x2b && buffer[1] == 0x2f &&
buffer[2] == 0x76)
enc = Encoding.UTF7;
else if (buffer[0] == 0xFE && buffer[1] == 0xFF)
// 1201 unicodeFFFE Unicode (Big-Endian)
enc = Encoding.GetEncoding(1201);
else if (buffer[0] == 0xFF && buffer[1] == 0xFE)
// 1200 utf-16 Unicode
enc = Encoding.GetEncoding(1200);
else if (validatUtf8whitBOM(srcFile))
enc = UTF8Encoding(false);
return enc;
private bool validateUtf8whitBOM(string FileSource)
bool bReturn = false;
string TextUTF8 = "", TextANSI = "";
//lread the file as utf8
StreamReader srFileWhitBOM = new StreamReader(FileSource);
TextUTF8 = srFileWhitBOM .ReadToEnd();
srFileWhitBOM .Close();
//lread the file as ANSI
StreamReader srFileWhitBOM = new StreamReader(FileSource,Encoding.Defaul,false);
TextANSI = srFileWhitBOM .ReadToEnd();
srFileWhitBOM .Close();
// if the file contains special characters is UTF8 text read ansi show signs
if(TextANSI.Contains("Ã") || TextANSI.Contains("±")
bReturn = true;
return bReturn;

Oracle ODBC and DAO extremely slow

Hi
I'm using Microsoft DAO 3.6 with an Oracle ODBC connection (version 9.2) in an VB6 application.
Opening an updateable dynaset is extremely slow, I have measured the performance with Oracle ODBC, Microsoft Oracle ODBC and Microsoft SQL-Server ODBC as follows:
Oracle ODBC: 1952 ms
MS Oracle ODBC: 360 ms
MS SQL-Server ODBC: 40 ms (connection to a MS SQL-Server)
If I use the SQLpassthrough option the result is about 10 ms in all 3 cases, but the dynaset is readonly !
The testing program is as follows:
Option Explicit
Private Declare Function timeGetTime Lib "winmm.dll" () As Long
Private Sub Form_Load()
Dim wsdata As DAO.Workspace
Dim db As DAO.Database
Dim dbconn As String
Dim rs As DAO.Recordset
Dim start As Long
dbconn = "ODBC;DSN=AdhocitORA;UID=adhocit;PWD=adhocit;"
Set wsdata = DBEngine.Workspaces(0)
Set db = wsdata.OpenDatabase("", False, False, dbconn)
start = timeGetTime
Set rs = db.OpenRecordset("Select * from Afdeling", dbOpenDynaset)
rs.MoveLast
rs.MoveFirst
rs.Close
Set rs = Nothing
MsgBox "Time " & timeGetTime - start
End Sub
Is there any setting in the Oracle ODBC, that need to be adjusted ??
Erling

Why are you doing a moveLast followed by a moveFirst? That's going to force Oracle to retrieve all the records in the recordset. If there are a lot of records, you'll probably see a benefit if you increase the prefetch in the Oracle ODBC DSN configuration.
Justin

Report Script and UTF8 in 9.3.1

Hi guys, I have an issue with report scripts and UTF8 in 9.3.1.
Basically my problem is that if I run the RS from EAS, the output is a txt file in UTF8.
If I run it through Esscmd, the output is a ANSI txt.
I need to info to load a non UTF Cube.
I use English/Latin1 locale, the same settings I used in 9.2. In fact is the same RS, but now it acts weird.
I have this problem in 2 different clients using 9.3.1.
Any suggestion?
The Essbases servers are Non unicode

Thanks, I know that they should export to the local format of the client. That is the weird stuff, because, as I understand the ouput should be the same if I run it through EAS and if I use Esscmd from the same machine.
When I used MaxL it also export in UTF8. I have only tried MaxL through EAS, so I don't know if I run it through the command line if it will export in UTF8.
We use Non unicode basically because we have had some problems with Planning and some character set in previous releases and to keep it simple.

ORACLE-ODBC and ACCESS2000

Hi,
I am working with Oracle-ODBC-Driver 8.01.61.00 (from 28.04.2000) and Access2000 under Windows2000 in a german-version.
Access removes the comma by decimal-types.
In Oracle I have 5.12 but access shows me 512 !!! :-((
Who can help me ???
This is very, very, very urgent !!!!
Thanks !
Hallo,
Ich habe Oracle-ODBC-Treiber 8.01.61.00 vom 28.4.2000 dazu Access2000 und Windows2000 auf deutsch.
Leider entfernt Access alle Dezimaltrenner bei Zahlen mit Dezimalstellen.
In der Oracle steht 5,12 und Access zeigt mir 512 an !!!
Wer kann helfen ??? Es ist dringend !!!
Danke

Note that the original poster asked the same question a day earlier in the thread "ACCESS and ODBC and Numeric's" which has received a couple of replies.
Justin Cave
ODBC Development

Discoverer Viewer and UTF8

Dear friends
I have problem with Discoverer Viewer - version 9.0.2.39.02 and UTF8.My infra has a UTF8 and my App server is too, but when i run a query on Discoverer Viewer (web) i see the header of table with WE8MSWIN1252 then it's not readable!!! How can i change this nls_lang to UTF8 or how can i solve this problem.
thanks alot.

Hi,
Were you able to find the solution to this problem? I have installed 9ias successfully. I could log to OEM and created a public connection but I cannot go to
http://hostname.domain:7779/discoverer/viewer
I get an error "Internal Server Error".
Any help is welcome
Vani

ODBC and OCI not working with OBIEE 11g Client on 64 bit machine

Hi,
I have Installed OBIEE 11g client (Admin tool) on Windows XP Professional 64 bit machine.. I have Installed 11g database on the same machine but while importing Metadata using admin tool. getting "Connection has failed" error message... Can someone tell me whats the issue..
ODBC and OCI works fine with OBIEE 10g Admin..

Hi,
Oracle_BI1\network\admin folder gets installed with full obiee 11g Installlation and I have installed only 11g client as I have OBIEE 10g already Installed on my machine..
Installation is as follows:
ORACLE 11g CLIENT Installation is as follows:
C:\Program Files\Oracle Business Intelligence Enterprise Edition Plus Client
ORACLE 10g Installation is as follows:
C Drive:
ORACLE BI
ORACLE BI Data ( everything is working with this)
latest Log Info is as follows:
[2012-02-02T11:05:00.000+00:00] [OracleBIServerComponent] [ERROR:1] [] [] [ecid: ] [tid: 24fc] [nQSError: 16001] ODBC error state: 08004 code: 12154 message: [Oracle][ODBC][Ora]ORA-12154: TNS:could not resolve the connect identifier specified.
[2012-02-02T11:05:16.000+00:00] [OracleBIServerComponent] [ERROR:1] [] [] [ecid: ] [tid: 2f3c] [nQSError: 16001] ODBC error state: 08004 code: 12154 message: [Oracle][ODBC][Ora]ORA-12154: TNS:could not resolve the connect identifier specified.
[2012-02-02T11:32:53.000+00:00] [OracleBIServerComponent] [ERROR:1] [] [] [ecid: ] [tid: 2f3c] [nQSError: 16001] ODBC error state: 08004 code: 12154 message: [Oracle][ODBC][Ora]ORA-12154: TNS:could not resolve the connect identifier specified.

ODBC and CLI info !

Hi All
This is for my knowledge sake : can someone tell me (or clear my doubts) , With Example , about ODBC and CLI ???
What I know about ODBC is we configure data source in Windows ODBC administrator, and depending upon whether Application is 32/64 bit, 32/64 bit ODBC is invoked, am I correct ?
Where do we use ODBC or CLI ?
Why should we prefer CLI over ODBC ?
Thanks a lot for the knowledge imparted, in advance.

The release notes that accompany each download will have a pretty detailed listing of the feature changes and the bug fixes. Are you looking for more than that?
Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC

Reason to explain these performances of odbc and occi...

I wrote some simple test programs to compare the performances of odbc and occi accessing Oracle.
These programs were wrote in C++. The occi and odbc were given under almost the same circumstance.
The result is: when executing query(simple select) the odbc is fast than occi. When executing insert sentences, the occi is much more fast than odbc does.
My computer environment: Oracle 10g, Oracle instant cilent, VS 8.0.
Do they all access oracle through OCI interface? How to explain the result?
Someone give me some hints , please.

Hi,
With the limited info available, I can't offer any specific suggestions that will help here.
Usually sqlnet tracing is a good thing to check to see if maybe it's round trips causing the discrepency, packet/buffer sizes, query differnces, etc.
Server side sql tracing is often useful as well to check bind types, query execution, etc.
Yes, both Oracle's ODBC driver and OCCI are written in OCI, but they have completely different implmentations: array fetching, statement caching, buffer sizes, etc. Simply because they're both written in OCI doesnt mean they share an implmentation.
Hope it helps,
Greg

ArchLinux and UTF8

It is not the best place for this post but since we have to start from the kernel....
I tried to send and e-mail about this to the mailing list and I receved no answer so lets try by the forum.
Basically I started to think about that since samba 3.0 (that use UTF8 as
default) came out.
By looking at Archlinux:
Kernel NLS option is compiled with iso8859-1 (western european)
Kernel Remote filesystem NLS use cp437 (united state)
There is no iso10496/utf8 fonts in the system.
An interesting step for arch should be moving to utf8.
Opinion??

Basically is this.
Normally you can't visualize all the possible chars in the world.
In fact you chose the "type of char" you want to see/support on your system.
For instance if you use iso8859-1 charset you can visualize accetend letters like òàùèìéç etc. basically the ones you find in western european language (like Italian French German Spanish Portugues....) but if you try to visualize something in Russian Bulgarian Rumenian etc. there are different kind of letters/accent and instead of the correct char you probably see question mark under windows or strange char with linux. Imagine if you try to read a japanese document.
This limitation of "chars quantity" was due to the necessity to use only 8 bit for the coding in order to save space in memory and hd.
Now modern computers have huge ammount of memory and so UTF8 came out.
Basically this standard provide the coding for any chars in the planet. By setting this on your computer and programs you are able to read any kin of document with any coding. Offcourse not everybody need it but anyway all the system (operating systems and programs) are moving to this standard.

Internationalisation ServletFilter and UTF8 Character-Encoding

Hello to all
I use a non common but nice way to internationalize my web-application.
I use a ServletFilter to replace the text-keys. So static-resources stays static-resources that can be cached and don't need to be translated each time they are requested.
But there is a little problem to get it working with utf-8.
In my opinion there is only one way to read the response-content. I have to use a own HttpServletResponseWrapper like recommended under [http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Servlets8.html#82361].
If I do so it is no more possible to use ServletQutputStream (wrapper.getOutputStream()) to write back the modified/internationalized content (e.g. with german umlauts/Umlaute) back to the response in the right encoding.
Writer out = new BufferedWriter(new OutputStreamWriter(wrapper.getOutputStream(), "UTF8"));Using PrintWriter for writing does not work. Because umlauts are send in the wrong encoding (iso-8859-1). With the network-sniffer Wireshark I've seen that � comes as FC that means as iso-8859-1 encoded character.
It obviously uses the plattforms default-encoding although the documentation does not mentions this explictly for the Constructor ([PrintWriter(java.io.Writer,boolean)|http://java.sun.com/j2se/1.4.2/docs/api/java/io/PrintWriter.html#PrintWriter(java.io.Writer,%20boolean)]).
So my questions:
1. Is there a way to get response-content without loosing option to call wrapper.getOutputStream().
or
2. can I set the encoding for my printwriter
or
3. can I encode the content before writing it to the printwriter and will this solve the problem
new String(Charset.forName("UTF8").encode(content).array(), "UTF8") did not work.
Here comes my code:
The Filter to tanslate the resources/response:
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import de.modima.util.lang.Language;
public class TranslationFilter implements Filter
     private static final Log log = LogFactory.getLog(TranslationFilter.class);
     public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException
          String lang = Language.setLanguage((HttpServletRequest) request);
          CharResponseWrapper wrapper = new CharResponseWrapper((HttpServletResponse)response, "UTF8");
          PrintWriter out = response.getWriter();
          chain.doFilter(request, wrapper);
          String content = wrapper.toString();
          content = Language.translateContent(content, lang);
          content += "                                                                                  ";
          wrapper.setContentLength(content.length());
          out.write(content);
          out.flush();
          out.close();
     public void destroy(){}
     public void init(FilterConfig filterconfig) throws ServletException{}
     }The response-wrapper to get acces to the response content:
import java.io.CharArrayWriter;
import java.io.PrintWriter;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class CharResponseWrapper extends TypedSevletResponse
     private static final Log log = LogFactory.getLog(CharResponseWrapper.class);
     private CharArrayWriter output;
     public String toString()
          return output.toString();
     public CharResponseWrapper(HttpServletResponse response, String charsetName)
          super(response, charsetName);
          output = new CharArrayWriter();
     public PrintWriter getWriter()
          return new PrintWriter(output, true);
     }The TypedResponse that takes care for setting the right http-header informations according to the given charset:
import java.nio.charset.Charset;
import java.util.StringTokenizer;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletResponseWrapper;
public class TypedSevletResponse extends HttpServletResponseWrapper
     private String type;
     private String charsetName;
      * @param response
      * @param charsetName the java or non-java name of the charset like utf-8
     public TypedSevletResponse(HttpServletResponse response, String charsetName)
          super(response);
          this.charsetName = charsetName;
     public void setContentType(String type)
          if (this.type==null && type!=null)
               StringTokenizer st=new StringTokenizer(type,";");
               type=st.hasMoreTokens()?st.nextToken():"text/html";
               type+="; charset="+getCharset().name();
               this.type=type;
          getResponse().setContentType(this.type);
   public String getContentType()
      return type;
   public String getCharacterEncoding()
          try
               return getCharset().name();
          catch (Exception e)
               return super.getCharacterEncoding();
     protected Charset getCharset()
          return Charset.forName(charsetName);
     }Some informations about the enviroment:
OS: Linux Debian 2.6.18-5-amd64
Java: IBMJava2-amd64-142
Apserver: JBoss 3.2.3
Regards
Markus Liebschner
Edited by: MaLie on 30.04.2008 11:52
Edited by: MaLie on 30.04.2008 11:54 ... typo
Edited by: MaLie on 30.04.2008 12:04

Hello cndvg
yes I did.
I found the solution in this forum at [Filter inconsistency Windows-Solaris?|http://forum.java.sun.com/thread.jspa?threadID=520067&messageID=2518948]
You have to use a own implementation of ServletOutputStream.
public class TypedServletOutputStream extends ServletOutputStream
     CharArrayWriter buffer;
     public TypedServletOutputStream(CharArrayWriter aCharArrayWriter)
          super();
          buffer = aCharArrayWriter;
     public void write(int aInt)
          buffer.write(aInt);
     }Now the CharResponseWrapper looks like this.
public class CharResponseWrapper extends TypedSevletResponse
     private static final Log log = LogFactory.getLog(CharResponseWrapper.class);
     private CharArrayWriter output;
     public String toString()
          return output.toString();
     public CharResponseWrapper(HttpServletResponse response, String charsetName)
          super(response, charsetName);
          output = new CharArrayWriter();
     public PrintWriter getWriter() throws IOException
          return new PrintWriter(output,true);
     public ServletOutputStream getOutputStream()
          return new TypedServletOutputStream(output);
     }Regards
MaLie

ODBC and UTF8 charset

Similar Messages

Maybe you are looking for