Problems with non-ASCII characters on Linux Unit Test Import

I found a problem with non-ASCII characters in the Unit Test Import for Linux. This problem does not appear in the Unit Test Import for Windows.
I have attached a Unit Test export called PROC1.XML It tests a procedure that is included in another attachment called PROC1.txt. The unit test includes 2 implementations. Both implementations pass non-ASCII characters to the procedure and return them unchanged.
In Linux, the unit test import will change the non-ASCII characters in the XML file to xFFFD. If I copy/paste the the non-ASCII characters into the Unit Test after the import, they will be stored and executed correctly.
Amazon Ubuntu 3.13.0-45-generic / lubuntu-core
Oracle 11g Express Edition - AL32UTF8
SQL*Developer 4.0.3.16 Build MAIN-16.84
Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
In Windows, the unit test will import the non-ASCII characters unchanged from the XML file.
Windows 7 Home Premium, Service Pack 1
Oracle 11g Express Edition - AL32UTF8
SQL*Developer 4.0.3.16 Build MAIN-16.84
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
If SQL*Developer is coded the same between Windows and Linux, The JVM must be causing the problem.

Set the System property "mail.mime.decodeparameters" to "true" to enable the RFC 2231 support.
See the javadocs for the javax.mail.internet package for the list of properties.
Yes, the FAQ entry should contain those details as well.

Similar Messages

Problem with non-ASCII characters on TTY

Although I'm not a native speaker, I want my system language to be English (US), since that's what I'm used to. However I have a lot of files which have German language in their file names.
My /etc/locale.conf has en_US.UTF-8 and de_DE.UTF-8 enabled. My /etc/locale.conf contains only the line
LANG=en_US.UTF-8
The German file names show up fine within Dolphin and Konsole (ls -a). But they look weird on either of the TTYs (the "console" you get to by pressing e.g. ctrl+alt+F1). They have other characters like '>>' or the paragraph symbol where non-ASCII characters should be. Is it possible to fix this?

I don't think the console font is the problem. I use Lat2-Terminus16 because I read the Beginner's Guide on the wiki while installing the system.
My /etc/vconsole.conf:
KEYMAP=de
FONT=Lat2-Terminus16
showconsolefont even shows me the characters missing in the file names; e.g.: Ö, Ä, Ü

Filling clob with non ascii characters

Hello,
I have had some problems with clobs and usage of german
umlauts (��). I was'nt able to insert or update
strings containing umlaute in combination with string
binding. After inserting or updating the umlaut
characters were replaced by strange (spanish) '?'
which were upside down.
However, it was working when I did not use string bindung.
I tried varios things, after some time I tracked
the problem down to to oracle.toplink.queryframework.SQLCall.java. In the
prepareStatement(...) you find something
like
ByteArrayInputStream inputStream = new ByteArrayInputStream(((String) parameter).getBytes());
// Binding starts with a 1 not 0.
statement.setAsciiStream(index + 1, inputStream,((String) parameter).getBytes().length);
I replaced the usage of ByteArrayInputStram with CharArrayReader:
// TH changed, 26.11.2003, Umlaut will not work with this.
CharArrayReader reader = new CharArrayReader(((String) parameter).toCharArray());
statement.setCharacterStream(index + 1, reader, ((String) parameter).length() );
and this worked.
Is there any other way achieving this? Did anyone
get clobs with non ascii characters to work?
Regards -- Tobias
(Toplink 9.0.3, Clob was mapped to String, Driver was Oracle OCI)

I don't think the console font is the problem. I use Lat2-Terminus16 because I read the Beginner's Guide on the wiki while installing the system.
My /etc/vconsole.conf:
KEYMAP=de
FONT=Lat2-Terminus16
showconsolefont even shows me the characters missing in the file names; e.g.: Ö, Ä, Ü

Problems with non-ascii keywords

I have some problems with non-ascii keywords that makes the whole keyword feature useless for me. I don't know if I'm doing something wrong or if there something I'm missing completely.
The problem is that when I enter something like "grön" iPhoto sometimes refuses me to type in "grön" on another photo and "eats" the "ö". And if I select the matching keyword from the popup list iPhoto has changed the originally "grön" to "gr¨ön" (if that comes out right on the web). Here are a few screenshots to show what happens
If someone knows what happens I would really appriciate some hints on how to avoid this.
Note that adding/editing keywords works just fine in Aperture.

There is a long standing issue with iPhoto and non-ascii characters. I know of no solution.
iPhoto menu -> Provide iPhoto Feedback and report it as a bug.
Regards
TD

Problem with non-ASCII file name in content disposition header

Hi All,
I am facing some problems with the non-ASCII file name incase of content-disposition header. I read from the RFC 2183 that if the file name contains non-ASCII characters then the same should be encoded before sending to browser. I did the same but realized 2 problems:
1. The name of the file is truncated in case the file name is slightly long for e.g. ��j�b�g��.txt
2. Also when the same file is opened in notepad, the title is showing encoded name %E6%9C%80%E4%B8%8A%E4%BD%8D.....
Overall, I feel that the browser is not understanding or responding to the encoded header values.
Is there any solution to this problem? I am using Microsoft IE 6.0.
The code snippet is given below:
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
          String fileName = "��j�b�g��.txt";
          fileName = URLEncoder.encode(fileName, "UTF-8");
          resp.setCharacterEncoding("UTF-8");
          resp.setHeader("Content-Disposition", "attachment; filename=\"" + fileName + "\"");
          resp.setContentType("application/download-binary");
          String s = "This is inside txt file";
          resp.getOutputStream().write(s.getBytes("UTF-8"));
          return;
     }Any help or pointer would be highly appreciated.
Thanks and Regards,
Ashish

The MIME standards for non-ASCII filenames are not widely implemented.
Many mailers use an ad hoc method for encoding filenames. JavaMail
supports both methods, but you need to set properties, such as the
mail.mime.encodefilename property. See the JavaMail javadocs for
the javax.mail.internet package.

Cannot rename file with non-ASCII characters when using the

My application moves files from one directory to another by calling File[] srcFiles = srcDir.listFiles() to get a list of files in the source directory, and then calling srcFiles.renameTo(destFile) to rename each file.
This does not work (renameTo returns false and the file is not moved) under the following circumstances:
- the file's leaf name contains non-ASCII characters, for example "�"
- the OS is Solaris 9
- the LANG and LC_* environment variables are unset, i.e. the C locale is being used
If I set the LANG environment variable to, for example, en_GB.UTF-8 then the rename succeeds.
I have tried calling srcFiles[index].getName().getBytes("UTF-8") and the non-ASCII characters are being replaced with ? (0x3f) characters when LANG is unset.
Is this a bug in the JRE? I would argue that since my code does not actually manipulate the filename (I just use the File object that File.listFiles() gives me) then the rename should succeed. Of course I would not expect the file name to be displayed correctly if I printed it out.
I have reproduced this behaviour with JDK 1.4.2_05 and 1.5.0_04 on Solaris 9.
Francis

Thanks for the info Alan.
I considered setting the locale in the environment (this sounds like the "correct" fix to me and we might implement it later), but this application shares a WebLogic server with many other applications so we would have to do a huge amount of testing to make sure that the locale change wouldn't break the other apps. In the end I worked around the problem by making the code that generates the filenames in the first place strip out any non-ASCII characters (the names of the files are not critically important).
Looking forward to JSR-203, in the meantime perhaps a note about this behaviour in the java.io.File javadoc would be useful.

Can't get the attachment filename out of a Part (with non ascii characters)

Hello, all and happy new year :)
My issue is with non ascii filename in attachments... Yes i've read the FAQ : http://www.oracle.com/technetwork/java/faq-135477.html#encodefilename
I can't get the filename out of the BodyPart for those kind of attachments
here's my unit test :
     * contains various parts from various mailer encoded in different ways...
     private enum EncodedFileNamePart{
          OUTLOOK("Content-Type: text/plain;\n name=\"=?iso-8859-1?Q?c'estd=E9j=E0no=EBl=E7ac'estcool.txt?=\" \nContent-Transfer-Encoding: 7bit\nContent-Disposition: attachment;\n filename=\"=?iso-8859-1?Q?c'estd=E9j=E0no=EBl=E7ac'estcool.txt?=\" \n\nnoel 2010\n","c'estdéjànoëlçac'estcool.txt"),
          GMAIL("Content-Type: text/plain; charset=US-ASCII; name=\"=?ISO-8859-1?B?ZOlq4G5v62znYWNlc3Rjb29sLnR4dA==?=\"\nContent-Disposition: attachment; filename=\"=?ISO-8859-1?B?ZOlq4G5v62znYWNlc3Rjb29sLnR4dA==?=\"\nContent-Transfer-Encoding: base64\nX-Attachment-Id: f_giityr5r0\n\namluZ2xlIGJlbGxzIQo=\n","déjànoëlçacestcool.txt"),
          THUNDERBIRD("Content-Type: text/plain;\n name=\"=?ISO-8859-1?Q?d=E9j=E0no=EBl=E7acestcool=2Etxt?=\"\nContent-Transfer-Encoding: 7bit\nContent-Disposition: attachment;\n filename*0*=ISO-8859-1''%64%E9%6A%E0%6E%6F%EB%6C%E7%61%63%65%73%74%63%6F;\n filename*1*=%6F%6C%2E%74%78%74\n\njingle bells!\n","déjànoëlçacestcool.txt"),
          EVOLUTION("Content-Disposition: attachment; filename*=ISO-8859-1''d%E9j%E0no%EBl.txt\nContent-Type: text/plain; name*=ISO-8859-1''d%E9j%E0no%EBl.txt; charset=\"UTF-8\" \nContent-Transfer-Encoding: 7bit\n\njingle bells\n","déjànoël.txt"),
          String content=null;
          String target=null;
          private EncodedFileNamePart(String content,String target){
               this.content=content;
               this.target=target;
          public Part get(){
               try{
               ByteArrayInputStream bis = new ByteArrayInputStream(this.content.getBytes());
               Part part = new MimeBodyPart(bis);
               bis.close();
               return part;
               catch(Throwable e){
                    return null;
          public String getTarget(){
               return this.target;
     @Test
     public void testJavamailDecode() throws MessagingException, UnsupportedEncodingException{
          System.setProperty("mail.mime.encodefilename", "true");
          System.setProperty("mail.mime.decodefilename", "true");
          for(EncodedFileNamePart part : EncodedFileNamePart.values())
               assertEquals(part.name(),MimeUtility.decodeText(part.get().getFileName()),part.getTarget());
I take a NullPointerExcepion in the decodeText because getFileName() return null for the EVOLUTION case, and work well with OUTLOOK, THUNDERBIRD and GMAIL.
Evolution's content type is "Content-Disposition: attachment; filename*=ISO-8859-1''d%E9j%E0no%EBl.txt" wich doesn't look like the other (looks like the RFC 2616 or RFC5987 to do it.)
How can i handle this situation except by writting my own decoder?
Thanks for your answers!
Edited by: user13619058 on 4 janv. 2011 07:44

Set the System property "mail.mime.decodeparameters" to "true" to enable the RFC 2231 support.
See the javadocs for the javax.mail.internet package for the list of properties.
Yes, the FAQ entry should contain those details as well.

[AS] Problem with non English characters in file path

I wrote a script that exports a pdf file from ID, rasterizes it in PS, applies an action, saves it as another pdf file, and finally creates a Mail message, and attaches the file to it (the last part is written in AppleScript).
The problem is that it doesn't work when the path to this file contains non English characters.
This works:
make new attachment with properties {file name:"/Volumes/Macintosh HD/BackUp Tetard/Test.pdf"}
but this doesn't:
make new attachment with properties {file name:"/Volumes/Macintosh HD/BackUp Têtard /Test.pdf"}
I remember vaguely that I read somewhere that AppleScript can work with Unicode — in other words with such characters — starting from some version, don't remember which exactly, but it seems to me — Leopard.
I am on Mac OS X 10.4.11 right now. Will updating solve this problem? Does anybody know any solution to this problem: a scripting addition, some hidden setting, etc.
I made a little test: used a Russian character — ё and it works, but when I use — ê (Dutch) it doesn't. May it have something to do with the Region setting in International panel?
Thanks in advance,
Kasyan

Kasyan, as of Leopard AppleScript treats all text as Unicode pre this you can specify 'as Unicode text'. Try a test with these.
-- Leopard
set x to POSIX path of (path to desktop)
-- Pre Leopard
set x to POSIX path of (path to desktop as Unicode text)
-- Leopard
set x to POSIX path of (choose file without invisibles)
-- Pre Leopard
set x to POSIX path of ((choose file without invisibles) as Unicode text)

Unable to play videos with non-ASCII-characters in filename

Hi!
I use a MediaPlayer to display MP4-videos in my application. This works quite well. Unfortunately I have a problem if the filename of the video to be shown contains non-ASCII-charcaters.
I get the following message:
-->file:D:\daten\avi\�� .MPG
Error: Unable to realize com.sun.media.amovie.AMController@4b7651
Failed to realizeThe first line shows the filename I pass to the setMediaLocation()-method of the MediaPlayer-object.
What's wrong? If I rename the file to ABC.mpg it works fine.
Thanks for your help
Thomas

Hi!
I use a MediaPlayer to display MP4-videos in my application. This works quite well. Unfortunately I have a problem if the filename of the video to be shown contains non-ASCII-charcaters.
I get the following message:
-->file:D:\daten\avi\�� .MPG
Error: Unable to realize com.sun.media.amovie.AMController@4b7651
Failed to realizeThe first line shows the filename I pass to the setMediaLocation()-method of the MediaPlayer-object.
What's wrong? If I rename the file to ABC.mpg it works fine.
Thanks for your help
Thomas

Publishing pages with non-ASCII characters to folder

There is a weirdness/anachronism in exported file names from iWeb'08.
I have a page "Mökin katto" (that is Finnish).
"ls" in Terminal shows the exported folder ok, like
Mökin_katto.html
Mökinkattofiles/
Tarring and uploading to show on my Apache web server did not work so good, though. I changed the server to operate in UTF-8 locale and also use UTF-8 as the default character set. Still no success. At this point, I wanted to check the exported folder in the source (my MacBook). Tab completion didn't work:
% Mö<TAB>
didn't give any results. At this point, I wrote a simple Python script to dump the filenames. I created pages with different Scandinavian characters and capitalizations to demonstrate (I have cleaned the output a bit to remove unnecessary filenames):
% ls
MäkiÄn_katto.html MökiÖn_katto.html feed.xml
MäkiÄnkattofiles/ MökiÖnkattofiles/ index.html
MåkiÅn_katto.html Mökin_katto.html
MåkiÅnkattofiles/ Mökinkattofiles/
% ~/repos/scripts/misc/dirdump.py
MäkiÄn_katto.html 'M(4d)' 'a(61)' cc88 'k(6b)' 'i(69)' 'A(41)'cc 88 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '.(2e)' 'h(68)' 't(74)' 'm(6d)' 'l(6c)'
MäkiÄnkattofiles 'M(4d)' 'a(61)' cc88 'k(6b)' 'i(69)' 'A(41)'cc 88 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '_(5f)' 'f(66)' 'i(69)' 'l(6c)' 'e(65)' 's(73)'
MåkiÅn_katto.html 'M(4d)' 'a(61)' cc8a 'k(6b)' 'i(69)' 'A(41)'cc 8a 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '.(2e)' 'h(68)' 't(74)' 'm(6d)' 'l(6c)'
MåkiÅnkattofiles 'M(4d)' 'a(61)' cc8a 'k(6b)' 'i(69)' 'A(41)'cc 8a 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '_(5f)' 'f(66)' 'i(69)' 'l(6c)' 'e(65)' 's(73)'
Media 'M(4d)' 'e(65)' 'd(64)' 'i(69)' 'a(61)'
Mökin_katto.html 'M(4d)' 'o(6f)' cc88 'k(6b)' 'i(69)' 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '.(2e)' 'h(68)' 't(74)' 'm(6d)' 'l(6c)'
Mökinkattofiles 'M(4d)' 'o(6f)' cc88 'k(6b)' 'i(69)' 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '_(5f)' 'f(66)' 'i(69)' 'l(6c)' 'e(65)' 's(73)'
MökiÖn_katto.html 'M(4d)' 'o(6f)' cc88 'k(6b)' 'i(69)' 'O(4f)'cc 88 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '.(2e)' 'h(68)' 't(74)' 'm(6d)' 'l(6c)'
MökiÖnkattofiles 'M(4d)' 'o(6f)' cc88 'k(6b)' 'i(69)' 'O(4f)'cc 88 'n(6e)' '_(5f)' 'k(6b)' 'a(61)' 't(74)' 't(74)' 'o(6f)' '_(5f)' 'f(66)' 'i(69)' 'l(6c)' 'e(65)' 's(73)'
Scripts 'S(53)' 'c(63)' 'r(72)' 'i(69)' 'p(70)' 't(74)' 's(73)'
Apparently Ö and ö are translated to the sequences '"O" 0xcc 0x88' and '"o" 0xcc 0x88', Ä and ä to '"A" 0xcc 0x88' and '"a" 0xcc 0x88' and finally Å and å to the sequences '"A" 0xcc 0x8a' and '"a" 0xcc 0x8a'. Looking into this a bit more, those ("0xcc 0x88" and "0xcc 0x8a") are the UTF-8 encodings of COMBINING DIAERESIS (U+308) (http://www.fileformat.info/info/unicode/char/0308/index.htm) and COMBINING RING ABOVE (U+030A) (http://www.fileformat.info/info/unicode/char/030a/index.htm).
The generated links in the pages are in the short (Latin1 equivalent) form, but the filenames are in this format. The meaning of the strings is the same, but e.g. Apache doesn't internally canonicalize the paths, which results in broken URLs. I think iWeb should export the filenames and the URLs in identical UTF-8 strings (which quite likely should be the Latin1-equivalent Unicode code points). Especially as e.g. Terminal and bash only work with the short forms.
Is there an option to make iWeb behave this way?

I tried checking whether changing my keyboard layout (I've created the current one with Ukelele) to one which wrote decomposed characters changed things; it didn't. The filenames and generated html-files contained identical entries; I could use tab-completion for the generated filenames, though. (Bash was otherwise a bit confused about the change, so I can't recommend this)
By canonicalizing the filenames with a script on the server (a Debian GNU/Linux system) end I can now make it work; the root cause of the discrefence between the file name and the URLs in the generated HTML is still a mystery. This would've worked if iWeb wrote decomposed UTF-8 characters to the URLs, also.

Problems with non-English characters in iTunes 7.7

Since "upgrading" to iTunes 7.7 every file containing characters not normally found in English (e.g.: å, ß, ç, é, î, ñ, ø, ü, etc.) gets a mane change when iTunes plays the track. For example, a song called "Baião" suddenly becomes "Bai.o" and if the artist or album name contains accented characters, these get scrambled as well. The files themselves also get jettisoned from their folders in the iTunes Library, which causes big problems.

Kasyan, as of Leopard AppleScript treats all text as Unicode pre this you can specify 'as Unicode text'. Try a test with these.
-- Leopard
set x to POSIX path of (path to desktop)
-- Pre Leopard
set x to POSIX path of (path to desktop as Unicode text)
-- Leopard
set x to POSIX path of (choose file without invisibles)
-- Pre Leopard
set x to POSIX path of ((choose file without invisibles) as Unicode text)

Difficulties retrieving documents with non ascii characters in name

Hello
I have a very simple container with only one document in it: *<ação nome="ação"/>*
When using the dbxml shell to retrieve it, no problem:
dbxml> open teste.dbxml
dbxml> get "ação"
1 documents found
dbxml> print
<?xml version="1.0" encoding="UTF-8"?>
<a├º├úo nome="a├º├úo"/>
(The strange characters appear because my terminal uses Latin1 encoding.)
But I cannot retrieve the document with a very simple Java program (left out basic environment setup for clarity - I will put the full code in a follow-up post):
String name = new String("ação"); +// will cause: com.sleepycat.dbxml.XmlException: Error: Document not found: ação, errcode = DOCUMENT_NOT_FOUND+
// String name = new String("ação".getBytes(), utf8); +// will cause: com.sleepycat.dbxml.XmlException: Error: Document not found: ação, errcode = DOCUMENT_NOT_FOUND+
String name = new String("ação".getBytes(), latin1); +// will cause: com.sleepycat.dbxml.XmlException: Error: Document not found: aÃ§Ã£o, errcode = DOCUMENT_NOT_FOUND+
XmlDocument doc = container.getDocument(tx, name);
Almost certainly this problem has to do something with the encoding of the name, but I cannot see why.
Thank you very much for looking into this,
Greetings,
Koen

As promised, the full Java code of this basic program:
package info.carrapateira;
import com.sleepycat.dbxml.*;
import com.sleepycat.db.*;
import java.io.*;
import java.nio.charset.*;
public class DbxmlTest2 {
     static Charset utf8 = Charset.forName("UTF-8");
     static Charset latin1 = Charset.forName("ISO-8859-1");
     public static void main(String[] args) {
          EnvironmentConfig envCfg = new EnvironmentConfig();
          envCfg.setAllowCreate(true);
          envCfg.setInitializeCache(true);
          envCfg.setInitializeLocking(true);
          envCfg.setInitializeLogging(true);
          envCfg.setRunRecovery(true);
          envCfg.setTransactional(true);
          Environment env = null;
          XmlManager manager = null;
          XmlContainer container = null;
          XmlTransaction tx = null;
          try {
               env = new Environment(new File(args[0]), envCfg);
               XmlManagerConfig cfg = new XmlManagerConfig();
               cfg.setAdoptEnvironment(true);
               manager = new XmlManager(env, cfg);
               XmlContainerConfig ctCfg = new XmlContainerConfig();
               ctCfg.setTransactional(true);
               container = manager.openContainer("teste.dbxml", ctCfg);
               tx = manager.createTransaction();
               String name = new String("ação"); // com.sleepycat.dbxml.XmlException: Error: Document not found: ação, errcode = DOCUMENT_NOT_FOUND
               // String name = new String("ação".getBytes(), utf8); // com.sleepycat.dbxml.XmlException: Error: Document not found: ação, errcode = DOCUMENT_NOT_FOUND
               //String name = new String("ação".getBytes(), latin1); // com.sleepycat.dbxml.XmlException: Error: Document not found: aÃ§Ã£o, errcode = DOCUMENT_NOT_FOUND
               XmlDocument doc = container.getDocument(tx, name);
               tx.commit();
          } catch (Exception e) {
               e.printStackTrace();
               try {
                    tx.abort();
               } catch (Exception e1) {
                    e1.printStackTrace();
          } finally {
               try {
                    container.close();
                    manager.close();
               } catch (Exception e) {
                    e.printStackTrace();
}

[SOLVED]Gnumeric and problem with non-US characters

I am trying here to get Gnumeric to recognize the UTF-8 or ISO-8859-15 characters, but there even doesn't seem to be any locale option in menus? Googling didn't show up anything spectacular.
I tried this:
LC_ALL=fr_FR gnumeric &
but got an error from command line:
(gnumeric:7559): Gtk-WARNING **: Locale not supported by C library.
Using the fallback 'C' locale.
And everything is still in the US format, even how dates are represented.

Thanks, rdoggsv - I got the menus in my language in gnumeric and as a bonus - I got the euro sign to work with your advice
But still I don't have characters like ä,ö,õ,ü at gnumeric. I get them nicely at kwrite, konsole, konqueror, openoffice and opera.
This missing chars - thing seems to be Gnome-specific. For example I can't get those chars visible at Gimp, graveman and galculator. But in Abiword the chars are visible in the editor's text editing field but not when for example I try to save a file.

Cannot login with password containing non-ascii characters

Hello,
I have web application, form based login. UTF-8 is specified "everywhere".
And it works, except for passwords.
If user register itself with password containing non-ascii characters, it is correctly written in database, but when doing either programmatic login or normal form based login, if fails.
If the password is only ascii, it works.
Username of login could be ascii or non-ascii, it doesn't matter, both works.
I'm using sun java application server 9.1.
jdbc realm.
I'm not using hashing passwords, just clean (now)
I tried configure realm Charset: UTF8 as last chance, but it doesn't work either.
The problem is only with non-ascii characters in password.
Any help very appreciated
Thanks a lot

hi,
I know all that, but that's not the case. My app uses preparedStatements, everything is properly configured, in all pages, utf-8 is going from user to db and back without any problems.
The only problem is with password field. As I am using form based login, with jdbc realm configured (again, nicely working when only ascii characters), I have very little chance to do something bad through the login phase.
I'm not talking about special characters, I'm talking about non-ascii characters, let's say - Chinese, arabish, Russian alphabet etc.
When user registers (my code), the fields are properly written to db. I have checked that, trust me.
But the Sun app server realm seems to have some problems with the password field.
(realm uses jdbc connection to mysql, the url contains all extra parameters to be sure about utf8. there is nothing more what can be configured...)
If I try other alphabet codes in login and ascii in password, it works. But soon, as I use other alphabet code also in password, it doesn't work anymore.
My only idea is, that I could try MD5 to create ascii only characters (I hope it works that way) on the client with javascript and then set Digest to MD5 in realm configuration. But still, it seems very strange. The clear way storage should also function? (now set Digest to 'none')
Is it a bug of Sun App Server?
thanks

Replacing non-ASCII characters with HTML charcter references

Hi All,
In Oracle 10g or greater is there a built-in function that will convert a string with non-ASCII characters like this
a b č 뮼
into an ASCII string with HTML character references like this?
a b & # x 0 1 0 D ; & # x B B B C ;
(note I had to include spaces between each character in the sample code for message to prevent the forum software from converting my text)
I tried using
utl_i18n.escape_reference( val, 'us7ascii' )
but for some reason it returns
a b c & # x B B B C ;
Note how it converted the Western European character "č" to its unaccented counterpart "c", not "& # x 0 1 0 D ;" (is this a bug?).
I also tried a custom solution using regexp_replace and asciistr (which I can't include here because the forum software chokes on it) but it only returns the correct result for values <=4000 characters long. Unfortunately asciistr doesn't appear to accept CLOB values larger than 4000 characters. It returns an error message like
(ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion (actual: 30251, maximum: 4000) ).
I'm looking for a solution that works on CLOB data of any size.
Thanks in advance for any insight you can provide.
Joe Fuda

So with that (UTF8) in mind, let's take another look.....
As shown below, I used a AL32UTF8 database.
Note: I did not use a unicode capable tool for querying. So I set console mode code page to 1250 just to have č displayed properly (instead of posing as an è).
Also, as a result of using windows-1250 for client character set, in the val column and in the second select's ncr column (iso8859-1), è (00e8) has been replaced with e through character set conversion going from server back to client.
Running the same code on a database with a db character set such as we8mswin1252, that doesn't define the č (latin small c with caron) character, would yield results with a c in the ncr column.
C:\>chcp 1250
Aktuell teckentabell: 1250
C:\>set nls_lang=.ee8mswin1250
C:\>sqlplus test/test
SQL*Plus: Release 11.1.0.6.0 - Production on Fri May 23 21:25:29 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the OLAP option
SQL> select * from nls_database_parameters where parameter like '%CHARACTERSET';
PARAMETER              VALUE
NLS_CHARACTERSET       AL32UTF8
NLS_NCHAR_CHARACTERSET AL16UTF16
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'us7ascii') NCR from dual;
VAL NCR
č e c e
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'we8iso8859p1') NCR from dual;
VAL NCR
č e &# x10d; e     <- "è"
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'ee8iso8859p2') NCR from dual;
VAL NCR
č e č &# xe8;
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'cl8iso8859p5') NCR from dual;
VAL NCR
č e &# x10d; &# xe8;In the US7ASCII case, where it should be possible for all non-ascii characters to be escaped, it seems as if the actual escape step is skipped over.
Hope this helps to understand whether utl_i8n is usable or not in your case.
Message was edited by:
orafad
Fixed replaced character references :)

Problems with non-ASCII characters on Linux Unit Test Import

Similar Messages

Maybe you are looking for