Non ascii characters are padding without reason

Hi all,
here is my issue :
I have description field in my jsp page, when user enters some non ASCII characters as shown below :
This is user had entered the input string is :
"Hello Ã?Â?Ã?Â?Ã?Â?Ã?Â Hello" (without quotes)
When he edits for the first time without change anything in the the string and save, then it became :
"Hello Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã? Hello" (without quotes)
Then again one more time, it became :
"Hello Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â?Ã?Â? Hello" (without quotes)
Like wise it is getting increased its size. I'm saving the data to Oracle 10g database from jsp.
Please suggest what to do to rectify the problem.
Thanks & regards,
Achchayya

But I didnt changed anything from server.xml except two changes
1) from : <?xml version='1.0' encoding='iso-8851-1'?>
    to : <?xml version='1.0' encoding='UTF-8'?>
2)
from : <Connector port="8888" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
to : <Connector port="8888" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
               URIEncoding="UTF-8"/>here is my JSP :
<%@ page pageEncoding="UTF-8" %>
<html>
<head>
<title>The servlet example </title>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body>
<h1>A simple web application</h1>
<form method="POST" action="HelloWorld">
<label for="name">Enter your name </label>
<input type="text" id="name" name="name"/><br><br>
<input type="submit" value="Submit Form"/>
<input type="reset" value="Reset Form"/>
</form>
</body>
</html>Servlet Code :
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class HelloWorld extends HttpServlet {
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
* Get the value of form parameter
     System.out.println("response char set "+response.getCharacterEncoding());
     System.out.println("request char set "+request.getCharacterEncoding());
response.setCharacterEncoding("UTF-8");
String name = request.getParameter("name");
String welcomeMessage = "Welcome "+name;
System.out.println("Name : "+name);
* Set the content type(MIME Type) of the response.
response.setContentType("text/html");
PrintWriter out = response.getWriter();
* Write the HTML to the response
out.println("<html>");
out.println("<head>");
out.println("<title> A very simple servlet example</title>");
out.println("</head>");
out.println("<body>");
out.println("<h1>"+welcomeMessage+"</h1>");
out.println("</body>");
out.println("</html>");
out.close();
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
     response.setCharacterEncoding("UTF-8");
     doPost(request, response);
} web.xml :
<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.4"
xmlns="http://java.sun.com/xml/ns/j2ee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
<servlet>
<servlet-name>HelloWorld</servlet-name>
<servlet-class>HelloWorld</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>HelloWorld</servlet-name>
<url-pattern>/HelloWorld</url-pattern>
</servlet-mapping>
<welcome-file-list>
<welcome-file>
HelloWorld.html
</welcome-file>
</welcome-file-list>
<context-param>
     <param-name>PARAMETER_ENCODING</param-name>
     <param-value>UTF-8</param-value>
</context-param>
</web-app>please have a look at my files and suggest me BalusC.
Thanks for your patience.
regards,
Achchayya
Edited by: achayya on Oct 16, 2009 7:26 AM

Similar Messages

Non ASCII characters are converted to '?' or ASCII characters

Non ASCII symbols like æ ø in the xml file have been converted to ? or other ascii characters.
What could be the reason behind this.

Mayil wrote:
This file we are loading through the Flex application in the front end.
Through java class file we are making changes to this city.xml file and adding and deleting this information in the city.xml.
Now suddenly, i dont know what happen.. 'ø' in the city name has replaced with the '?'
If we try to chaange this to 'ø' also, it again changes to '?'.
I dont know how to rectify this error.I would suggest you start by finding out when it happens. Does it happen as soon as you change the XML through this mysterious "java class file"? Or does it happen when Flex reads it? And is the underlying file actually changing, or are you just seeing those question marks after Flex handles the file?
In short a much better problem description is necessary.

Problems with password including non-ASCII characters

I am a German language user with a German keyboard but an English OS as main language. Therefore my passwords (simple user and admin) includes non-ASCII characters used in German, French and Spanish language, which increases security. This works fine in the majority of login scenarios. There are, however, 3 scenarios where neither my non-ASCII simple user nor my non-ASCII admin PW are accepted:
1) running "sudo" in Terminal;
2) When I try to shut down and another user account is still open. Doing this brings up a login window asking for the PW of the other user that does not accept non-ASCII;
3) Using Leopard/SnowLeopard CacheCleaner. Upon opening, this app asks for an admin PW, but does not recognize non-ASCII.
Am I right in assuming that this has to do with non-ASCII PWs? I thought ASCII times were gone given the remarkable language flexibility of Mac OS over the years. I know this stupid problem only from Win XP. There it is even worse.
Is there a way to overcome this problem without always temporarily changing my PW? Thanks.

I think the problem is with the applications themselves and should be reported to the developer. Although some non-ASCII characters are acceptable for an admin password, in my experience most Unix systems don't like non-ASCII characters in passwords. It may be easier to avoid them if you can.
OS X should simply request your admin password to shut down when another user account is open. An alert dialog usually appears warning that the other user is still logged in and giving you the option to log the other account out then shut down. But in my experience the only authorization needed is for your admin account.

Cannot rename file with non-ASCII characters when using the

My application moves files from one directory to another by calling File[] srcFiles = srcDir.listFiles() to get a list of files in the source directory, and then calling srcFiles.renameTo(destFile) to rename each file.
This does not work (renameTo returns false and the file is not moved) under the following circumstances:
- the file's leaf name contains non-ASCII characters, for example "�"
- the OS is Solaris 9
- the LANG and LC_* environment variables are unset, i.e. the C locale is being used
If I set the LANG environment variable to, for example, en_GB.UTF-8 then the rename succeeds.
I have tried calling srcFiles[index].getName().getBytes("UTF-8") and the non-ASCII characters are being replaced with ? (0x3f) characters when LANG is unset.
Is this a bug in the JRE? I would argue that since my code does not actually manipulate the filename (I just use the File object that File.listFiles() gives me) then the rename should succeed. Of course I would not expect the file name to be displayed correctly if I printed it out.
I have reproduced this behaviour with JDK 1.4.2_05 and 1.5.0_04 on Solaris 9.
Francis

Thanks for the info Alan.
I considered setting the locale in the environment (this sounds like the "correct" fix to me and we might implement it later), but this application shares a WebLogic server with many other applications so we would have to do a huge amount of testing to make sure that the locale change wouldn't break the other apps. In the end I worked around the problem by making the code that generates the filenames in the first place strip out any non-ASCII characters (the names of the files are not critically important).
Looking forward to JSR-203, in the meantime perhaps a note about this behaviour in the java.io.File javadoc would be useful.

CMSDK Non-ASCII Characters and WebFolders

Hi,
i have the follow problems with the CMSDK and Microsoft.
windows-explorer:
It is impossible to enter a folder that contains non-ascii characters in the name.(the clientrequest will never send.)
After the doubleclick on the foldername, i get a errormessage an then the url in the editbox is ISO-8859-1 encoded, but this url will never send to the sever.
Other operations like create, rename, ... have no problems with non-ascii chars.
with the IE, i can enter without a problem.
MS Word:
I can't save any file with non-ascii chars in the name.
Only this methods are send:
PROPFIND
PROPFIND
GET
GET
But never a "PUT", without a non-ascii char in the name the traffic looks like this:
PROPFIND
PROPFIND
GET
GET
PROPFIND
LOCK
PUT
It is also impossible to enter a folder containing non-ascii characters in the name, form the word filesave-dialog.
URL UTF-8 encoding is enabled in the IE options and other operations (MOVE,COPY) are send correctly UTF-8 encoded.
is there any solution?
thanks
Maik

Have you set the following DAV Server configuration property:
IFS.SERVER.PROTOCOL.DAV.Webfolders.DefaultCharset
for your domain?
You have to configure it for the character set that you want your clients to use when connecting to iFS via WebDAV.
(You can use the web admin tool to change this property.)
The reason for this is that the Microsoft WebFolders client software does not transmit the client's character set to the server, so the server has no way of knowing what to expect.

Cannot login with password containing non-ascii characters

Hello,
I have web application, form based login. UTF-8 is specified "everywhere".
And it works, except for passwords.
If user register itself with password containing non-ascii characters, it is correctly written in database, but when doing either programmatic login or normal form based login, if fails.
If the password is only ascii, it works.
Username of login could be ascii or non-ascii, it doesn't matter, both works.
I'm using sun java application server 9.1.
jdbc realm.
I'm not using hashing passwords, just clean (now)
I tried configure realm Charset: UTF8 as last chance, but it doesn't work either.
The problem is only with non-ascii characters in password.
Any help very appreciated
Thanks a lot

hi,
I know all that, but that's not the case. My app uses preparedStatements, everything is properly configured, in all pages, utf-8 is going from user to db and back without any problems.
The only problem is with password field. As I am using form based login, with jdbc realm configured (again, nicely working when only ascii characters), I have very little chance to do something bad through the login phase.
I'm not talking about special characters, I'm talking about non-ascii characters, let's say - Chinese, arabish, Russian alphabet etc.
When user registers (my code), the fields are properly written to db. I have checked that, trust me.
But the Sun app server realm seems to have some problems with the password field.
(realm uses jdbc connection to mysql, the url contains all extra parameters to be sure about utf8. there is nothing more what can be configured...)
If I try other alphabet codes in login and ascii in password, it works. But soon, as I use other alphabet code also in password, it doesn't work anymore.
My only idea is, that I could try MD5 to create ascii only characters (I hope it works that way) on the client with javascript and then set Digest to MD5 in realm configuration. But still, it seems very strange. The clear way storage should also function? (now set Digest to 'none')
Is it a bug of Sun App Server?
thanks

Replacing non-ASCII characters with HTML charcter references

Hi All,
In Oracle 10g or greater is there a built-in function that will convert a string with non-ASCII characters like this
a b č 뮼
into an ASCII string with HTML character references like this?
a b & # x 0 1 0 D ; & # x B B B C ;
(note I had to include spaces between each character in the sample code for message to prevent the forum software from converting my text)
I tried using
utl_i18n.escape_reference( val, 'us7ascii' )
but for some reason it returns
a b c & # x B B B C ;
Note how it converted the Western European character "č" to its unaccented counterpart "c", not "& # x 0 1 0 D ;" (is this a bug?).
I also tried a custom solution using regexp_replace and asciistr (which I can't include here because the forum software chokes on it) but it only returns the correct result for values <=4000 characters long. Unfortunately asciistr doesn't appear to accept CLOB values larger than 4000 characters. It returns an error message like
(ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion (actual: 30251, maximum: 4000) ).
I'm looking for a solution that works on CLOB data of any size.
Thanks in advance for any insight you can provide.
Joe Fuda

So with that (UTF8) in mind, let's take another look.....
As shown below, I used a AL32UTF8 database.
Note: I did not use a unicode capable tool for querying. So I set console mode code page to 1250 just to have č displayed properly (instead of posing as an è).
Also, as a result of using windows-1250 for client character set, in the val column and in the second select's ncr column (iso8859-1), è (00e8) has been replaced with e through character set conversion going from server back to client.
Running the same code on a database with a db character set such as we8mswin1252, that doesn't define the č (latin small c with caron) character, would yield results with a c in the ncr column.
C:\>chcp 1250
Aktuell teckentabell: 1250
C:\>set nls_lang=.ee8mswin1250
C:\>sqlplus test/test
SQL*Plus: Release 11.1.0.6.0 - Production on Fri May 23 21:25:29 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the OLAP option
SQL> select * from nls_database_parameters where parameter like '%CHARACTERSET';
PARAMETER              VALUE
NLS_CHARACTERSET       AL32UTF8
NLS_NCHAR_CHARACTERSET AL16UTF16
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'us7ascii') NCR from dual;
VAL NCR
č e c e
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'we8iso8859p1') NCR from dual;
VAL NCR
č e &# x10d; e     <- "è"
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'ee8iso8859p2') NCR from dual;
VAL NCR
č e č &# xe8;
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'cl8iso8859p5') NCR from dual;
VAL NCR
č e &# x10d; &# xe8;In the US7ASCII case, where it should be possible for all non-ascii characters to be escaped, it seems as if the actual escape step is skipped over.
Hope this helps to understand whether utl_i8n is usable or not in your case.
Message was edited by:
orafad
Fixed replaced character references :)

Problem searching some PDF files in Acrobat Reader – Non-ASCII characters

Acrobat Reader cannot search some .pdf files. I have put an example document up on Scribd here.
Any attempt to search for any word that can be clearly seen to be in the document fails with “No matches were found.”
This example document is NOT a scanned document – words and characters can be selected.
A hex display tool shows that the characters in a PDF document that can be successfully searched are in the ASCII/1252 range (A=0x41, etc).
Copying and pasting characters in the example document to a hex display tool shows that the characters in the document are not in the ASCII range.
For example the letters A to Z in the example document are in the range ‘A’ = 0xDF (decimal 223), ‘B’ = 0xDE (decimal 222), through to ‘Z’ = 0xC6 (decimal 198).
However, characters in these non-ASCII ranges are displayed perfectly by Acrobat Reader, as can be see if the example document is opened.
Therefore, as Acrobat Reader knows what these characters are, it doesn’t seem unreasonable to say that it should be able to search for and find them.
Tests were performed using Acrobat Reader X v10.1.4.
Can anyone say what this problem is?

Hi Pat, thanks for your reply.
Your reference to the title of that page being 'HARNESSES' indicates that, when you view that document in Adobe Reader, you are seeing 'HARNESSES', not
"ØßÎÒÛÍÍÛÍ". And that the remainder of the document is similarly being displayed in readable English language.
Yes as you say, you can search for 'ß' and get hits on 'A' (to use that as an example) in the example document.
But the need to form a word to be searched for into whatever code mapping this is using (for example having to enter "ØßÎÒÛÍÍ" for HARNESSES - I'm not even sure how that would be entered from a keyboard) doesn't seem to be very convenient.
Its clear the example document is using some code mapping other than ASCII / Windows-1252 (which has 'A' as 0x41). But it is also clear that Adobe Reader knows what that mapping is, and knows to use it, as its displaying (for example) 'A' for the code 0xDF.
So I guess the question is - why isn't Adobe Reader's knowledge of this mapping being extended to its search input?

Non - ASCII characters in textinput box

Hi,
I have a flex application where I have a TextInput box. If you paste the following (non-ascii characters) into it:
"" ‘’¡¢£¤¥¦§¨©ª«¬ ®¯°±²³´µ¶·¸¹º®¯°±²³µ´¶µ·¹¸º»¼½¾¿ÀÁÈÉÊËÌÍÏÎÐÒÔØÖöõôóòññó"
all you are left with is:
I am guess its some configuration that I am missing and it would be something trivial for someone who knows the issue
Any help is well appreciated.
Thanks
Hemant

You may need to embed the font:
http://livedocs.adobe.com/flex/3/html/help.html?content=fonts_04.html
If this post answers your question or helps, please mark it as such.

Validation for non-ASCII characters

Hi all,
Requirement: I have to apply a validation on on fields like Name and Address in applicationdefination.xml. When a user types non-ASCII characters and navigates to next page then it should display the error message. Thus, I have to restrict my user to ASCII values only.
Present Situation: I'm using regular expression for this problem. In Jheadstart there is an option regular expression under the heading Validation. I have written following values in regular expression and Regular Expression Error Message options.
Regular Expression
^\s*[\w\.\,\-\_\#\'\/\\\ u0022\u0026\*\;\:\s]+\s*$
Regular Expression Error Message
It is important to note that foreign characters are not accepted on our system. Please ensure only standard English letters are entered
Since, i was getting error in jspx page due to double quotes(") and ampercent(&), So i have replaced the double quotes(") and amprecent(&) by their unicodes. Thus, the expression has become like ^\s*[\w\.\,\-\_\#\'\/\\\u0022\u0026\*\;\:\s]+\s*$.
This expression is validating many characters like Ã,µ,Ç,Ï,Ö,§,¥,{,} but not all non ASCII characters like ѓ є ѕ ї Њ Щ Ώ Ω Ϊ Ά Ή Θ Λ Ξ Π τ ẫ ờ Ỡ Ứ Ỷ ự Ẁ ỹ ị Ọ ň ũ ť ţ Έ Ϊ ﻍ. Thus, its not fulfilling the requirement.
Please suggest some valid solution to this problem. It’s very urgent.

Hi,
The validation seems to be performed in Java or Javascript depending on the layout (I'm sorry I can't remember the exact details). The expression suggested above by theEternalStudent works very well in Java, but not in Javascript.
We came up with an expression which works in both. It rejects strings which contain &# by doing a lookahead before the main pattern - you might want to expand this to look for &#nnn; but for our purposes &# is enough.
Here is the "platform neutral" solution:
(?!.*\u0026#.*)^[\w\.\,\-\_\#\'\/\\\u0022\u0026\*\;\:\s]+$
I think in future we will write a javascript function and amend the templates to call it directly.
thanks,
Michael

[SOLVED] KDEmod - problem with mounting b/c of non-ASCII characters

Hi guys!
I finally set aside a few gigabites for Archlinux - it is no more in a virtual machine So far I managed to configure everything with the excellent wiki. It's runnin' and kickin'. I run accross only one problem:
When I insert a CD with a label that has non-ASCII characters (some Polish ones in my case) and I click on it's icon in Konqueror I get the message that "file such-and-such doesn't exist" - and the Polish characters are clearly misspelled (it is not a fonts' problem - I double checked). I can access the folder either via console or via konqueror if I go to the /media folder, though.
Any ideas how I can fix it? If you need more info, let me know.
Last edited by JeremyTheWicked (2008-05-31 14:46:07)

You're welcome . Now it's advisable for you to edit the title of your initial post: add [SOLVED]. Perhaps more clear wording would be in order, too, for the benefit of the search engine. The problem seems to be a trifle in retrospect, but somehow it takes some effort to find the solution, doesn't it ?

Search for users and non-ASCII characters

I am having a little issue with the "Accounts - Find Users" functionality. The search breaks on what I assume is non-ASCII characters (we use the following three up here in Denmark: �, �, �). To be precise, I have a user with the first name "J�rgen". Searching for first names starting with "J" works just fine but "J�" returns zero matches.
My setup is with two machines, one (A) holding the MySQL database and one (B) serving Identity Manager on top of tomcat.
Both A and B are RHEL boxes, and both have da_DK.UTF-8 as default locale.
MySQL's /etc/my.cnf file has the following entry (as recommended in create_waveset_tables.mysql):
[mysqld]
default-character-set=utf8
default-collation=binFor clarity, some functionality works just fine in Identity Manager with these non-ASCII characters such as adding a user whose name contains non-ASCII characters (not only �� but also � for example). At the moment, it appears to be the search functionality which is not working correctly as I would expect it to. I'm still on the fence concerning whether I've missed something in terms of configuration, or whether this is a limitation.
Does anyone know whether this problem is on my side or the software's side?

I am having a little issue with the "Accounts - Find Users" functionality. The search breaks on what I assume is non-ASCII characters (we use the following three up here in Denmark: �, �, �). To be precise, I have a user with the first name "J�rgen". Searching for first names starting with "J" works just fine but "J�" returns zero matches.
My setup is with two machines, one (A) holding the MySQL database and one (B) serving Identity Manager on top of tomcat.
Both A and B are RHEL boxes, and both have da_DK.UTF-8 as default locale.
MySQL's /etc/my.cnf file has the following entry (as recommended in create_waveset_tables.mysql):
[mysqld]
default-character-set=utf8
default-collation=binFor clarity, some functionality works just fine in Identity Manager with these non-ASCII characters such as adding a user whose name contains non-ASCII characters (not only �� but also � for example). At the moment, it appears to be the search functionality which is not working correctly as I would expect it to. I'm still on the fence concerning whether I've missed something in terms of configuration, or whether this is a limitation.
Does anyone know whether this problem is on my side or the software's side?

Replacing non-ascii characters in String

I have a site where the user enters data in a rich text
editor (ktml4) that gets stored into a database (mysql). There are
non ascii characters getting into the data, I'm assuming that they
are copying and pasting from Word. Unfortunately in this situation,
changing that process isn't an option.
Currently, this is the only character that is causing me
problems:
http://www.zvon.org/other/charSearch/PHP/search.php?request=ffa0&searchType=3
I would just like to replace the non-ascii characters with a
space when I read them from the database. Something like:
#Replace(result.column, '\xffa0', ' ')#
However, I believe that code looks for the string "\xffa0",
not the character \xffa0.
Is there anyway to do this?

quote:
Originally posted by:
BuckLemke
quote:
Originally posted by:
Dan Bracuk
rereplace might work.
Can you give an example of how to pass a non-ascii character
to REReplace?
Regular expressions are not my strength, but the approach I
was considering was, "if it's not an ascii character, make it a
space". Then you pass the entire string at once.

Non-ASCII Characters Appear In Path Name

I've had this happen before over SSH via PuTTY on a Windows Host but now it's happening just in Linux. I'm testing out my first attempt at remastering the Arch Live CD and my home directory never got created while running rc.local, when I tried to execute it manually I got block characters at the beginning and end of the path. I tried to type the command manually and got the same thing. I enabled en_US ISO-8859-1 along with en_US.UTF-8 UTF-8 in /etc/locale.gen and re-ran locale.gen but it didn't seem to fix my issue. Anyone have any idea on how to fix this?
Last edited by brando56894 (2013-08-30 17:01:04)

What do you mean by non-ASCII characters?
The ASCII character set contains characters for all byte values, however not all of them are displayable characters. Is it non-displayable characters you want?

Find non ascii characters on Table

I am trying to find all non-ascii characters in a specific field on a 10.2G table.
Any select statement that can help?
Thanks!

Charlov wrote:
I am trying to find all non-ascii characters in a specific field on a 10.2G table.
Any select statement that can help?
Thanks!Perhaps something like this...
select *
from table
where regexp_like(column,'[^ -'||chr(126)||']')Assuming you are taking ascii characters as being space to chr(126)

Non ascii characters are padding without reason

Similar Messages

Maybe you are looking for