CF8 - XmlFormat not escaping High ASCII characters

In CF8, we have a problem where XmlFormat is not escaping
High ASCII characters. This was working just fine on our CF7
instance, but in CF8, it is not escaping all characters. I am aware
of the long-standing problem with escaping Windows-1252 characters,
but now we are experiencing an issue with basic high ASCII
characters, like chr(233) and chr(244). Is anyone else experiencing
this issue? We have not installed Update 1 to CF8 yet. I don't see
a fix for this in the release note, but any word on if this is
fixed by the updater?
Here is a test to demonstrate the issue:
<cfset myString = "The Islamic Republic of Mauritania's
(République Islamique de Mauritanie) 2007 estimated population
is 3,270,000. Cote d'Ivoire and Côte d'Ivoire">
<cfset myNewString = XmlFormat(myString)>
<cfoutput>#myNewString#</cfoutput>

BKBK,
Thanks for the info. Adding the processingdirective does help
show that these characters are being escaped, however, the behavior
has changed somewhat between CF7 and CF8, as we were not using a
processingdirective in CF7, and this was working as advertised.
Where this is giving us a problem is after we create an XML
document using CFXML, (ensuring that we XmlFormat any strings), we
then validate that document against a schema, and we are all of a
sudden getting errors during validation for invalid characters
within the XML. We are using ToString() after creating the XML
document with CFXML, and our process is the same as we were using
in CF7. That is why I was curious if anyone else was having this
same issue... because something definitely changed between CF7 and
CF8 with XML processing.

Similar Messages

Convert smart quotes and other high ascii characters to HTML

I'd like to set up Dreamweaver CS4 Mac to automatically convert smart quotes and other high ASCII characters (m-dashes, accent marks, etc.) pasted from MS Word into HTML code. Dreamweaver 8 used to do this by default, but I can't find a way to set up a similar auto-conversion in CS 4. Is this possible? If not, it really should be a preference option. I code a lot of HTML emails and it is very time consuming to convert every curly quote and dash.
Thanks,
Robert
Digital Arts

I too am having a related problem with Dreamweaver CS5 (running under Windows XP), having just upgraded from CS4 (which works fine for me) this week.
In my case, I like to convert to typographic quotes etc. in my text editor, where I can use macros I've written to speed the conversion process. So my preferred method is to key in typographic letters & symbols by hand (using ALT + ASCII key codes typed in on the numeric keypad) in my text editor, and then I copy and paste my *plain* ASCII text (no formatting other than line feeds & carriage returns) into DW's DESIGN view. DW displays my high-ASCII characters just fine in DESIGN view, and writes the proper HTML code for the character into the source code (which is where I mostly work in DW).
I've been doing it this way for years (first with GoLive, and then with DW CS4) and never encountered any problems until this week, when I upgraded to DW CS5.
But the problem I'm having may be somewhat different than what others have complained of here.
In my case, some high-ASCII (above 128) characters convert to HTML just fine, while others do not.
E.g., en and em dashes in my cut-and-paste text show as such in DESIGN mode, and the right entries
    –
    —
turn up in the source code. Same is true for the ampersand
    &
and the copyright symbol
    ©
and for such foreign letters as the e with acute accent (ALT+0233)
    é
What does NOT display or code correctly are the typographic quotes. E.g., when I paste in (or special paste; it doesn't seem to make any difference which I use for this) text with typographic double quotes (ALT+0147 for open quote mark and ALT+0148 for close quote mark), which should appear in source code as
    “[...]”
DW strips out the ASCII encoding, displaying the inch marks in DESIGN mode, and putting this
    "[...]"
in my source code.
The typographic apostrophe (ALT+0146) is treated differently still. The text I copy & paste into DW should appear as
    [...]’[...]
in the source code, but instead I get the foot mark (both in DESIGN and CODE views):
I've tried adjusting the various DW settings for "encoding"
    MODIFY > PAGE PROPERTIES > TITLE/ENCODING > Encoding:
and for fonts
    EDIT > PREFERENCES > FONTS
but switching from "Unicode (UTF-8)" to "Western European" hasn't solved the problem (probably because in my case many of the higher ASCII characters convert just fine). So I don't think it's the encoding scheme I use that's the problem.
Whatever the problem is, it's caused me enough headaches and time lost troubleshooting that I'm planning to revert to CS4 as soon as I post this.
Deborah

Web services and High Ascii characters

Hello Everyone
I have a problem regarding webservices and sending high ascii
characters.
My little application consists of two files: one is cfm that
makes a view and the second is cfc that might be working as the
component or web service.
The goal of the application is to handle text files, divide
them according to some rules and then send back to the user.
Sometimes the text file contains few high ascii characters.
Everything works fine as long as the cfc is called as the
component. Unfortunately when I call the cfc as the webservice I
get an error:
The web service operation caused an invocation exception.The
root cause was that: java.lang.IllegalArgumentException: The char
'0x1' in 'java.lang.IllegalArgumentException: and so on....
AxisFault
faultCode: {
http://xml.apache.org/axis/}HTTP
faultSubcode:
faultString: (500)Internal Server Error
faultActor:
faultNode:
faultDetail:
{}:return code: 500
Do you know how can I solve my problem? I've tried different
stuff with encoding but it didn't work.
Thanks for all answers.

Whatever Coldfusion's means of communication with the database, by default, it represents text meant for the browser in UTF-8.

Email can not display Extended Ascii Characters

Hi every One,
In my Project there is a One option in that i can send Product Information as well as Customer Address to specify Email id , In Customer Address Have some Ascii Characters .After Sending a Mail when i Check Email that Ascii Character replace by ? .
I am trying that one , still i dont know where is Going wrong, Please any one Know Regading this Problem let me Know
with Regards
Asif

I sure hope you get that sorted out. sounds annoying
right, back to the java.....

Searching on high ascii characters

Hi all,
I am writing a search engine built on Oracle text (otherwise why would i be posting here??) to search lists of medical articles. Many of the article titles have special characters in the title like cedilla's and umlaut's. Some of my users will have european keyboards and will be searching using these characters, other will not and would demote such characters to the low ascii value. Currently all special characters are stored in the table as html encoded values, but this could easily be changed to something else that is supported by Oracle Text.
Example: I have a article called "Behçet's syndrome" this needs to match if someone search with Behçet or Behcet.
EMP has two records one with ename = Behcet and the other ename = Behçet
SELECT ENAME
FROM EMP
WHERE contains (ENAME, 'Behcet', 100) > 0
only returns one row.
I have looked at using SYN to provide this functionality and it works but means that i continually have to update and maintain that list of synonyms which is a chore and something i want to avoid.
Is there a way to build a index that covers both possibilities? So provide a list of special characters and what they would degrade into? I feel that there is a very simple elegant solution to this just waiting for me, any suggestions very welcome.
thanks
Toby

SCOTT@10gXE> CREATE TABLE articles (id NUMBER, title VARCHAR2 (30))
2 /
Table created.
SCOTT@10gXE> INSERT ALL
2 INTO articles VALUES (1, 'Behçet''s syndrome')
3 INTO articles VALUES (2, 'Behcet''s syndrome')
4 INTO articles VALUES (2, 'compulsive tuning disorder')
5 SELECT * FROM DUAL
6 /
3 rows created.
SCOTT@10gXE> EXEC CTX_DDL.CREATE_PREFERENCE ('your_lexer', 'BASIC_LEXER')
PL/SQL procedure successfully completed.
SCOTT@10gXE> EXEC CTX_DDL.SET_ATTRIBUTE ('your_lexer', 'BASE_LETTER', 'YES')
PL/SQL procedure successfully completed.
SCOTT@10gXE> CREATE INDEX articles_idx ON articles (title)
2 INDEXTYPE IS CTXSYS.CONTEXT
3 PARAMETERS ('LEXER your_lexer')
4 /
Index created.
SCOTT@10gXE> SELECT token_text FROM dr$articles_idx$i
2 /
TOKEN_TEXT
BEHCET
COMPULSIVE
DISORDER
SYNDROME
TUNING
SCOTT@10gXE> SELECT * FROM articles WHERE CONTAINS (title, 'Behçet') > 0
2 /
        ID TITLE
         1 Behçet's syndrome
         2 Behcet's syndrome
SCOTT@10gXE> SELECT * FROM articles WHERE CONTAINS (title, 'Behcet') > 0
2 /
        ID TITLE
         1 Behçet's syndrome
         2 Behcet's syndrome
SCOTT@10gXE>

High ASCII in discussion forums

When using Safari 2.0.3 on many discussion forums, "High ASCII" characters such as greek letters and mathematical symbols will not display in my posts. I wonder what could be the problem, because if I type the same characters in Camino, they display properly in my posts.

When using Safari 2.0.3 on many discussion forums,
"High ASCII" characters such as greek letters and
mathematical symbols will not display in my posts. I
wonder what could be the problem, because if I type
the same characters in Camino, they display properly
in my posts.
Many discussion forums use a Latin-1 encoding. This means the browser must convert anything above ascii into numerical references like & #12345;. Mozilla-type browsers do this, but Safari does not. Forums intended to accomodate Greek or Math should use UTF-8, like this one does.

Validation for non-ASCII characters

Hi all,
Requirement: I have to apply a validation on on fields like Name and Address in applicationdefination.xml. When a user types non-ASCII characters and navigates to next page then it should display the error message. Thus, I have to restrict my user to ASCII values only.
Present Situation: I'm using regular expression for this problem. In Jheadstart there is an option regular expression under the heading Validation. I have written following values in regular expression and Regular Expression Error Message options.
Regular Expression
^\s*[\w\.\,\-\_\#\'\/\\\ u0022\u0026\*\;\:\s]+\s*$
Regular Expression Error Message
It is important to note that foreign characters are not accepted on our system. Please ensure only standard English letters are entered
Since, i was getting error in jspx page due to double quotes(") and ampercent(&), So i have replaced the double quotes(") and amprecent(&) by their unicodes. Thus, the expression has become like ^\s*[\w\.\,\-\_\#\'\/\\\u0022\u0026\*\;\:\s]+\s*$.
This expression is validating many characters like Ã,µ,Ç,Ï,Ö,§,¥,{,} but not all non ASCII characters like ѓ є ѕ ї Њ Щ Ώ Ω Ϊ Ά Ή Θ Λ Ξ Π τ ẫ ờ Ỡ Ứ Ỷ ự Ẁ ỹ ị Ọ ň ũ ť ţ Έ Ϊ ﻍ. Thus, its not fulfilling the requirement.
Please suggest some valid solution to this problem. It’s very urgent.

Hi,
The validation seems to be performed in Java or Javascript depending on the layout (I'm sorry I can't remember the exact details). The expression suggested above by theEternalStudent works very well in Java, but not in Javascript.
We came up with an expression which works in both. It rejects strings which contain &# by doing a lookahead before the main pattern - you might want to expand this to look for &#nnn; but for our purposes &# is enough.
Here is the "platform neutral" solution:
(?!.*\u0026#.*)^[\w\.\,\-\_\#\'\/\\\u0022\u0026\*\;\:\s]+$
I think in future we will write a javascript function and amend the templates to call it directly.
thanks,
Michael

Normal string ASCII characters are not shown after using string reversal.

In a VI that I have written, there is a point where I take a simple 5 character string and input it to a reverse string node. I also have an indicator on the output of the reverse string node so I can look at the reversed string. Both the input and output indicators are in normal display mode. For the input indicator I can see the input string with no problem. But on the indicator at the output of the reverse string node I see nothing. If I then put each of the indicators in HEX display mode, both indicators display the hex equivalent characters of the original ASCII characters, and the output indicator shows the characters reversed from that of the input indicator, as expected. So why when I am in normal display mode, are the characters in the output indicator not showing up? Thanks.

What are the ASCII values that you are trying to display? Perhaps they are non-printable ASCII values.
Perhaps you can post a simple VI with the values in a string control and saved as a default.
EDIT: After seeing Rod's message, that seems very likely, especially considering that there is likely an EOL character if you are doing serial communication with some other device.
Message Edited by Ravens Fan on 03-23-2009 12:04 PM

A Download servlet: non-ASCII characters not working

This is my servlet used for file download:
public void doPost(HttpServletRequest request, HttpServletResponse response) {
String filepath = request.getParameter("filepath");
String filename = request.getParameter("filename");
response.setContentType("application/zip");
response.setHeader("Content-Disposition", "attachment;filename=\""+filename+"\";");
ServletOutputStream sos = null;
BufferedInputStream bis = null;
try {
    sos = response.getOutputStream();
    bis = new BufferedInputStream(new FileInputStream(source));
    byte buffer[] = new byte[2048];
    int c;
    while((c = bis.read(buffer)) != -1)
      sos.write(buffer, 0, c);
} catch(Exception e) {
} finally {
    bis.close();
    sos.close();
}It does not work when the filename contains non-ASCII characters (e.g., extended ASCII, CJK ...)
What do I fix this? Thanks!

One possiblitiy that occurs to me is you have too many encoding things going on and you are sorta "over-encoding" things, as it were....
All I can think to do is give you this sample JSP page that I created when I was trying to figure all this web encoding stuff with forms back in the day. So perhaps you can use this as a basis for your own page.
// _lang.jsp
<%@ page language="java" contentType="text/html; charset=UTF-8" %>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
     <title></title>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body bgcolor="#ffffff" background="" text="#000000" link="#ff0000" vlink="#800000" alink="#ff00ff">
<%
request.setCharacterEncoding("UTF-8");
String str = "\u7528\u6237\u540d";
String name = request.getParameter("name");
%>
req enc: <%= request.getCharacterEncoding() %><br />
rsp enc: <%= response.getCharacterEncoding() %><br />
str: <%= str %><br />
name: <%= name %><br />
<br />
<a href="_lang.jsp?name=<%= java.net.URLEncoder.encode(str, "UTF-8") %>">as link</a>
<br />
<br />
<form method="GET" action="_lang.jsp" encoding="UTF-8">
Name: <input type="text" name="name" value="" >
<input type="submit" name="submit" value="GET Submit" />
</form>
<form method="POST" action="_lang.jsp" encoding="UTF-8">
Name: <input type="text" name="name" value="" >
<input type="submit" name="submit" value="POST Submit" />
</form>
</body>
</html>

Replacing non-ASCII characters with HTML charcter references

Hi All,
In Oracle 10g or greater is there a built-in function that will convert a string with non-ASCII characters like this
a b č 뮼
into an ASCII string with HTML character references like this?
a b & # x 0 1 0 D ; & # x B B B C ;
(note I had to include spaces between each character in the sample code for message to prevent the forum software from converting my text)
I tried using
utl_i18n.escape_reference( val, 'us7ascii' )
but for some reason it returns
a b c & # x B B B C ;
Note how it converted the Western European character "č" to its unaccented counterpart "c", not "& # x 0 1 0 D ;" (is this a bug?).
I also tried a custom solution using regexp_replace and asciistr (which I can't include here because the forum software chokes on it) but it only returns the correct result for values <=4000 characters long. Unfortunately asciistr doesn't appear to accept CLOB values larger than 4000 characters. It returns an error message like
(ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion (actual: 30251, maximum: 4000) ).
I'm looking for a solution that works on CLOB data of any size.
Thanks in advance for any insight you can provide.
Joe Fuda

So with that (UTF8) in mind, let's take another look.....
As shown below, I used a AL32UTF8 database.
Note: I did not use a unicode capable tool for querying. So I set console mode code page to 1250 just to have č displayed properly (instead of posing as an è).
Also, as a result of using windows-1250 for client character set, in the val column and in the second select's ncr column (iso8859-1), è (00e8) has been replaced with e through character set conversion going from server back to client.
Running the same code on a database with a db character set such as we8mswin1252, that doesn't define the č (latin small c with caron) character, would yield results with a c in the ncr column.
C:\>chcp 1250
Aktuell teckentabell: 1250
C:\>set nls_lang=.ee8mswin1250
C:\>sqlplus test/test
SQL*Plus: Release 11.1.0.6.0 - Production on Fri May 23 21:25:29 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the OLAP option
SQL> select * from nls_database_parameters where parameter like '%CHARACTERSET';
PARAMETER              VALUE
NLS_CHARACTERSET       AL32UTF8
NLS_NCHAR_CHARACTERSET AL16UTF16
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'us7ascii') NCR from dual;
VAL NCR
č e c e
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'we8iso8859p1') NCR from dual;
VAL NCR
č e &# x10d; e     <- "è"
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'ee8iso8859p2') NCR from dual;
VAL NCR
č e č &# xe8;
SQL> select unistr('\010d \00e8') val, utl_i18n.escape_reference(unistr('\010d \00e8'),'cl8iso8859p5') NCR from dual;
VAL NCR
č e &# x10d; &# xe8;In the US7ASCII case, where it should be possible for all non-ascii characters to be escaped, it seems as if the actual escape step is skipped over.
Hope this helps to understand whether utl_i8n is usable or not in your case.
Message was edited by:
orafad
Fixed replaced character references :)

Ascii characters removal

hii experts,
this is shreya.i have a payload from where i need to remove the special characters searching them through their ascii codes.
Please suggest a solution to my problem.

Hi Shreyasen Gupta,
I think this issue not related to character encoding (ascii or UTF-8). It is related to well formed XML.
Please note:- quote ("), apostrophe ('), ampersand (&), less than (<), greater than (>) are special characters in XML. They should not be present in data.
http://www.w3.org/TR/REC-xml/ The ampersand character (&) and the left angle bracket (<) must not appear in their literal form, except when used as markup delimiters, or within a comment, a processing instruction, or a CDATA section. If they are needed elsewhere, they must be escaped using either numeric character references or the strings " & " and " < " respectively.
Escape characters u201C &quot ; , u2018 &apos ; , & amp ; , < &lt ; , > &gt ;
Not Well formed XML:
<Name>Raghuu2019s</Name> <Name> Raghu & Vamsee </Name>
Well formed XML:
<Name>Raghu's</Name> <Name>Raghu & Vamsee</Name>
It is very difficult (if not impossible) to escape & and <, to make XML 100 % well formed in PI. Escaping should be done at source side, when XML is getting generated.
If you have no control on sender side, then find out in which element of XML these characters are coming and then write a Java mapping to escape the special characters. This solution will not fix the issue to 100% accuracy, but I feel it is the best available solution. Note, Graphical and XML mapping cannot take Not Well formed XML as input, so they cannot be used.
Regards,
Raghu_Vamsee

Non ascii characters being sent from a parameter in a form

Hi!
I have seen many topics posted on passing non ascii characters through parameters from one servlet to another and converting them into whatever format is necessary.
However, I have not seen anyone answer the following question. I have a jsp page (html) with the character encoding set to utf-8. The user inputs some data in to a text field which is inside a form. The data could be in non ascii characters such as hebrew or arabic. This form is then sent to another jsp where i try to retreive the data from teh text field. No matter what i do, i cannot get the data presented correctly. It is either question marks or other wierd symbols.
I have tried every permetation of encoding of the actual html page, the ecoding of the string from request.getParameter etc but it still is not presented on the new html page correctly.
Can anyone help??
Spencer

Ok, I solved the problem.
I had to put at the top request.setCharacterEncoding("utf-8");
Spencer

How can I convert ASCII characters to ISO8859?

Hi All,
I have written a little application that renames a TV episode by scraping a TV listing site for the episode name. It is written in SWT and works great apart from on small problem. When getting the html back from the site, it sometimes contains special ASCII characters that are not in the ISO8859 (Windows filesystem) character set.
For example, this is the line that I have to parse:
<td style='padding-left: 6px;' class='b2'><a href='/Prison_Break/episodes/569183/03x01'>Orientaci��n</a></td>When viewing it in a browser, it is:
<td style="padding-left: 6px;" class="b2"><a href="/Prison_Break/episodes/569183/03x01">Orientaci�n</a></td>Notice that the o in the title has an accent on it. While researching this problem I stumbled across 'HTML Entities to ISO 8859-1 Converter' at http://www.inweb.de/chetan/English/Resources/Java/HTML%202%20ISO.html. This open source project takes in an html entity like & and returns '&'.
So that is not quite what I want, as my BufferedReader is converting the html entity into the ASCII representation already. I need a way of detecting a non ISO8859 character within an ASCII string, and hopefully replacing its natural 'equivalent' (would be o in this case).
Does anyone know how I could do it without having to check for every special char and replacing (not really an option unless someone has done it before!!)
If not that then, perhaps another way to attack the problem?
Any help greatly appreciated ;)
Dave

Hi,
NZ_Dave wrote:
For example, this is the line that I have to parse:
<td style='padding-left: 6px;' class='b2'><a href='/Prison_Break/episodes/569183/03x01'>Orientaci��n</a></td>
This is coded in UTF-8. If you convert the bytes to a String using the UTF-8 encoding, then you will have the correct characters "Orientaci�n" in the string.
Check your parser where it converts the bytes (coming from e.g. an InputStream) to characters. Use UTF-8 as the charset when doing that conversion.

How can I use ASCII Characters on the iPad?

I have an iPad (1) and use iOS 4.3.5. Is there an easier Way of using ASCII Characters than having to use the "copy-Paste" procedure? If not, can we expect to see the feature of entering the ASCII code straight through the standard keyboard soon?

Norisouro wrote:
. Is there an easier Way of using ASCII Characters than having to use the "copy-Paste" procedure?
You need to give some details about what it is you want to do, because ASCII Characters are what are already on the keyboard. It is non-ASCII that you might need to copy/paste.
http://en.wikipedia.org/wiki/ASCII
I think you can be sure that Apple is never going to include a feature in iOS that has you input special characters by typing in numbers like Windows does it. Mac's have always used a different approach.

Non US-ASCII characters in download file names

I am trying to implement a simple file download in a JSP, and trying to get IE, Firefox and Opera to all display and handle non US-ASCII characters in the suggested download file name. Only concerned with Windows platform for now. Here's the code I am currently using:
String agent = request.getHeader("USER-AGENT");
if (null != agent && -1 != agent.indexOf("MSIE"))
String codedfilename = URLEncoder.encode(cfrfilename, "UTF8");
response.setContentType("application/x-download");
response.setHeader("Content-Disposition","attachment;filename=" + codedfilename);
else if (null != agent && -1 != agent.indexOf("Mozilla"))
String codedfilename = MimeUtility.encodeText(cfrfilename, "UTF8", "B");
response.setContentType("application/x-download");
response.setHeader("Content-Disposition","attachment;filename=" + codedfilename);
else
response.setContentType("application/x-download");
response.setHeader("Content-Disposition","attachment;filename=" + cfrfilename);
}This URL encodes the file name if the browser is IE, MIME encodes it if the browser is Mozilla, and sends plain UTF-8 (the encoding of the JSP) for all other browsers. I get "cfrfilename" from translated properties files, and the string can contain characters from any character set - Chinese, Thai, Korean, etc.
This code works correctly for IE - the file name is displayed correctly in the file Save as dialog, and it is saved correctly on disk, no matter which character set is used.
For Firefox, the file name is displayed correctly in the file Save as dialog, but it is only saved correctly to disk if the file name is in a character set supported by the system locale. This seems to be a known Firefox bug (not fully using the Windows Unicode APIs), so nothing I can do about that.
Nothing seems to work for Opera, however - I cannot get the file name to display correctly in the file Save as dialog, no matter which method I use (I have tried URL encoding and MIME encoding in addition to the plain UTF-8).
Has anybody implemented something similar that works for at least these 3 browsers?

I tested your code today,
                     dialog           save           open
Firefox 1.5          OK                 OK               OK
IE 6.0                OK                 OK                NGdailog: filename show in download popup dialog
save: save to disk from dialog
open: open directly from dailog

CF8 - XmlFormat not escaping High ASCII characters

Similar Messages

Maybe you are looking for