Removing non UTF-8 character set from xml in OSB

Hi,
We have a OSB service where we are receiving lot special characters like (~, â€“ ) in the data between xml tags. As a result these messages are failing in our subsequent EDI systems (though they are getting processed successfully in our OSB). How do I remove these non UTF-8 characters when processing a xml message in OSB?
If I set the Request-Encoding to UTF-8 to the proxy service that is receiving these messages, would these messages be rejected?
Thanks,
Aditya

Hi,
No silver bullet here... I think you will need a java call in order to clean up the special characters from your message...
Cheers,
Vlad

Similar Messages

Oracle 11gr2 - change character set from utf-8 to weiso8859p15

Hi guys,
I've a new and empty oracle 11gr2 database for my test environment, during the installation i've choosed the UTF-8 character's set but now i've see that in the production environment there is the WEISO8859 character's set.
I have a dump of the prod. env. (done with Exp function).
How can I change the character set from utf-8 to weiso8859p15 ?
I found this procedure:
SHUTDOWN IMMEDIATE or a SHUTDOWN NORMAL;
STARTUP MOUNT;
ALTER SYSTEM ENABLE RESTRICTED SESSION;
ALTER SYSTEM SET JOB_QUEUE_PROCESSES=0;
ALTER SYSTEM SET AQ_TM_PROCESSES=0;
ALTER DATABASE OPEN;
ALTER DATABASE NATIONAL CHARACTER SET new_character_set;
SHUTDOWN IMMEDIATE; -- or SHUTDOWN NORMAL;
STARTUP;
is it correct ?
Thx.
Kind Regards,
Stefano.

evil.stefano wrote:
Hi guys,
I've a new and empty oracle 11gr2 database for my test environment, during the installation i've choosed the UTF-8 character's set but now i've see that in the production environment there is the WEISO8859 character's set.
I have a dump of the prod. env. (done with Exp function).
How can I change the character set from utf-8 to weiso8859p15 ?
I found this procedure:
SHUTDOWN IMMEDIATE or a SHUTDOWN NORMAL;
STARTUP MOUNT;
ALTER SYSTEM ENABLE RESTRICTED SESSION;
ALTER SYSTEM SET JOB_QUEUE_PROCESSES=0;
ALTER SYSTEM SET AQ_TM_PROCESSES=0;
ALTER DATABASE OPEN;
ALTER DATABASE NATIONAL CHARACTER SET new_character_set;
SHUTDOWN IMMEDIATE; -- or SHUTDOWN NORMAL;
STARTUP;
is it correct ?NO
leave the character set alone.
UTF-8 will not be a problem.

Invalid Characters shown in UTF-8 character set

There is an XMLP report whose template output character set is ISO-8859-1. The character set ISO-8859-1 is required for this report as per Spanish Authorities. When the report is run, output gets generated in the output directory file of application server. This output file doesn't contain any invalid characters.
But when the output is opened from SRS window, which opens it in a browser, the invalid characters are shown for characters like Ñ , É etc.
Investigation done:
Found that the output generated on the server is having ISO encoding and hence doesn't contain any invalid characters. Whereas the output generated from SRS window, it is in UTF encoding, so it seems the invalid characters are displayed when conversion takes place from ISO to UTF-8 format.
Created the eText output using the data xml and template using BI publisher tool, the output is in ISO encoding. So if i go and change the encoding to UTF-8 by opening it in explorer or Notepad++, invalid charcters are shown for Ñ, É etc.
Is there any limitation, that output from SRS window will show only in UTF-8 encoding? If not then please suggest.
Thanks,
Saket
Edited by: 868054 on Aug 2, 2012 3:05 AM
Edited by: 868054 on Aug 2, 2012 3:05 AM

Hi Srini,
When customer is viewing output from the SRS window, then it contains invalid characters because it is in UTF-8 character set. Customer is on Oracle OnDemand so they cannot take the output generated on the server.Every time they have to raise a request to Oracle for the output file. So the concern here is, why don't the output from SRS window show output with valid characters ?
The reason could be conversion of ISO format to UTF-8. How could this be resolved ? Does SRS window output cannot generate in ISO format ?
A quick reply will be appreciated as customer is chasing for an update.
Thanks,
Saket
Edited by: 868054 on Aug 7, 2012 11:08 PM

Non UTF-8 Character in V$session

Hello
The oracle clusterware process racgimon is putting a Non utf-8 character into v$session and so causing problems. The value is NT-AUTORITÄT\SYSTEM. Any ideas of how to correct this ?
Database Parameters
NLS_LANGUAGE     GERMAN
NLS_TERRITORY     GERMANY
NLS_CHARACTERSET     AL32UTF8
Thank you.

I did something along that line (but I used a different test account since System account localisation is different on my test machine) - see below.
So far it would seem the case is that certain characters (code) in v$session.osuser is not handled properly, in general, not only for racgimon.exe.
Perhaps the assumption is that os account names for 'osuser' consists of ascii characters only (and read as such, even if codes 0x80-FF is used).
The value of CharacterSet from dump() output above is misleading for this column as it isn't 'typed' correctly (as you say, non utf-8 representation), as it would be with regular tables'. Still, I believe something similar to "pass-through scenario" is going on.
Note on the use of [url http://download.oracle.com/docs/cd/B28359_01/server.111/b28286/functions027.htm]convert function, as it should not be used, however it says "+...the only practical use of the function is to correct data that has been stored in a wrong character set.+" which would seem to apply here, so I guess this temporary use of convert is ok.
The account setup for test is AüTÄT, just to involve some non-ascii characters.
SQL> select osuser
, dump(osuser,16) dump_incorrect_chars
, convert(osuser,'al32utf8','we8mswin1252') osusercorr
, dump(convert(osuser,'al32utf8','we8mswin1252'),16) dump_corrected
from v$session
where program = 'sqlplus.EXE';
OSUSER               DUMP_INCORRECT_CHARS
OSUSERCORR           DUMP_CORRECTED
Odd7\A¿T¿            Typ=1 Len=10: 4f,64,64,37,5c,41,fc,54,c4,54
Odd7\AüTÄT           Typ=1 Len=12: 4f,64,64,37,5c,41,c3,bc,54,c3,84,54I also found that oracle.exe process itself, started under Localsystem, is using a "common" name instead. It shows up as SYSTEM instead of LocalSystem or principal name in v$session.osuser. Perhaps this is related to how/from where process is spawned, via SCM or not (e.g. Scheduler).
SQL> select sid, osuser, program from v$session;
SID OSUSER               PROGRAM
   2 SYSTEM               ORACLE.EXE (PMON)
15 SYSTEM               ORACLE.EXE (MMON)
16 SYSTEM               ORACLE.EXE (MMNL)
17 NT instans\SYSTEM    sqlplus.EXEI guess, to avoid this mess, we need to find some workaround such as:
- change the client process os account i.e. start as .\Localsystem or account other than Localsystem.
- change locale for the user account so that non localised name NT-AUTHORITY\SYSTEM is used
- change the way process is started, so that user name is picked up without the NT-AUTORITÄT prefix.
Hope this helps.
Edited by: orafad on May 25, 2011 1:59 PM
Edited by: orafad on May 25, 2011 2:25 PM

Approach to converting database character set from Western European to Unicode

Hi All,
EBS:12.2.4 upgraded
O/S: Red Hat Linux
I am looking for the below information. If anyone could help provide would be great!
INFORMATION NEEDED: Approach to converting database character set from Western European to Unicode for source systems with large data exceptions
DETAIL: We are looking to convert Oracle EBS database character set from Western European to Unicode to support Kanji characters. Our scan results show
both “lossy (110K approx.)” and “truncation (26K approx.)” exceptions in the database which needs to be fixed before the database is converted to Unicode.
Oracle Support has suggested to fix all open and closed transactions in the source Production instance using forms and scripts.
We’re looking for information/creative approaches who have performed similar exercises without having to manipulate data in the source instance.
Any help in this regard would be greatly appreciated!
Thanks for yourn time!
Regards,

There are two aspects here:
1. Why do you have such large number of lossy characters? Is this data coming from some very old eBS release, i.e. from before the times of the Java applet interface to Oracle Forms? Have you analyzed the nature of this lossy data?
2. There is no easy way around truncation issues as you cannot modify eBS metadata (make columns wider). You must shorten or remove the data manually through the documented eBS interfaces. eBS does not support direct manipulation of data in the database due to complex consistency rules enforced by the application itself (e.g. forms).
Thanks,
Sergiusz

Unable to migrate table, character set from WE8MSWIN1252 to AL32UTF8

Hi,
On our source db the character set is AL32UTF8
On our own db, we used the default character set of WE8MSWIN1252 .
When migrating one of the table, we get an error of this: ORA-29275: partial multibyte character
So in to alter our character set from WE8MSWIN1252 to AL32UTF8, we get this error:
ALTER DATABASE CHARACTER SET AL32UTF8
ERROR at line 1:
ORA-12712: new character set must be a superset of old character set
I would sure not like to reinstall the db and migrate the tables again. Thanks.

See this related thread - Re: Want to change characterset of DB
You can use the ALTER DATABASE CHARACTER SET command in very few cases. You will most likely have to recreate the database and re-migrate the data.
HTH
Srini

How to input unicode character set from oralce form 9i

Hi,
Can anyone show me how to input unicode character set from form 9i. I have designed a form and run it but when I input unicode charater in TEXT ITEM on form (FONT_NAME of this TEXT ITEM is New Roman, AriaTime l ...), but it display incorrectly nor stored it in Database.
Thank you !

Thank Duncan R Mills !
My setting NLS_CHARACTER in Database as follow :
SQL> SELECT * FROM NLS_DATABASE_PARAMETERS;
PARAMETER VALUE
NLS_LANGUAGE AMERICAN
NLS_TERRITORY AMERICA
NLS_CURRENCY $
NLS_ISO_CURRENCY AMERICA
NLS_NUMERIC_CHARACTERS .,
NLS_CHARACTERSET UTF8
NLS_CALENDAR GREGORIAN
NLS_DATE_FORMAT DD-MON-RR
NLS_DATE_LANGUAGE AMERICAN
NLS_SORT BINARY
NLS_TIME_FORMAT HH.MI.SSXFF AM
PARAMETER VALUE
NLS_TIMESTAMP_FORMAT DD-MON-RR HH.MI.SSXFF AM
NLS_TIME_TZ_FORMAT HH.MI.SSXFF AM TZH:TZM
NLS_TIMESTAMP_TZ_FORMAT DD-MON-RR HH.MI.SSXFF AM TZH:TZM
NLS_DUAL_CURRENCY $
NLS_COMP BINARY
NLS_NCHAR_CHARACTERSET UTF8
NLS_RDBMS_VERSION 8.1.7.0.0
18 rows selected.
Even if I can'nt input unicode character on Oracle Forms, It display incorrectly though I set exactly font_name.

Changing Character Set from AL16UTF16 to AL32UTF8

Hi,
I am stuck with a requirement to change the Character Set from AL16UTF16 to AL32UTF8.
I am trying to install the Oracle Content Database & the installer expects the target database to have a Character Set of AL32UTF8. The current Character Set of the Database is AL16UTF16. I am unable to change the Character set as it simply complains that AL32UTF8 is not a super-set of AL16UTF16.The documents that I have consulted mention that such a transitionis not recommended.
How do I convince the installer to continue the installation with AL16UTF16 ? Or, is it possile at all to change the Character Set from AL16UTF16 to AL32UTF8 ?
Please do let me know your thoughts on this.
Regards,
Sandeep

Is your data size above GBs? If not this might help before installation I guess ->
Changing the Database Character Set of an Existing Database
http://download-uk.oracle.com/docs/cd/B19306_01/server.102/b14225/ch11charsetmig.htm#sthref1476
Kind regards,
Tonguç

Changing the Database Character Set From WE8MSWIN1252 to AR8MSWIN1256

good morning everybody,
I need your help to know all step which it is necessary to follow to change the Database Character Set From WE8MSWIN1252 to AR8MSWIN1256
thank you

If you have not already done so, read up on Character Set Migration and using the CSSCAN tool to verify the database before making any changes.

Unable to alter character set from WE8MSWIN1252 to AL32UTF8

Hi,
I am trying to migrate some tables to my database.
On the source DB, the character set is AL32UTF8
On My database, the character set is WE8MSWIN1252.
When I tried to alter the character set, I get this error message:
ALTER DATABASE CHARACTER SET AL32UTF8
ERROR at line 1:
ORA-12712: new character set must be a superset of old character set
Any help? Thanks.

Pl refrain from posting duplicate threads :-)
Unable to migrate table, character set from WE8MSWIN1252 to AL32UTF8
Srini

Parsing xml in clob field remove non utf-8 characters

hi all,
i have an issue where a stored procedure runs during a nightly process and parses xml contained in a clob field. There are some records that contain non-utf8(they paste characters from word)characters and therefore the parse fails when using performing an xquery select against the clob field as xmltype. I was wondering if anyone knew of a handy way to handle such a situation? I have looked around a bit and not found anything that seemed tailored to my situation, and I would like something a bit more generic than doing a replace on individual characters that i have found causing issues...thx in advance! -- jp

Hi,
Like BluShadow I'm curious to see a test case...
Depending on the way it's been created, the encoding declared in the XML prolog doesn't necessarily reflects the actual encoding of the content.
What's your database character set, and version?
Please also post the error message you get (LPX-00200 probably?).

UTF/Japanese character set and my application

Blankfellaws...
a simple query about the internationalization of an enterprise application..
I have a considerably large application running as 4 layers.. namely..
1) presentation layer - I have a servlet here
2) business layer - I have an EJB container here with EJBs
3) messaging layer - I have either Weblogic JMS here in which case it is an
application server or I will have MQSeries in which case it will be a
different machine all together
4) adapter layer - something like a connector layer with some specific or
rather customized modules which can talk to enterprise repositories
The Database has few messages in UTF format.. and they are Japanese
characters
My requirement : I need thos messages to be picked up from the database by
the business layer and passed on to the client screen which is a web browser
through the presentation layer.
What are the various points to be noted to get this done?
Where and all I need to set the character set and what should be the ideal
character set to be used to support maximum characters?
Are there anything specifically to be done in my application code regarding
this?
Are these just the matter of setting the character sets in the application
servers / web servers / web browsers?
Please enlighten me on these areas as am into something similar to this and
trying to figure out what's wrong in my current application. When the data
comes to the screen through my application, it looks corrupted. But the asme
message when read through a simple servlet, displays them without a problem.
Am confused!!
Thanks in advance
Manesh

Hello Manesh,
For the database I would recommend using UTF-8.
As for the character problems, could you elaborate which version of WebLogic
are you using and what is the nature of the problem.
If your problem is that of displaying the characters from the db and are
using JSP, you could try putting
<%@ page language="java" contentType="text/html; charset=UTF-8"%> on the
first line,
or if a servlet .... response.setContentType("text/html; charset=UTF-8");
Also to automatically select the correct charset by the browser, you will
have to include
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> in the
jsp.
You could replace the "UTF-8" with other charsets you are using.
I hope this helps...
David.
"m a n E s h" <[email protected]> wrote in message
news:[email protected]...
Blankfellaws...
a simple query about the internationalization of an enterpriseapplication..
>
I have a considerably large application running as 4 layers.. namely..
1) presentation layer - I have a servlet here
2) business layer - I have an EJB container here with EJBs
3) messaging layer - I have either Weblogic JMS here in which case it isan
application server or I will have MQSeries in which case it will be a
different machine all together
4) adapter layer - something like a connector layer with some specific or
rather customized modules which can talk to enterprise repositories
The Database has few messages in UTF format.. and they are Japanese
characters
My requirement : I need thos messages to be picked up from the database by
the business layer and passed on to the client screen which is a webbrowser
through the presentation layer.
What are the various points to be noted to get this done?
Where and all I need to set the character set and what should be the ideal
character set to be used to support maximum characters?
Are there anything specifically to be done in my application coderegarding
this?
Are these just the matter of setting the character sets in the application
servers / web servers / web browsers?
Please enlighten me on these areas as am into something similar to thisand
trying to figure out what's wrong in my current application. When the data
comes to the screen through my application, it looks corrupted. But theasme
message when read through a simple servlet, displays them without aproblem.
Am confused!!
Thanks in advance
Manesh

Changing database character set from US7ASCII to AL32UTF8

Our database is running on Oracle database 10.1.0.4.0 (AIX) The following are its parameters:
SQL> select value from NLS_DATABASE_PARAMETERS where parameter='NLS_CHARACTERSET';
VALUE
US7ASCII
We would like to change the database character set to AL32UTF8. After following Metalink notes: 260192.1 (which helped us resolve "Lossy" and "Truncated" data, the final output of the CSSCAN utility is:
[Scan Summary]
All character type data in the data dictionary are convertible to the new character set
All character type application data are convertible to the new character set
[Data Dictionary Conversion Summary]
The data dictionary can be safely migrated using the CSALTER script
We have no (0) Truncation and Lossy entries on the .txt file. We only have Changeless and Convertible. Now accdg to the documentation, we can do a FULL EXP and FULL IMP. But it did not detail how to do the conversion on the same database. The discussion on the document tells how to do it from one database to another database. But how about on the same database?
We cannot use CSALTER as stated on the document.
(Step 6
Step 12
12.c) When using Csalter/Alter database to go to AL32UTF8 and there was NO "Truncation" data, only "Convertible" and "Changeless" in the csscan done in point 4:)
After performing a FULL export of the database, how can we change its character set? What do we need to do the the existing database to change its character set to AL32UTF8 before we import back our dump file into the same database?
Please help.

There you are! Thanks! Seems like I am right in my understanding about the Oracle Official Documentation. Thanks!
Hmmmmm...when you say:
*"you can do selective export of only convertible tables, truncate the tables, use CSALTER, and re-import."*
This means that:
1. After running csscan on database PROD, i will take note of the convertible tables in the .txt output file.
2. Perform selective EXPORT on PROD (EXP the convertible tables)
3. Truncate the convertible tables on PROD database
4. Use CSALTER on PROD database
5. Re-import the tables into PROD database
6. Housekeeping.
Will you tell me if these steps are the correct one? Based on our scenario: This is what i have understood referring to the Official Doc.
Am i correct?
I really appreciate your help Sergiusz.

Reading in Latin Extended-A character set from a text file

Hello all,
I am writing a small program that reads in a text file containing special characters (beyond the ASCII char set) and converting it into "regular" characters. For example I would read in a uaccent and replace it with a u.
Now I realize that Unicode support is built into Java from ground up but it goes only so far, you actually have to have the relevant character set to read it. My code is as follows:
InputStreamReader inStreamReader = new InputStreamReader(new FileInputStream("input.txt"), "ISO-8859-1");
BufferedReader bufferedReader = new BufferedReader(inStreamReader);
String line = null;
StringBuffer buff = new StringBuffer();
while((line = bufferedReader.readLine()) != null) {
char[] charArray = line.toCharArray();
for(int i = 0; i < charArray.length; i++) {
int x = (int)charArray;
switch(x) {
case 224: // this is agrave .. we need to replace it with a
buff.append('a');
break;
case 230: // this is aelig .. we need to replace it with ae
buff.append("ae");
break;
///////// and so on
Since I am reading in as ISO-8859-1, this works up to unicode 255. For the rest of the characters, apparently I need a Latin Extended-A and Latin Extended-B character set. How can I get that installed on my Windows OS machine? I am using jdk 1.4.1 on Windows XP. Any help is appreciated.
Thanks,
-vk4t

vkat wrote:
Since I am reading in as ISO-8859-1, this works up to unicode 255. For the rest of the characters, apparently I need a Latin Extended-A and Latin Extended-B character set. How can I get that installed on my Windows OS machine? I am using jdk 1.4.1 on Windows XP. Any help is appreciated.If your file has characters outside of 8859-1's range (0 - 255), then it isn't ISO-8859-1 encoded. You need to know what encoding was used to store the file. It sounds like you it actually may be Unicode text, in which case you need to know which encoding (UTF8, UTF16, etc) was used.

How to change the Character Set from AL32UTF8 to WE8DEC

Hello!!
I want to know how to change the character set in the database from AL32UTF8 to WE8DEC.
I tried to use the comand ALTER DATABASE CHARACTER SET but I got an error because WE8DEC is not a superset of AL32UTF8.
I need to import tables from a server that uses WE8DEC. So when I do the import to my server, wich has AL32UTF8 , I can't import the rows that include an Ñ.
So I want to change the caracter set to WE8DEC. How can I do it?
Is it needed to change the language configuration? The remote server has AMERICAN_AMERICA, and my server has MEXICAN SPANISH_MEXICO (both uses text in spanish).
Thanks a lot!!

When you export from WE8DEC server what did you use
for NLS_LANG char set? And when importing?
The export was done in another computer because I can't do the export in the server (I have 10g, and the remote server has 9i and the export is not working). In my server, when I tried to do the import, the NS_LANG value was MEXICAN SPANISH_MEXICO.WE8MSWIN1252.
When I try to import to my database I got the error:
import done in WE8DEC character set and AL16UTF16 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
export client uses WE8MSWIN1252 character set (possible charset conversion)
. importing USRMCR06's objects into PRIMARIZACION
. . importing table "CHG_FONDOS_MARZO_CD_MOR"
IMP-00019: row rejected due to ORACLE error 12899
IMP-00003: ORACLE error 12899 encountered
ORA-12899: value too large for column "PRIMARIZACION"."CHG_FONDOS_MARZO_CD_MOR".
"NOMBRE" (actual: 41, maximum: 40)
Column 1 16623436
Column 2
Column 3 Pymes_1
Column 4

Removing non UTF-8 character set from xml in OSB

Similar Messages

Maybe you are looking for