Arabic Characters Issue

Question

Hi
Following is our environment
DB Version: Oracle 11g R2 (11.2.0.3.0) 64bit
OS : OEL 5U2
Characterset: AMERICAN_AMERICA.AR8MSWIN1256
We have an opensource application built on php. CHARSET was defined as AR8MSWIN1256 in the configuration file. Data was inserted using browser based interface. Use access the system using IE 8 since it is web based application. Data comprise of both Arabic and English. It is displayed perfectly fine using browser ie Arabic shows proper Arabic characters and English shows correct English.
Problem started when a requirement came to generate some reports using Oracle Reports 10g. Arabic data show as garbage in the reports. Vendor was asked to provide some solution. They suggested to change the CHARSET to AL32UTF8 in the config file of the application. After making the changes, the new data that is inserted in the database can be seen correctly for both languages regardless of the tool (be it SQLDeveloper or Oracle Reports). The old data, however, shows garbage. Surprisingly when data is extracted using scripts in flat files and then imported in the test database after setting environment variable NLS_LANG=AMERICAN_AMERICA.UTF8 using @/../filename.sql, it converts it properly. Since data is huge and this is taking way too much time, I was hoping if someone could help me in quickly perform this migration? Already tried CSSCAN/CSALTER and DMU but did not succeed. DMU stopped at 5th stage when it was supposed to issue ALTER DATABASE CHARACTER SET command.
Currently I am installing database with AL32UTF8 characterset and planning to import the schema in it. Is my approach correct? Any help will be highly appreciated.
Thank you
myuser@DB>select dump(name,1016) from  mytable where id like '1';
DUMP(NAME,1016)
Typ=1 Len=28 CharacterSet=AR8MSWIN1256: d9,81,d8,b1,d9,8a,d9,82,20,d8,b9,d9,85,d9,84,20,d8,a7,d9,84,d9,82,d8,b7,d8,a7,d8,b9
myuser@DB>select dump(name,1016) from  mytable where id like '2';
DUMP(NAME,1016)
Typ=1 Len=93 CharacterSet=AR8MSWIN1256: d8,aa,d8,b9,d8,a7,d9,85,d9,8a,d9,85,20,d9,85,d9,86,20,d8,a7,d9,84,d9,85,d8,af,d9,8a,d8,b1,20,d8,a7,d9,84,d8,aa,d9,86,d9,81,d9,8a,d8,b0,d9,8a,20,d9,84,d9,84,d9,82
myuser@DB>select value from NLS_DATABASE_PARAMETERS where parameter LIKE '%CHARACTERSET';
VALUE
AR8MSWIN1256
UTF8Edited by: user560883 on May 4, 2012 3:34 AM
Edited by: user560883 on May 4, 2012 3:39 AM

Answer 1

Hi Sergiusz
Firstly I would like to thank you for replying.
Secondly I was able to move further with the DMU (the issue was due to more sessions that I had to kill to allow DMU to proceed with ALTER DATABASE CHARACTER SET command).
Here is what I have done so far without any luck.
1) Created a test instance with source database characterset ie AR8MSWIN1256 and performed a full import using DataPump (impdp).
2) Used DMU to perform conversion/migration to AL32UTF8. (this step took almost 5 hours).
3) Used SQLDeveloper to verify the results and data still show as garbage.
Thing that worked for me but is way to slow and not reliable.
1) Save data in file (.sql) using Windows XP client or windows 2003 server. Basically I am generating the whole SQL statements for UPDATE eg UPDATE table SET xx)
2) SET NLS_LANG=AMERICAN_AMERICA.UTF8 on command prompt of Windows XP client and use SQLPLUS to open sql prompt.
3) Use @path\file.sql (that was generated in earlier step) to update records.
4) Verify via SQLDeveloper that arabic shows correctly.
Issue with above approach is since we have millions of records stored in multiple tables, sometimes while generating the file with SQL Statements, the server CPU touch 100% and then freeze. Also file size grow exponentially and to verify how much of the work was done prior to freezing becomes very difficult.
I still have the DB after using the DMU tool intact so if you could think of some other workaround that will be great.
Again, thank you for helping me with this.