Please Help !!!!! -- Reagrding csscan report - 'lossy conversion' data

Question

Hello -
1. We have one database (db1) in characterset US7ASCII and does store some special characters like microsoft word's curly quotes etc.(eg “abc”) and renders and displays them correctly from Web application in the browser.
2. We have another database(db2) in characterset WE8ISO8859P1 character set behaves the same as above.
But the issues faced are as follows -
1. When we try to export the data from db1 (US7ASCII) and imported in db2 (WE8ISO8859P1). It could not convert microsoft word's curly quotes etc.. could not convert correctly and replacing those with upside-down questionmarks.We tried different NLS_LANG options while export/import. But nothing seems to be working.
Question - Not able to understand why the problem for the data conversion when both characterset databases work fine with those special characters, stores them fine.Besides WE8ISO8859P1 is strict superset of US7ASCII.
2. We decided to alter database character set of db1 (US7ASCII) to WE8ISO8859P1 , as we need to store french characters in the future.
WE8ISO8859P1 is strict superset of US7ASCII and both are single bytes, storage 8 bits for 8859P1 and 7 bits for ASCII7.
We ran csscan and found weird report with exception,telling that some of the data would have 'lossy conversion'. To me it seems that data could be those special chars like 'microsoft word's curly quotes etc..'
Questions -
1. Why the 'lossy conversion' ? when target is the superset of current one ?
2. when it says 'lossy conversion' what does it mean ? will the data will be corrupted for those chars (like upside question marks)?
3. How to fix the 'lossy conversion' data before altering the database character set ?
I am very confused with all of the above...Really need help on how to proceed on this.
Any help would be very much appreciated....
Thanks
Rama

Answer 1

You appear to have invalid data in the database. Understanding how you can have invalid data in the database, and yet be able to have the applications appear to work, though, is a bit complicated.
When data is being exchanged between the database and the client, there are two character sets to be concerned with-- the database character set and the client character set. If these two character sets are different, Oracle will automatically convert between the two character sets. If the two character sets are identical, however, the data gets passed between client and server is not validated at all-- the server assumes that the client is passing valid data.
The problem comes, however, when the client passes in data which is not valid. If I set the NLS_LANG on my client to specify that I am sending ASCII data, and the database uses the US7ASCII character set, I can send and receive Chinese data without a problem, so long as every application knows the real character set, treats the data appropriately internally, and lies about the true character set of the data when communicating with the database. Obviously, though, you cannot validly store Chinese data-- you've basically turned the VARCHAR2 column into a binary field-- the problem is that the data will be corrupted if the application doesn't know how to decode it properly.
In your case, the US7ASCII character set only encodes the first 128 characters (0-127), which does not include any of the special Microsoft characters. If you were to try to transfer those characters to an ISO 8859-1 database, the character set conversion that would be necessary would fail, so the data would be corrupted.
Fixing the data corruption is relatively simple, provided you've understood the explanation above. You can, for example, write a simple OCI application that selects the columns with the Windows characters and writes them to a file. The client NLS settings would be US7ASCII but the file would, presumably, use the Windows-1252 encoding. Then, you would alter the database character set, corrupting the Windows characters. You could then load the file you created before altering the database character set, assuming you changed the client NLS settings to correctly identify the encoding of the data.
Obviously, you will want to test this extensively before starting on your production database. You'll also probably require downtime to reload the data.
Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC