Problem in Database convertion from US7ASCII to UTF8

Question

Hi,
We are facing the following problem while converting the database from US7ASCII to UTF8:
We have recently changed the database character set from US7ASCII to UTF8 for the internationalization
purpose. We ran the Character set scanner utility and it did report that some data may pose problems.
We followed the the below mentioned process to convert into UTF8 -
1) alter database character set utf8
2) alter database national character set utf8.
Now we find some problem while working with the old data in our application which is java based.
We are getting the following error "java.sql.SQLException: Fail to convert between UTF8 and UCS2: failUTF8Conv".
We further analyzed our data and found some interesting things :
e.g.
DB - UTF8.
NL_LANG also set to UTF8.
Select name from t1 where name like 'Gen%';
NAME
Genhve
But when we find out the length of the same data it show like this
NAME LENGTH(NAME) VSIZE(NAME)
Genhve 4 6
The question is why is it showing length as 4 only and when we try to use a substr function
its extracting like the following :-
select name,substr(name,4,1) from t1 where name like 'Gen%';
NAME SUB
Genhve hve
We have execute the above queries on US7ASCII DB and it is working fine, length it shows 6
and using SUBSTR it extracts just 'h' as well.
We also used dump function on the UTF8 Db for the above query,,this is the result :-
select name,length(name),vsize(name),dump(name) from t1 where name like 'Gen%';
NAME LENGTH(NAME) VSIZE(NAME) DUMP(NAME)
Genhve 4 6 Typ=1 Len=6: 71,101,110,232,118,101
We checked a lot with the data and it seems 'h' (accented e) is posing the problem.
We want to know where is the problem and how to overcome this.
Further, we tried all of the following :
1)
Export Server: US7ASCII
Export Client: did not set NLS_LANG / NLS_CHAR, so presumably US7ASCII as well
Import Client: did not set NLS_LANG / NLS_CHAR, so presumably US7ASCII as well
Import Server: UTF8
RESULT: Acute e became h
2)
Export Server: US7ASCII
Export Client: did not set NLS_LANG / NLS_CHAR, so presumably US7ASCII as well
Import Client: NLS_LANG=AMERICAN_AMERICA.UTF8 and NLS_CHAR=UTF8
Import Server: UTF8
RESULT: IMP 00016 error
3)
Export Server: US7ASCII
Export Client: NLS_LANG=AMERICAN_AMERICA.UTF8 and NLS_CHAR=UTF8
Import Client: did not set NLS_LANG / NLS_CHAR, so presumably US7ASCII as well
Import Server: UTF8
RESULT: Acute E became h
4)
Export Server: US7ASCII
Export Client: NLS_LANG=AMERICAN_AMERICA.UTF8 and NLS_CHAR=UTF8
Import Client: NLS_LANG=AMERICAN_AMERICA.UTF8 and NLS_CHAR=UTF8
Import Server: UTF8
RESULT: Acute e became h
5)
Tried using Update sys.props$
set value$='UTF8'
where name='NLS_CHARACTERSET'
RESULT: Acute e shows properly but it gives problem in the application
"java.sql.SQLException: Fail to convert between UTF8 and UCS2: failUTF8Conv"
Looking further it was observed the following:
when you try this command on a column 'city' in a table which contains 'Genhva' (note the acute e after n), it shows
command: select length(city), vsize(city),substr(city,4,1),city from cities
Result: 4 6 hva Genhva
if you see the value of substr(city,4,1) , you will see the problem. Also note that the length shows 4 and size shows 6. Moreover, when these records are selected through JDBC Driver, it starts giving problems.
6)
Actually the above (point no. 5) is similar to changing the character set of the database with 'ALTER DATABASE CHARACTER SET UTF8'. Same problem is observed then too.
7)
We have also tried to with another method, that is by changing the third byte of the export file which specifies the character set, to the UTF8 code by editing the export file with a Hexdecimal editor. After import the same problem has been observed as defined in (5) and (6) above.
We have no more ideas how to migrate without corrupting the data. Of course we have known the records where these characters occur through the Oracle's cssacn utility but we do not want to manually rectify each and every record by replacing it with an ASCII character. Any other idea as to how this can be accomplised?
Thanx
Ashok

Answer 1

The problem you have is that although your original database is defined as US7ASCII, the data it contains is more than is covered by this code page (as the reply on Sept 4 to the previous posting already said).
This has probably happened because the client was also defined as US7ASCII, and when the DB and client are defined as having the same character set no conversion (or checdking) takes place when data is passed between them. However if you are using a Windows client then it will in fact be using Windows code page 1252 (Latin-1) or similar, and this allows many more characters, including h (accented e). So a user can enter all these characters and store them in the database, and similarly read them from the database, because data transfer is transparent.
When you did ALTER DATABASE CHARACTER SET UTF8 this will only change the label on the database, but not affect the contents. However only part of the contents are valid UTF8, any character above 7F (like h) is invalid. If your original client now uses the database, code page transformation will take place because the client and DB have different character sets defined. The invalid codes can then cause problems.
Without being able to explain what has happened in detail, it may help to see what your h (dec 232, x'E8') looks like. The actual data has not changed (you can see this as it is reported as 232). However the binary code there (11101000) is invalid UTF8. UTF8 encodes a character in 1 to 4 bytes, and the first bits in a UTF8 character tell how many bytes it uses. 0xxx tell it is one byte (same as the corresponding USASCII character), 110x that it uses 2 bytes, 1110 that it uses 3 bytes etc. So if you interpret what is there as UTF8 it looks like the first byte of a 3-byte character, which explains why the substringing is giving you the other 2 bytes as well.
Can you fix this without losing data? I believe yes. First you should check what other characters are being flagged by the scan. See if these are contained in another standard character set. If they are only Western European accentet characters then WE8ISO8859P1 is probably ok, but watch out for the euro sign which Windows has at x'80', an undefined character in ISO8859-1.
You can see the contents of the Microsoft Windows Codepage 1252 at: http://www.microsoft.com/globaldev/reference/sbcs/1252.htm
For a listing of the US-ASCII defined characters see http://czyborra.com/charsets/iso646.html and for ISO 8859-1 see http://czyborra.com/charsets/iso8859.html#ISO-8859-1
If all is well, you can first ALTER DATABASE CHARACTER SET to WE8ISO8859P1. This will not change any data, but ensure that all the data gets exported. Then export the DB and import it to a UTF8 DB. This will convert the non-US-ASCII characters to Unicode. You will also have to change the clients character set to something other than USASCII or they will just see ? for the other characters.
Good Luck!