How to recognize invalid UTF8-codepoints ?

Question

Not all possible byte-sequence are valid UT8-characters.
( See http://en.wikipedia.org/wiki/Utf-8 )
For instance the byte-sequnence 195 and 34 (decimal) is invalid.
Oracle seems to have no problem with it. It stores the sequence in the database
and if it should be displayed in a client like sql-developer a hollow quad is shown.
But some application has problems with such invalid codepoints.
Because of performance reasons I don’t want to extract every codepoint and analyze every byte in it in PL/SQL.
Does anyone knows a better solution (for Oracle 9.2) to recognize or remove such invalid codepoints ?

Answer 1

Michael
We had all kinds of trouble like this with inconsistent character sets on Windows, Linux and Oracle. Any time you export, import, ftp a SQL script, etc, you can corrupt the data; what's a valid code point in one charset may not be in another.
The hollow quad you see (depends on your charset; I have seen upside down '?') tells you that there's no mapping from the bytes in the database to the display on your client - assuming a character set conversion is happening. I think you should be able to make use of explicit character set conversion to identify your problem text:
- convert from a narrower set (eg UTF-8) to a wider set (UTF-16) using CONVERT()
- then you should get this behaviour (see http://download-uk.oracle.com/docs/cd/B19306_01/server.102/b14225/ch7progrunicode.htm#sthref814)
"The NLS_NCHAR_CONV_EXCP initialization parameter controls the behavior of data loss during character type conversion. When this parameter is set to TRUE, any SQL statements that result in data loss return an ORA-12713 error and the corresponding operation is stopped." ... "In PL/SQL, when data loss occurs during conversion of SQL CHAR and NCHAR datatypes, the LOSSY_CHARSET_CONVERSION exception is raised for both implicit and explicit conversion"
Handle the exception and you can log where you have problems.
For my money, I would strongly recommend using a single byte character set unless you absolutely must support non-Latin characters. We didn't think the whole thing through, and use of multi-byte characters broke lost of code.
Have fun!
Regards Nigel