Strugging with exporting data out from an Unicode database

Question

Background information
Server: Sun Solaris 5.10; 10g
Client: Windows 2000; 10g, TOAD, Oracle ODBC 10.2.0.1
select * from v$NLS_PARAMETERS
NLS_LANGUAGE AMERICAN
NLS_TERRITORY AMERICA
NLS_CURRENCY $
NLS_ISO_CURRENCY AMERICA
NLS_NUMERIC_CHARACTERS .,
NLS_CALENDAR GREGORIAN
NLS_DATE_FORMAT DD-MON-RR
NLS_DATE_LANGUAGE AMERICAN
NLS_CHARACTERSET AL32UTF8
NLS_SORT BINARY
NLS_TIME_FORMAT HH.MI.SSXFF AM
NLS_TIMESTAMP_FORMAT DD-MON-RR HH.MI.SSXFF AM
NLS_TIME_TZ_FORMAT HH.MI.SSXFF AM TZR
NLS_TIMESTAMP_TZ_FORMAT DD-MON-RR HH.MI.SSXFF AM TZR
NLS_DUAL_CURRENCY $
NLS_NCHAR_CHARACTERSET AL16UTF16
NLS_COMP BINARY
NLS_LENGTH_SEMANTICS BYTE
NLS_NCHAR_CONV_EXCP FALSE
We import SAS data (Windows Latin character set) into Oracle, use OWB for ETL, export the results to SAS. Per regulatory requirements, character columns cannot exceed 200 in length.
Problem scenario
Data that cause the trouble (200 characters, with a degree sign at the 77th position):
XXX PT PRIOR TO MSFC NOT TO USE WALKER, PT STATED SHE NEEDED IT LAST VISIT 2° BAD HEADACHE DECREASED BALANCE, WHICH WAS LATER FOUND TO BE SINUS INFECTION. ASKED PT NOT TO USE WALKER THIS TIME, PT SAID
Degree sign is U+00B0 in UTF-8, or 0xB0 (176) in ASCII. Though, I found out select ascii('°') from dual would return 49480 (or, 0xC2 0xB0).
In order to accommodate the import, Source.COMMENTX is VARCHAR2(201). Using OWB, we are mapping this to Target.COVAL which is VARCHAR2(200).
To get around ORA-12899: value too large for column, we use the expression convert(Source.COMMENTX, 'WE8ISO8859P1', 'AL32UTF8')).
Although viewing Target.COVAL shows a ¿ (true in TOAD, SQL*Plus), dump(COVAL) confirms the 77th character is 176:
DUMP(COVAL)
Typ=1 Len=200: [...],32,50,176,32,[...]
Desirable outcome
Store and display the text in a VARCHAR2(200) column without compromising the high-bit ASCII characters, e.g., degree sign, micro sign (i.e., Greek character mu), copyright sign, etc.
Questions
1. Is it a wrong assumption that AL32UTF8 supports the high-bit ASCII characters (i.e., characters between 128 and 255)? If not, why do the clients display the inverted question mark instead of degree sign when executing select chr(176) from dual?
2. The aforementioned DUMP statement seems to confirm ASCII 0xB0 (i.e., not 0xC2 0xB0, or 0xBF) is being stored in the database at the 77th position. Why do my applications via ODBC interpreted and replaced it as 0xBF, which is the inverted question mark?
Avenues attempted without the desirable outcome
1. Changing Target.COVAL from VARCHAR2(200) to NVARCHAR2(200) or VARCHAR2(200 CHAR) would make SAS (data access through ODBC) think the length is 400 or 800, respectively [Note: The vendor claims it is ODBC 3.0 compliant]
2. Through Microsoft's ODBC Test software, this is the output for describe column all against select COLVAL from Target:
icol, szColName, pcbColName, pfSqlType, pcbColDef, pibScale, *pfNullable
1, COMMENTX, 8, SQL_WVARCHAR=-9, 200, 0, SQL_NULLABLE=1

Answer 1

Degree sign is U+00B0 in UTF-8, or 0xB0 (176) in
ASCII. Though, I found out select ascii('°') from
dual would return 49480 (or, 0xC2 0xB0).Well, U+00B0 represents 'degree sign' in Unicode, and the UTF-8 encoded value for this code point is C2 B0. ASCII does not include a degree sign, and 176 is not a ASCII code value (only 0-127). The function ascii will just return the decimal form of the encoded value, in the character set of the database (not necessarily ASCII, or US7ASCII as it is called in Oracle).
>
To get around ORA-12899: value too large for column,
we use the expression convert(Source.COMMENTX,
'WE8ISO8859P1', 'AL32UTF8')).This part I don't understand. And where are you storing this? In the same AL32UTF8 database? I think this might be your problem.
>
Although viewing Target.COVAL shows a ¿ (true in
TOAD, SQL*Plus), dump(COVAL) confirms the 77th
character is 176:Yes, since 176 is an invalid value in UTF-8. U+0079 is encoded as 79, U+0080 is encoded as C2 80 - notice the "leap" there. If I would input 176 in a "utf-8 decode" I would get "out of range" or NaN back. Similarily, if you have managed to illegally store 176 as a character encoded value in a AL32UTF8 database, and are trying to retrieve that, involving a conversion to client character set, you would get the replacement character ¿ meaning "bad conversion".
>
DUMP(COVAL)
Typ=1 Len=200: [...],32,50,176,32,[...]
Try
select chr(49480) from dual;
- but you need to do this from a tool such as Oracle SQL Developer (it's free) that can handle Unicode ouput.