Convertin Cp1252 right curly quotes into UTF8

Question

I was not sure where to post this.
I was having some problems trying to convert CP1252 right curly quote into UTF8. Other Cp1252 characters were converting correctly.
The right curly quote was read from the database as 3 bytes. The first 2 bytes were mapping to legitimate UTF8 tokens but the third byte was not.
According to this chart:
http://www.io.com/~jdawson/cp1252.html
The right curly quote is “e2809d”.
Byte1 Byte2 Byte3
e2 80 9d (hexadecimal)
/342 /200 /235 (octal)
The octals are in a big private static final Java string in sun.nio.cs.MS1252 except “/235” is missing. So the process gets the first 2 bytes right but chokes on the third.
Hacking my own encoder class with the “/235” in the right place to made it work.
I was wondering if this was a bug in sun.nio.cs.MS1252 ?
Thx!
Edited by: langal on Feb 2, 2010 10:37 PM

Answer 1

Ah, the old double-encoding ploy! It looks like the text was encoded as UTF-8, then the resulting byte array was decoded as windows-1252, then the result was encoded again as UTF-8. // original string:
”
// encoded as UTF-8:
E2 80 9D
// decoded as windows-1252 (the third character
// is undisplayable, but valid):
”
// encoded as UTF-8:
C3 A2 E2 82 AC C2 9D The conversion to windows-1252 followed Microsoft's practice instead of its specification, so the "9D" control character is stored safely in its UTF-8 encoding, "C2 9D". Unfortunately, Java follows the spec, so getting the character back out is not so easy (and that maybe-bug is relevant--my apologies). It treats '\u009D' as an invalid character and converts it to 0x3F, the encoding for the question mark.
I did some checking, and it seems U+201D (Right Double Quotation Mark: ”) is the only one of the added characters in the "80..9F" range that causes this problem. If the data is otherwise intact, you can work around this problem by looking for the relevant byte sequence in the intermediate processing stage and replacing it: byte[] bb0 = { (byte)0xC3, (byte)0xA2, (byte)0xE2, (byte)0x82, (byte)0xAC, (byte)0xC2, (byte)0x9D };
// This is the string 'x' from your sample code.
String x = new String(bb0, "UTF-8");
System.out.println(x);  // ”
byte[] bb1 = x.getBytes("windows-1252"); // E2 80 3F
for (int i = 0; i < bb1.length - 2; i++)
  if ((bb1[i+2] & 0xFF) == 0x3F &&
      (bb1[i+1] & 0xFF) == 0x80 &&
      (bb1[i]   & 0xFF) == 0xE2)
    bb1[i+2] = (byte)0x9D;
String s = new String(bb1, "UTF-8");
System.out.println(s);  // &#x201D; The byte sequence "E2 80 3F" is unlikely to occur naturally in windows-1252, and it's invalid in UTF-8. Of course, this would only be a temporary measure, while you clean up your database as @jtahlborn said.