Multibyte character help needed!

Question

I posted earlier but got no response. I would like to rephrase, so hopefully someone out there can help me. I have researched this for two days, and have not been able to find an answer.
When I try to store a multibyte string in a String class, some characters are getting stored as '?' instead of the actual character. This seems to be a default storage for unknown characters. The problem I have is that this only occurs in my Java code. It works well in C, and will even work fine when I copy it to this forum message. Here is a sample:
String original = "ƒg�[ƒJƒrƒ‹ 2F Š”Ž�‰�Ž� IACEƒgƒ‰ƒxƒ‹ IACEƒgƒ‰ƒxƒ‹";
Notice the third character is a '�'.
When I try to display the contents of the variable, I get:
ƒg?[ƒJƒrƒ‹ 2F Š”Ž�‰�Ž� IACEƒgƒ‰ƒxƒ‹ IACEƒgƒ‰ƒxƒ‹
Notice the third character is now '?'.
When I display the array of bytes as hex values, I get 0x3f (?), when it should be 0x81.
Why does this get changed? How can I prevent it?

Answer 1

>
What happens if you wrap System.out in an
OutputStreamWriter created with different
encodings?BufferedWriter bufOut = new
BufferedWriter(new OutputStreamWriter(System.out,
"UTF-8"))What happens to your data if you write
to a file?Tried your wrapped System.out suggestion. Also tried to read the string from file into the String class and write it back to file. In both cases, I get the same results. Another thing is that in my test I print out all of the bytes, and the '?' is stored as 0x3f which is hex for '?' (see below). The question marks in the utf-8 (except the one I am struggling with) are caused by the HTML conversion. They are stored properly in the String class.
It seems to boil down to the fact that the String class is converting the byte for some reason. But why?
Here is the complete output of my test:
Cp1252
original = ƒg?[ƒJƒrƒ‹ 2F Š”Ž�‰�Ž� IACEƒgƒ‰ƒxƒ‹ IACEƒgƒ‰ƒxƒ‹
roundTrip = ƒg?[ƒJƒrƒ‹ 2F Š”Ž�‰�Ž� IACEƒgƒ‰ƒxƒ‹ IACEƒgƒ‰ƒxƒ‹
utf8Bytes[0] = 0xc6
utf8Bytes[1] = 0x92
utf8Bytes[2] = 0x67
utf8Bytes[3] = 0x3f
utf8Bytes[4] = 0x5b
utf8Bytes[5] = 0xc6
utf8Bytes[6] = 0x92
utf8Bytes[7] = 0x4a
utf8Bytes[8] = 0xc6
utf8Bytes[9] = 0x92
utf8Bytes[10] = 0x72
utf8Bytes[11] = 0xc6
utf8Bytes[12] = 0x92
utf8Bytes[13] = 0xe2
utf8Bytes[14] = 0x80
utf8Bytes[15] = 0xb9
utf8Bytes[16] = 0x20
utf8Bytes[17] = 0x32
utf8Bytes[18] = 0x46
utf8Bytes[19] = 0x20
utf8Bytes[20] = 0xc5
utf8Bytes[21] = 0xa0
utf8Bytes[22] = 0xe2
utf8Bytes[23] = 0x80
utf8Bytes[24] = 0x9d
utf8Bytes[25] = 0xc5
utf8Bytes[26] = 0xbd
utf8Bytes[27] = 0xc2
utf8Bytes[28] = 0xae
utf8Bytes[29] = 0xe2
utf8Bytes[30] = 0x80
utf8Bytes[31] = 0xb0
utf8Bytes[32] = 0xc3
utf8Bytes[33] = 0xaf
utf8Bytes[34] = 0xc5
utf8Bytes[35] = 0xbd
utf8Bytes[36] = 0xc3
utf8Bytes[37] = 0x90
utf8Bytes[38] = 0x20
utf8Bytes[39] = 0x49
utf8Bytes[40] = 0x41
utf8Bytes[41] = 0x43
utf8Bytes[42] = 0x45
utf8Bytes[43] = 0xc6
utf8Bytes[44] = 0x92
utf8Bytes[45] = 0x67
utf8Bytes[46] = 0xc6
utf8Bytes[47] = 0x92
utf8Bytes[48] = 0xe2
utf8Bytes[49] = 0x80
utf8Bytes[50] = 0xb0
utf8Bytes[51] = 0xc6
utf8Bytes[52] = 0x92
utf8Bytes[53] = 0x78
utf8Bytes[54] = 0xc6
utf8Bytes[55] = 0x92
utf8Bytes[56] = 0xe2
utf8Bytes[57] = 0x80
utf8Bytes[58] = 0xb9
utf8Bytes[59] = 0x20
utf8Bytes[60] = 0x49
utf8Bytes[61] = 0x41
utf8Bytes[62] = 0x43
utf8Bytes[63] = 0x45
utf8Bytes[64] = 0xc6
utf8Bytes[65] = 0x92
utf8Bytes[66] = 0x67
utf8Bytes[67] = 0xc6
utf8Bytes[68] = 0x92
utf8Bytes[69] = 0xe2
utf8Bytes[70] = 0x80
utf8Bytes[71] = 0xb0
utf8Bytes[72] = 0xc6
utf8Bytes[73] = 0x92
utf8Bytes[74] = 0x78
utf8Bytes[75] = 0xc6
utf8Bytes[76] = 0x92
utf8Bytes[77] = 0xe2
utf8Bytes[78] = 0x80
utf8Bytes[79] = 0xb9
utf8Bytes: �’g?[�’J�’r�’�€� 2F ���€?�����€������? IACE�’g�’�€��’x�’�€� IACE�’g�’�€��’x�’�€�
defaultBytes[0] = 0x83
defaultBytes[1] = 0x67
defaultBytes[2] = 0x3f
defaultBytes[3] = 0x5b
defaultBytes[4] = 0x83
defaultBytes[5] = 0x4a
defaultBytes[6] = 0x83
defaultBytes[7] = 0x72
defaultBytes[8] = 0x83
defaultBytes[9] = 0x8b
defaultBytes[10] = 0x20
defaultBytes[11] = 0x32
defaultBytes[12] = 0x46
defaultBytes[13] = 0x20
defaultBytes[14] = 0x8a
defaultBytes[15] = 0x94
defaultBytes[16] = 0x8e
defaultBytes[17] = 0xae
defaultBytes[18] = 0x89
defaultBytes[19] = 0xef
defaultBytes[20] = 0x8e
defaultBytes[21] = 0xd0
defaultBytes[22] = 0x20
defaultBytes[23] = 0x49
defaultBytes[24] = 0x41
defaultBytes[25] = 0x43
defaultBytes[26] = 0x45
defaultBytes[27] = 0x83
defaultBytes[28] = 0x67
defaultBytes[29] = 0x83
defaultBytes[30] = 0x89
defaultBytes[31] = 0x83
defaultBytes[32] = 0x78
defaultBytes[33] = 0x83
defaultBytes[34] = 0x8b
defaultBytes[35] = 0x20
defaultBytes[36] = 0x49
defaultBytes[37] = 0x41
defaultBytes[38] = 0x43
defaultBytes[39] = 0x45
defaultBytes[40] = 0x83
defaultBytes[41] = 0x67
defaultBytes[42] = 0x83
defaultBytes[43] = 0x89
defaultBytes[44] = 0x83
defaultBytes[45] = 0x78
defaultBytes[46] = 0x83
defaultBytes[47] = 0x8b
� � Ori
orig2: �’g?[�’J�’r�’�€� 2F ���€?�����€������? IACE�’g�’�€��’x�’�€� IACE�’g�’�€��’x�’�€�
Debuggee process terminated.
Process exited with exit code 0.