File I/O and encoding (J2SDK 1.4.2 on Windows)

Question

I encountered a strange behavior using the FileReader / Writer classes for serializing the contents of a java string. What I did was basically this:
String string = "some text";
FileWriter out = new FileWriter(new File("C:/foo.txt"));
out.write(string);
out.flush();
out.close();In a different method, I read the contents of the file back:
FileReader in = new FileReader(new File("C:/foo.txt"));
StringWriter out = new StringWriter();
char[] buf = new char[128];
for (int len=in.read(buf); len>0; len=in.read(buf)) {
    out.write(buf, 0, buf.length);
out.flush(); out.close(); in.close();
return out.toString();Problems arise as soon as the string contains non ascii characters. After writing and reading, the value of the string differs from the original. It seems that different character encodings are used when reading and writing, although the doc states that, if no explicit encoding is specified, the platform's default encoding (in my case CP1252) will be used.
If I use streams directly instead of writers, it does not work, either, as long as I do not specify the encoding when converting bytes to strings and vice versa.
When I specify the encoding (no matter which one, as long as I specify the same for reading as for writing), the resulting string is equal to the original one.
If I replace the FileReader and Writer by StringReader and StringWriter (bypassing the serialization), it works, too (without specifying the encoding).
Is this a bug in the file i/o classes or did I miss something?
Thanks for your help
Ralph

Answer 1

first.... if you are writing String objects via serialization, encoding doesn't matter whatsoever. Not sure you were saying you tried that, but just for future reference.
For String.getBytes() and String(byte[]) or InputStreamReader and OutputStreamWriter: If you don't specify an encoding, the system default (or default specified on the command-line or set in some other way) will be used in all cases.
For byte streams: If you are reading/writing bytes thru streams, then the character conversion is up to you. You call getBytes on a string or create a string with the byte[] constructor.
For readers/writers: If you are reading/writing characters thru readers/writers, then the character conversion is done by that class.
However, StringReader and StringWriter are just writing to/from String objects and they are writing Unicode char's, so it's really a special case.
Okay...
So if you have a string which has characters outside the range of the encoding being used (default or explicitly specified), then when it's written to the file, those characters are messed up. So say you have a Chinese character which needs 2 bytes. Generally, the 2 bytes are written, but when read back, that one character shows as 2. Whether 2 bytes are written or 1, probably depends on the encoding. But the result is the same, you get a munged up string.
Generally speaking, you are going to get better storage on most text when using UTF-8 as your encoding. You need to specify it always for reads and writes, or set it as the default. The reason is that chars are written in as many bytes as needed. And it'll support anything Unicode supports, thus anything String supports.