Question about character encoding

Question

Hey everyone,
I've been trying to read about character encodings and how Java IO uses them, however, there is something I still don't understand:
I am trying to read an HTML file encoded in UTF-8 and which contain some Arabic characters.
I used this code to save it on a file:
URL u = new URL("http://www.fatafeat.com");    // some Arabic website
URLConnection connection = u.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream(), "UTF-8"));
OutputStreamWriter writer = new OutputStreamWriter(new FileOutputStream("datat.txt"));
int c;
while ((c = reader.read())!=-1)
writer.write(c);However, the output on the file is always "??????"
I used another version of the code
URL u = new URL("http://www.fatafeat.com");     //some Arabic website
URLConnection connection = u.openConnection();
InputStream reader = connection.getInputStream();
FileOutputStream writer = new FileOutputStream("datat.txt");
int c;
while ((c = reader.read()) != -1)
writer.write(c);And I get on the file something like this " ÇáæÍíÏÉ áÝä "
I tried to open the file from a browser using a UTF-8 encoding, but still the same display. What am I missing here? Thanks

Answer 1

Upon further investigation, my initial hunch appears to be correct. The page in question is not correctly encoded UTF-8, so the Java decoder fails. By default, the java decoder fails silently and replaces malformed input with the character "\uFFFD". If you want more control over the decoding process, you need to read the data as bytes and use the CharsetDecoder class to convert to characters. Here is small example to illustrate:
import java.io.ByteArrayOutputStream;
import java.io.CharArrayWriter;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CoderResult;
import java.nio.charset.CodingErrorAction;
public class FatafeatToy3
    private static final Charset UTF8 = Charset.forName("UTF-8");
    private static final Charset DEFAULT_DECODER_CHARSET = UTF8;
    private final CharsetDecoder decoder;
    public FatafeatToy3(final Charset cs)
        this.decoder = cs.newDecoder();
        this.decoder.onMalformedInput(CodingErrorAction.REPORT);
        this.decoder.onUnmappableCharacter(CodingErrorAction.REPORT);
    public FatafeatToy3()
        this(DEFAULT_DECODER_CHARSET);
    public ByteBuffer pageSlurp (URL url) throws IOException
        ByteArrayOutputStream pageBytes = new ByteArrayOutputStream ();
        InputStream is = url.openStream();
        int ch;
        while ((ch = is.read()) != -1)
            pageBytes.write(ch);
        is.close();
        return ByteBuffer.wrap(pageBytes.toByteArray());
    public CoderResult decodeSome(ByteBuffer in, CharBuffer out)
        decoder.reset();
        CoderResult result = decoder.decode(in, out, true);
        if (result.isMalformed())
            System.err.printf("Malformed input detected at pos 0x%x%n", in.position());
        else if (result.isUnmappable())
            System.err.printf("Unmappable input detected at pos 0x%x%n", in.position());
        else if (result.isUnderflow())
            result = decoder.flush(out);
        return result;
    public static void main(String[] args) throws Exception
        FatafeatToy3 ft = new FatafeatToy3();
        ByteBuffer in = ft.pageSlurp(new URL("http://www.fatafeat.com"));
        System.out.printf("Page slurped contains %d bytes%n", in.capacity());
        CharBuffer out = CharBuffer.allocate(1); // one character at a time
        CharArrayWriter pageChars = new CharArrayWriter();
        CoderResult result = CoderResult.UNDERFLOW;
        while ( (! result.isError()) && in.remaining() > 0)
            result = ft.decodeSome(in, out);
            if (result.equals(CoderResult.OVERFLOW))
                out.flip();
                pageChars.append(out);
                out.clear();
}