URLDecoder / URLEncoder don't like non-ASCII, even with UTF-8 explicit

Question

Here's my code:
import java.net.*;
import java.io.*;
public class Decoder
    public static void main(String[] args)
     if (args.length != 1) {
         System.err.println("Usage: Decoder UTF-8-URL-encoded-string");
         System.exit(1);
     String sampleString = "auflerhalb";
     try {
         for (int k = 0; k < args[0].length(); k++) {
          System.out.println("Char is " + Character.codePointAt(args[0], k));
         System.out.println();
         String queryText = URLDecoder.decode(args[0], "UTF-8");
         for (int k = 0; k < queryText.length(); k++) {
          System.out.println("Char is " + Character.codePointAt(queryText, k));
         String newText = URLEncoder.encode(queryText, "UTF-8");
         System.out.println(newText);
         System.out.println();
         for (int k = 0; k < sampleString.length(); k++) {
              System.out.println("Char is " + Character.codePointAt(sampleString, k));
         System.out.println(URLEncoder.encode(sampleString, "UTF-8"));
     catch ( UnsupportedEncodingException e ) {
         System.err.println("Couldn't decode querytext as UTF-8");
     System.exit(0);
}The problem I'm encountering is that characters such as the German S-like character aren't being decoded correctly, or encoded correctly. When I run this file, on Linux, MacOS X or Solaris 10, I get this:
% java -version
java version "1.5.0_07"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_07-b03, mixed mode)
% java -cp . Decoder "au%DFerhalb"
Char is 97
Char is 117
Char is 37
Char is 68
Char is 70
Char is 101
Char is 114
Char is 104
Char is 97
Char is 108
Char is 98
Char is 97
Char is 117
Char is 65533
Char is 101
Char is 114
Char is 104
Char is 97
Char is 108
Char is 98
au%EF%BF%BDerhalb
Char is 97
Char is 117
Char is 223
Char is 101
Char is 114
Char is 104
Char is 97
Char is 108
Char is 98
au%C3%9Ferhalb
The decimal code point of the German S is 223; the hex code point is DF. Yet the decoder completely mangles the string containing %DF, and the internal string, which clearly has the right decimal code point, is mangled by the encoder; C3 9F has absolutely nothing to do with 223.
I can't figure out for the life of me what I'm doing wrong. I've taken a look at the source code for URLDecoder.java, and it really appears that it's broken; it decodes hex sequences into a byte array, but Java bytes are signed, between -128 and 128. Could this be what's going on? Hex code points up to 127 work fine; above it don't work at all. Try my function with %7E instead of %DF, and then with %81.
Can this really be broken?
Thanks in advance -
Sam Bayer
The MITRE Corporation
[email protected]

Answer 1

The decimal code point of the German S is 223; the hex code point is DF.
Yet the decoder completely mangles the string containing %DF, and the
internal string, which clearly has the right decimal code point, is mangled
by the encoder; C3 9F has absolutely nothing to do with 223.I don't think you understand what's going on. I haven't checked the Unicode table but let's assume that hex-DF is the Unicode code point for that character. That means it's in the range which UTF-8 will encode in two bytes. I haven't run it through the algorithm but it's quite likely that the result would be C3 9F, since that's what happened when you did it. It would certainly be two bytes and not one, contrary to what you expect.