Escaped data to UTF-8, i.e. 「

Question

Hi everyone!
I have a form that needs to be in ISO-8859-1 format, because the form data is inserted into a database that expects ISO-8859-1.
At one point, I need to get the data out of the database and insert it into another database that accepts only UTF-8 data.
When I insert asian characters into the first database, the Web server automatically converts the asian characters into escaped characters, i.e. 「 (but the & # version - the forum is converting the symbol here :-(). That's the way it's inserted into the database. That is okay. I want it that way.
When I get out data from that database, I get the escaped characters. Now, I need to convert the data to UTF-8 so I can put it into the other database.
Is there a way to convert the escaped data into UTF-8 data?
dailysun
Message was edited by:
dailysun

Answer 1

First off, you don't need to convert the data to UTF-8. When you insert it into the other database, you should be able to specify "UTF-8" as the encoding, and the conversion will be done for you.
So your real problem is merely to convert the HTML escapes back to the characters they represent. That should be a simple matter of extracting the numerical portion of the escape, parsing it with Integer.parseInt(), and casting the resulting int to a char. The regex package makes the process relatively painless:  private static Pattern numRefPattern =
      Pattern.compile("&#(?:(\\d++)|[Xx](\\p{XDigit}++));");
  public static String unescapeHTML(String source)
    Matcher m = numRefPattern.matcher(source);
    StringBuffer sb = new StringBuffer(source.length());
    while (m.find())
      // append any text between the last match and the current one
      m.appendReplacement(sb, "");
      // find out whether it was a decimal or hex escape and parse accordingly
      int value = m.start(1) != -1
                ? Integer.parseInt(m.group(1));
                : Integer.parseInt(m.group(2), 16);
      if (!Character.isDefined(value))
        throw new IllegalArgumentException("No such character: " + m.group());
      sb.append((char)value);
    // append whatever text followed the final match
    m.appendTail(sb);
    return sb.toString();
  }This code is untested, but the technique is one I use often.
If your data might contain characters whose Unicode values are too large to fit in a 16-bit char, the process becomes a bit more complicated, but (as of JDK 1.5) the Character class has methods to handle them, too. If you're only dealing with the major asian character sets (i.e., CJK), I don't think you have to worry about that.