German Umlauts OK in Test Environment, Question Marks (??) in production

Question

Hi Sun Forums,
I have a simple Java application that uses JFrame for a window, a JTextArea for console output. While running my application in test mode (that is, run locally within Eclipse development environment) the software properly handles all German Umlauts in the JTextArea (also using Log4J to write the same output to file-- that too is OK). In fact, the application is flawless from this perspective.
However, when I deploy the application to multiple environments, the Umlauts are displayed as ??. Deployment is destined for Mac OS X (10.4/10.5) and Windows-based computers. (XP, Vista) with a requirement of Java 1.5 at the minimum.
On the test computer (Mac OS X 10.5), the test environment is OK, but running the application as a runnable jar, german umlauts become question marks ??. I use Jar Bundler on Mac to produce an application object, and Launch4J to build a Windows executables.
I am setting the default encoding to UTF-8 at the start of my app. Other international characters treated OK after deployment (e, a with accents). It seems to be localized to german umlaut type characters where the app fails.
I have encoded my source files as UTF-8 in Eclipse. I am having a hard time understanding what the root cause is. I suspect it is the default encoding on the computer the software is running on. If this is true, then how do I force the application to honor german umlauts?
Thanks very much,
Ryan Allaby
RA-CC.COM
J2EE/Java Developer
Edited by: RyanAllaby on Jul 10, 2009 2:50 PM

Answer 1

So you start with a string called "input"; where did that come from? As far as we know, it could already have been corrupted. ByteBuffer inputBuffer = ByteBuffer.wrap( input.getBytes() ); Here you convert the string to to a byte array using the default encoding. You say you've set the default to UTF-8, but how do you know it worked on the customer's machine? When we advise you not to rely on the default encoding, we don't mean you should override that system property, we mean you should always specify the encoding in your code. There's a getBytes() method that lets you do that.
CharBuffer data = utf8charset.decode( inputBuffer ); Now you decode the byte[] that you think is UTF-8, as UTF-8. If getBytes() did in fact encode the string as UTF-8, this is a wash; you just wasted a lot of time and ended up with the exact same string you started with. On the other hand, if getBytes() used something other than UTF-8, you've just created a load of garbage. ByteBuffer outputBuffer = iso88591charset.encode( data );Next you create yet another byte array, this time using the ISO-8859-1 encoding. If the string was valid to begin with, and the previous steps didn't corrupt it, there could be characters in it that can't be encoded in ISO-8859-1. Those characters will be lost.
byte[] outputData = outputBuffer.array();
return new String( outputData ); Finally, you decode the byte[] once more, this time using the default encoding. As with getBytes(), there's a String constructor that lets you specify the encoding, but it doesn't really matter. For the previous steps to have worked, the default had to be UTF-8. That means you have a byte[] that's encoded as ISO-8859-1 and you're decoding it as UTF-8. What's wrong with this picture?
This whole sequence makes no sense anyway; at best, it's a huge waste of clock cycles. It looks like you're trying to change the encoding of the string, which is impossible. No matter what platform it runs on, Java always uses the same encoding for strings. That encoding is UTF-16, but you don't really need to know that. You should only have to deal with character encodings when your app communicates with something outside itself, like a network or a file system.
What's the real problem you're trying to solve?