Encoding problem with xml

Question

Hello,
I'm trying to save on disk an xml with utf-8 codification and compressed with "gzip" algorithm. For some reasons, I have to use a BufferedWriter, then I must convert the byte[] to String.
Afterwards, I read this document and I try to decompress it and save on a String variable.
If I do all this process using utf-8 encoding, the decompressing process throws an exception: Not in GZIP format.
But if I use iso-8859-1, then everything works OK.
I don't understand why using iso works, and why using utf-8 does not work (when the webservice specification says that the xml documents are sent in utf-8).
The code is the following (the static "myCharset" is the key: when I set iso works, and setting utf-8 does not work):
public class testCompress
public static String xmlOutput     = "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><consultaCiudadesPorPaisResponse xmlns=\"http://tempuri.org/\"><consultaCiudadesPorPaisResult><xml funcion=\"ConsultaCiudadesPorPais\" xmlns=\"\"><ROK>TRUE</ROK></xml></consultaCiudadesPorPaisResult></consultaCiudadesPorPaisResponse></soap:Body></soap:Envelope>";
public static String myCharset     = "iso-8859-1";
// Writes the compressed document to disk.
public static void writeDocument() throws Exception
  BufferedWriter writer = null;
  try 
   // Compress the "xmlOutput" converting this string to bytes using "utf-8" as specification says (in "compress" method)
   // Afterwards, I convert this byte[] to String using "myCharset".
   String compressedFile = new String(CompressionService.compress(xmlOutput, "utf-8", "gzip", 8), myCharset);
   // And write to disk using "myCharset" as the encoding used by "OutputStreamWriter".
   writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream("c:/cache"), myCharset), 8192);
   writer.write(compressedFile);
  catch (Exception e) { throw e; }
  finally
    if (writer != null)
     try { writer.close(); writer = null; } catch (IOException ioe) {}
// Reads the compressed document from disk.
public static byte[] readDocument() throws Exception
  BufferedReader reader = null;
  StringBuilder sb           = new StringBuilder();
  char[] buffer           = new char[8192];
  try
   // Open the file using "myCharset" for reading chars.
   reader = new BufferedReader(new InputStreamReader(new FileInputStream("c:/cache"), myCharset));
   int numchars = 0;
   while ((numchars = reader.read(buffer, 0, 8192)) >= 0) sb.append(buffer, 0, numchars);
   // And return the result as a byte[] encoded with "myCharset".
   return (sb.toString().getBytes(myCharset));
  catch (Exception e) { throw e; }
  finally
   if (reader != null)
    try { reader.close(); } catch (IOException ioe) {}
public static void main(String[] args) throws Exception
  writeDocument();
  byte[] file = readDocument();
  // DECOMPRESS FAILS IF myCharset = "utf-8", and works if myCharset = "iso-8859-1"
  System.out.println(com.vpfw.proxy.services.compress.CompressionService.decompress(file, "utf-8", "gzip", 8));
}

Answer 1

Okay, here's what's happening. You created a byte[] by encoding some text as UTF-8, then you ran that byte[] through a gzip deflater. The result is binary data that can only be understood by a gzip inflater; to any other software it just looks like garbage. Now you're taking a randomly-chosen encoding and pretending the binary data is really text that was encoded with that encoding.
Most encodings have limits on what kinds of input they can accept. For example, US-ASCII only uses the low-order seven bits of each byte; any byte with a value larger than 127 is invalid. When the encoder encounters such a byte, it inserts the standard replacement character, U+FFFD, in that spot. When you try to decode the string again as US-ASCII, the replacement character is what you see in that position; the original byte value is lost. In UTF-8, the bytes have to conform to [certain patterns|http://en.wikipedia.org/wiki/UTF-8#Description]; for example, any byte with a value greater than 127 has to be part of a valid two-, three- or four-byte sequence.
ISO-8859-1 is different. It's a single-byte encoding like ASCII, but it uses all eight bits of every byte. Furthermore, every possible byte value (0..255) maps to a character, so you can throw any random byte at it and tell it the byte represents a character, and it will believe you. Some of those values may map to control characters that would look like garbage if you displayed them, but they're valid. That means you can re-encode the string as ISO-8859-1 and get back the exact byte sequence you started with.
So that's why your code "works" when you use ISO-8859-1, but I strongly recommend that you find another way; making binary data masquerade as text is dangerously fragile. Why do you have to use a Writer anyway? Is it for transmission over a medium that only accepts text data? If so, you should use a Base64 encoder or similar tool that's designed for that purpose.