GZIP decompression of chunked data?

Question

I'm trying to decompress a chunked stream of GZIP compressed data, but I don't know how to solve this without major inefficient workarounds.
The data is coming from a web server, and is sent chunked. This means that before each chunk, the size of the chunk is announced in plain-text, or 0 to terminate.
Simply wrapping the socket stream with the GZIPInputStream, like in the examples, only works if the stream is entirely GZIP, but this is not the case here.
I have to repeatedly do a readLine on the input stream to get the length of the chunk, and then I need to send that amount of bytes from the input stream to the GZIP decompressor. I'm stuck here as I don't know a way to 'send' selected bytes to the decompressor. I only know how to create the decompressor by wrapping an existing input stream, and simply by creating said decompressor it consumes bytes from the input stream to verify if it is a GZIP compressed stream.
The only thing I can come up with is to store the entire compressed data in a huge String, wrap it in a custom-made InputStream subclass that streams bytes from my String, and wrap that in the decompressor. Is this really the only way?
For example, a webpage like this sends it's data chunked and GZIP-compressed: http://www.anidb.net
Edited by: 834306 on Feb 6, 2011 11:16 AM
Edited by: 834306 on Feb 7, 2011 4:54 AM

Answer 1

Thanks for your responses.
I indeed thought I had to de-chunk manually. I'm fairly new to all this networking stuff. I've never even heard of a HttpURLConnection.
What I did before was opening a Socket, sending the manually-constructed request header and then receiving and manually decoding the http header. Something like this (shortened for readability)
socket = new Socket(url.getHost(), port);
dataOutputStream = new DataOutputStream(socket.getOutputStream());
inputStream = new BufferedInputStream(socket.getInputStream());
String message = "GET " + url.getFile() + " HTTP/1.1" + //
          "\nHost: " + url.getHost() + "" + //
          "\nUser-Agent: Mozilla/1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.13)" + //
          "\nAccept: text/html,application/xml" + //
          "\nAccept-Language: en-us,en;q=0.5" + //
          "\nAccept-Charset: ISO-8859-1" + //
          "\nConnection: close\n\n");
dataOutputStream.write(message.getBytes());
<here a lot of code to decode the header and put the field-value pairs in a HashMap>
<here a lot of code to receive the body of the message, while taking into account the reported content-length and chunking, as well as sending updates to a ProgressListener>I tried HttpURLConnection and indeed it seems a lot easier than the way I did things. However, what I receive from the HttpURLConnection is de-chunked but still GZIP compressed. So it is not completely transparent as I hoped. I wrapped in a decoder and it works.
Here's what I have now:
HttpURLConnection connection = null;
try
     connection = (HttpURLConnection)url.openConnection();
     connection.addRequestProperty("Host", url.getHost());
     connection.addRequestProperty("User-Agent", "Mozilla/1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.13)");
     connection.addRequestProperty("Accept", "text/html,application/xml");
     connection.addRequestProperty("Accept-Language", "en-us,en;q=0.5");
     connection.addRequestProperty("Accept-Charset", "ISO-8859-1");
     connection.addRequestProperty("Connection", "close");
     connection.setUseCaches(false);
     connection.setDoInput(true);
     InputStream inp = connection.getInputStream();
     if (connection.getHeaderField("Content-Encoding").equals("gzip"))
          inp = new GZIPInputStream(inp);
     int i;
     while ((i = inp.read()) != -1)
          logout.write(i);
     connection.disconnect();
catch (IOException e)
     if (connection != null) connection.disconnect();
}Thanks a bunch.
Edited by: 834306 on Feb 7, 2011 5:01 AM