Chunked Transfer Encoding - Doesn't work due to implementation bug?

Question

Hello All,
I'm tackling the infamous "transfer-encoding: chunked" issue on trying to send POST data from the J2ME emulators and devices greater than 2048 bytes (or whatever the internal buffer size of a specific device might be). The nginx server on the other end always sends back a *411 - No content length specified* because it doesn't seem to recognize chunked transfer encoding. (despite the fact nginx is HTTP 1.1 compliant meaning it should be able to recognize it without rejection)
One thing I've noticed is that compared to a correctly chunked HTTP body which looks something like this:
*2*
ab
*4*
abcd
A
*0123456789*
*3*
foo
*0*
the HTTP bodies coming out of the MIDlet though Network Monitor in the emulator just look like this:
ababcd0123456789foo
It seems the chunked HTTP body does not have the chunk size indices that should be there for proper formatting.
I did my debugging on the Sun Java Network Monitor and maybe that tool "hides" the actual chunking indices
or the chunking takes place at a point beyond the emulator's snapshot of the data. Does anyone know if the Sun Java Network Monitor indeed shows the entire body of an HTTP POST in it's true state? The size of one packet I'm looking at is 2016 which is 32 bytes off 2048 - the supposed limit where data starts getting chunked. Could this "missing" 32 bytes from the log be the chunking indexes I am looking for? If the Network Monitor indeed shows the packet body in it's entirety then it seems like people are having issues sending chunked transfers because the format is wrong.
Anyway assuming J2ME's chunked transfer encoding is working correctly then how come nginx and many other supposedly HTTP 1.1 servers cannot understand the incoming format? Is Java's implementation of chunked transfer encoding somehow deviant from the set standard? I read a post at the following address
http://forum.java.sun.com/thread.jspa?forumID=76&threadID=454773
in which a user could not get chunked posts recognized by Apache until he changed the format of his HTTP body (read the last post on that thread for details on this).
In case no one can assist on the above how would I go about trying to intercept the HTTP data stream right after it leaves the emulator or a real device. Is this something that can only be done server side? If so how can I guarantee that the data I am receiving on the server side was not modified by another entity along the way?
Any help on this would be much appreciated. Thanks.
-Bob

Answer 1

We're still tackling the same issue but we've made a few discoveries:
* Nginx does not support chunked transfer encoding despite it's HTTP 1.1 status.
* The chunked transfer encoding format coming out of J2ME is correct. During some server side logging sessions we found that J2ME puts in the chunking indices and the finalizing 0 and carriage return to signify the end of the stream.
* The missing 32 bytes not shown in the Sun Java Network Monitor are the chunking indices - they are just hidden from you at that level of display. Since the J2ME emulator sends it's packets in 2048 bytes chunks when viewing the monitor you will just see the contents itself (2016 byte chunks). The other 32 bytes are the headers i.e.
7e0[carriage return]
which is a 32 bytes sequence (4 8 byte characters) with the last character being the carriage return. 7e0 in decimal is 2016 - the size of the displayed packet in the network monitor.
Anyone have any suggestions for a reliable server end receiver for chunked transfer encoding content? (preferably one which works well with Ruby on Rails)