Need an I/O Master

Question

First of all, this is a MIDlet so i cannot use java.net , only java.io and java.microedition.io, which lets me use InputStream and StreamConnection, etc.
I am trying to create a simple program to read data from a webpage and display it to the user. The page I need to access is a JSP page from a server maintained by Sony.com for the game PlanetSide. A sample address is:
http://myplanetside.station.sony.com/character.jsp?charId=382194&worldId=3
The problem that arises is I can only read a small portion of the page using an input stream. after a certain point of reading (always the same point), i constantly return a value of -1 from the stream. I have tried pausing the thread, putting read events into a timer, etc. but cannot read past the point in the stream. Separately, I have noticed that even on my DSL connection, the page loads in "spurts"; i.e. I navigated to the page, i saw a small portion appear, i clicked View=> Source and it had only downloaded the same portion i can download in my program. After a few seconds Internet Explorer downloads the rest of the page, but I cannot get my program to do this. To further frustrate me, I tried copying the HTML code to a separate web server of my own and my program works fine. This tells me there is no problem with the code to be downloaded just the way it is sent or something like that. As you may realize, I cannot simply connect to my web server because the page is a constantly updating page, so i cannot repeatedly copy it to my server. I use a simple InputStream.read( ) loop to read one byte at a time and cast it to a character and store in a StringBuffer object. I have been told to use a byte array, or InputStream.available( ) but nothing works for the file I need. Someone please help! I know it sounds hokey but i'd be willing to reward (via PayPal) someone who can give me a solution that works!!

Answer 1

the stream you request could be multipart..
check the http header upon recieve and check the chunked settings
if chunked, you will recieve a chunk of the page and a count of the next chunk size..
workaround. also try to specify http 1.0 in the request header.. thus the server "should" avoid chunked headers
best regards,
Per Fisker