Resolving a TCP connection "slowdown" problem

Question

SuSE 9.3, stock kernel
Intel architecture
Jrockit-R26.4.0-jdk1.5.0_06-linux-ia32
I have a problem that appears to be localized in Jrockit (or
localized in the application, which is localized in Jrockit), where a
persistent and high-volume TCP connection slows down over the course
of about an hour--and eventually, effectively halts.
The TCP traffic is a stream of data, arriving at a near-constant rate
of about 16K bytes per second, with the receiving end (with the JVM
and app) strictly sending TCP ACKs in reply.
Restarting the sending process, or shutting down and restarting the
connection with the JVM / app, both restore the connection to full
speed until, over the course of perhaps an hour (sometimes more,
sometimes less) the same symptoms appear.
The symptoms in network packet traces are that when the connection is
first opened, the sender transmits packets at the full MTU of the
Ethernet segment. Gradually, the number of full-MTU sized packets
are replaced with packets much smaller packets, until most packets
range from 1 to 4 bytes with the occasional 3xx-4xx byte packet and
the odd offlier of a full-MTU packet size.
Supporting symptoms of interest:
1) The TCP window does not shrink
2) The CPU on the JVM/app side tops out at around 20%, even with
mySQL running on the machine
3) The interval between successive ACKs transmitted from the JVM/app
size generally narrows over the course of the connection
4) TCP send queue on the sender becomes saturated (pegged at 90+ K)
5) TCP receive queue on the JVM/app side is almost always 0, and when
it is not zero is bursts up to a low number (<50) and then almost
immediately returns to 0
6) The app does not appear to present any general symptoms of
slowness; the rate of writes to the database does not appear to slow.
The writes are threaded and multiplexed
[4] strongly implies that the slowness is caused by the JVM/app side,
since if the sender app was slowing down for some reason its TCP send
queue would not be saturated.
I can copiously document everything stated, and additionally provide
much additional detail.
Any guidance on how to suss out role Jrockit or the app are playing
in this little drama would be very deeply appreciated.

Answer 1

Asked around and it seems unlikely that this is a JVM issue. We have never heard of this behavior before, and the network layer in the JVM doesn't do anything with MTU iirc with the possible exception of manual changes to socket options. It seems more likely that this is caused by the IP stack, the NIC device driver or something in the network configuration. Try making some changes here and see what happens. For instance:
1) Run client and server on the same machine, communicating through loopback
2) Try another Linux distro (CentOS 4.3, for instance)
3) Try a different NIC and/or a different device driver
In your Java code, check that you are closing all Socket objects properly. Leaving them to be closed by a finalizer can delay closing sockets resulting in a native resource leak. I don't see how that would cause the issue you describe, but you never know...