Floating Dead Threads and old collection

Question

We run an application inside a "container" (for lack of a better word) that creates threads to try and connect outgoing connections. We have hundreds of these outgoing connections defined and most of them end up getting connected. A few (5-6?) however remain unconnected (there isn't a listener on the other side, the IP is pointing to a loopback address, etc.). This causes the container to create threads every few seconds (per connection) to try and connect these connections.
We finally turned on memdbg for -Xverbose and we see that all of our old garbage collections are because of "Floating Dead Threads." Below is a sample of the gc logging.
--- [snip] ---
[memory ][Wed Dec 24 06:49:01 2008][02747] Changing GC strategy to Generational Concurrent Mark & Sweep
[memdbg ][Wed Dec 24 06:53:21 2008][02747] Adding 1 final handles from dying thread 11888 'Socket Opener'.
[memdbg ][Wed Dec 24 06:53:23 2008][02747] Adding 1 final handles from dying thread 11893 'Socket Opener'.
[memdbg ][Wed Dec 24 06:53:23 2008][02747] Adding 1 final handles from dying thread 11899 'Socket Opener'.
[memdbg ][Wed Dec 24 06:53:23 2008][02747] GC reason: Other, cause: Floating Dead Threads
[memdbg ][Wed Dec 24 06:53:23 2008][02747] Stopping of javathreads took 5.722 ms
[memdbg ][Wed Dec 24 06:53:23 2008][02747] old collection 14 started
--- [snip] ---
I have a ticket opened with the vendor to see if there's a way to stop creating all these threads (I imagine the answer is going to be do not define connections that won't get connected). My question here is the following: Is there a way to make JRockit not perform an old collection for these final handles of the dying threads?
The current JVM parameters are:
DEBUGGCFLAGS="-Xverbosetimestamp -Xverbose:gc,memory,memdbg -Xverboselog=/home/core/CORE/ttDESCore/bin/gc_output.core2.txt -Xmanagement:ssl=false,authenticate=false"
GCFLAGS="-Xmx24g -Xns512m -XgcPrio:pausetime -XpauseTarget=400ms -XXcallProfiling -XlargePages -XXlargeObjectLimit:8k -XXnoSystemGC"
Thx for any help!
Edited by: user10720003 on Dec 26, 2008 4:54 AM

Answer 1

Finally a reply! :) Thanks for taking the time to respond. A few points:
1) Sun's JVM doesn't seem to have this problem (or at least it doesn't trigger an old collection like JRockit). Any ideas as to why?
2) The finalizers are actually not in our application code (it's vendor code). For whatever reason, any connections (these are TCP/IP connections) that we define as initiate=true, the vendor creates a thread for each connection every 5 seconds to try and make the connection. If it succeeds the thread lives to perform additional work (and hence no finalizer needed). If it does not connect, the thread dies and a finalizer is added.
3) There were two connections that we had that did not make a successful connection so I had operations disable those connections. As soon as we did that, the "Add 1 handle for dying thread..." message went away in the memdbg log. However, JRockit still insisted on performing an old GC every 1.5 minutes for 'Floating Dead Threads'. Any idea why the old GC would still happen for this reason? Is it because all the finalizers still haven't been cleaned up?
A question about the finalize() method and finalizers. I looked through our code and we create about 80 objects that have a finalize() method implemented. These objects never go out of scope (so in reality, I suppose the finalize method doesn't need to be there). Does leaving it defined still hurt the gc performance? Or do finalizers only causes problems (i.e. kick off old gcs) once they become garbage?
Thx,
Josh
Edited by: user10720003 on Dec 29, 2008 8:03 AM
Edited by: user10720003 on Dec 29, 2008 8:15 AM
Edited by: user10720003 on Dec 29, 2008 8:21 AM