Deadlocking in java.io.UnixFileSystem?

Question

We're using Java 1.4.0 on Solaris 8. We're seeing our JVM hang after periods of intense file system activity. We can get a thread stack just before the hang where 243 of 338 threads appear to be busy with the file system. For instance:
"tcpConnection-6703-442" daemon prio=5 tid=0x9273a8 nid=0x2a7 runnable [c5280000..c52819c0]
at java.io.UnixFileSystem.list(Native Method)
at java.io.File.list(File.java:906)
at java.io.File.list(File.java:937)
"tcpConnection-6703-424" daemon prio=5 tid=0x31d6f0 nid=0x293 runnable [cbe80000..cbe819c0]
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:125)
at java.io.File.exists(File.java:679)
"tcpConnection-6703-414" daemon prio=5 tid=0x8ec5c8 nid=0x288 runnable [c0480000..c04819c0]
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:125)
at java.io.File.isDirectory(File.java:700)
"tcpConnection-6703-226" daemon prio=5 tid=0xeecc78 nid=0x1b1 runnable [c2380000..c23819c0]
at java.io.UnixFileSystem.checkAccess(Native Method)
at java.io.File.canRead(File.java:638)
I'm struggling to reconcile why the threads are listed as runnable even though the JVM will eventually become unresponsive, even to SIGQUIT.
How can i gain greater visibility on what's going on here? Any and all suggestions welcome!
Thanks!

Answer 1

We're seeing the exact same hang in lwp_mutex_lock on
ff31f144 lwp_mutex_lock (2d1e0)
fe0d5f84 __1cFMutexElock6M_v_ (2ac08, 2a, 0, 0, 1, fa000000) + 12c
fe12d3c0 __1cbCTwoGenerationCollectorPolicyRmem_allocate_work6MIii_pnIHeapWord__ (2fb40
, fe5cebc4, fe5ceb54, fe5313a2, 4800, 4948) + d4
The VM is also unresponsive to SIGQUIT.
We're running 1.4.2 on a 1 CPU Solaris 8 server, using both the parallel new gc (-XX:+UseParNewGC) along with the CMS gc for old space (-XX:+UseConcMarkSweepGC). We're also doing heavy filesystem I/O but I'm not sure that's affecting this. We haven't seen this issue until we started using those two garbage collectors together.
I don't have any solutions, just confirmation of your problem.