JVM falls down often. Why? (I have thread dump)

Question

Good day!
Sometimes (about each two days) our production server falls down. But we can't understand why.
There is no one record in logs and stdout.
When server has falled down two days ago, I make full stack dump (kill -3). This is the link to stackdump:
http://users.g.com.ua/~vitaly_b/work/stackdump2.zip
I have writed tool that analyzes stackdump.
I've seen that there is 5 locks that are in waiting state from more than one another threads. This locks are in waiting state for some minutes.
One of this lock (0x00002b58afb89630) is never locked in stack dump, but 85 threads waits for it. How can it be???
Another 5 locks are locked with threads, that makes primitive operations for very long time. For example this thread now in java.util.HashMap.newKeyIterator, but 89 other threads are waiting for lock <0x00002b58afb85b1> for few minutes:
"http-8080-Processor1533" daemon prio=1 tid=0x00002aaab2566160 nid=0x6acb runnable
[0x000000004c410000..0x000000004c412c20]
     at java.util.HashMap.newKeyIterator(HashMap.java:889)
     at java.util.HashMap$KeySet.iterator(HashMap.java:921)
     at java.util.HashSet.iterator(HashSet.java:154)
     at net.sf.ehcache.event.RegisteredEventListeners.notifyElementExpiry(RegisteredEventListeners.java:116)
     at net.sf.ehcache.Cache.remove(Cache.java:1017)
     at net.sf.ehcache.Cache.searchInMemoryStore(Cache.java:829)
     at net.sf.ehcache.Cache.get(Cache.java:654)
     - locked <0x00002b58afb85b18> (a net.sf.ehcache.Cache)
     at org.hibernate.cache.EhCache.get(EhCache.java:75)
     at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:96)
     at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2086)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:595)Other locks are waits in same state.
Stack dump analysis with my tool:
waiting to lock <xxx>::
<0x00002b58afb8bd18>: 1 times.
<0x00002b58afb9aa50>: 1 times.
<0x00002b58afb9c138>: 2 times.
<0x00002b58afb8b690>: 1 times.
<0x00002b597ae1ecf8>: 1 times.
<0x00002b58afb85b18>: 89 times.
<0x00002b58afb88548>: 1 times.
<0x00002b58afb96110>: 1 times.
<0x00002b58afb89630>: 85 times.
<0x00002b58afb8e5a0>: 1 times.
<0x00002b58afb98090>: 1 times.
<0x00002b58afbacd28>: 1 times.
<0x00002b58afb99380>: 1 times.
<0x00002b58afb9a0f8>: 1 times.
<0x00002b58afb90d48>: 1 times.
<0x00002b58a34127a8>: 4 times.
<0x00002b58b09c5138>: 29 times.
<0x00002b58afb91d18>: 1 times.
waiting on <xxx>::
<0x00002b58afbb0370>: 1 times.
<0x00002b58adae0ff8>: 1 times.
<0x00002b58adae0f48>: 1 times.
<0x00002b58adae0d38>: 1 times.
<0x00002b58afbb0518>: 1 times.
<0x00002b58adae0de8>: 1 times.
<0x00002b58af9a7ba0>: 1 times.
<0x00002b58adae0e98>: 1 times.
<0x00002b58afbaf898>: 1 times.
<0x00002b58b0930400>: 1 times.
<0x00002b58afbaff00>: 1 times.My system configuration:
jdk 1.5.0.11
tomcat 5.5.20
SuSe Linux 10.1
How can I solve this problem? Help me please!
Thanks!

Answer 1

Good day!
Sometimes (about each two days) our production server falls down. But we can't understand why.
There is no one record in logs and stdout.
When server has falled down two days ago, I make full stack dump (kill -3). This is the link to stackdump:
http://users.g.com.ua/~vitaly_b/work/stackdump2.zip
I have writed tool that analyzes stackdump.
I've seen that there is 5 locks that are in waiting state from more than one another threads. This locks are in waiting state for some minutes.
One of this lock (0x00002b58afb89630) is never locked in stack dump, but 85 threads waits for it. How can it be???
Another 5 locks are locked with threads, that makes primitive operations for very long time. For example this thread now in java.util.HashMap.newKeyIterator, but 89 other threads are waiting for lock <0x00002b58afb85b1> for few minutes:
"http-8080-Processor1533" daemon prio=1 tid=0x00002aaab2566160 nid=0x6acb runnable
[0x000000004c410000..0x000000004c412c20]
     at java.util.HashMap.newKeyIterator(HashMap.java:889)
     at java.util.HashMap$KeySet.iterator(HashMap.java:921)
     at java.util.HashSet.iterator(HashSet.java:154)
     at net.sf.ehcache.event.RegisteredEventListeners.notifyElementExpiry(RegisteredEventListeners.java:116)
     at net.sf.ehcache.Cache.remove(Cache.java:1017)
     at net.sf.ehcache.Cache.searchInMemoryStore(Cache.java:829)
     at net.sf.ehcache.Cache.get(Cache.java:654)
     - locked <0x00002b58afb85b18> (a net.sf.ehcache.Cache)
     at org.hibernate.cache.EhCache.get(EhCache.java:75)
     at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:96)
     at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2086)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:595)Other locks are waits in same state.
Stack dump analysis with my tool:
waiting to lock <xxx>::
<0x00002b58afb8bd18>: 1 times.
<0x00002b58afb9aa50>: 1 times.
<0x00002b58afb9c138>: 2 times.
<0x00002b58afb8b690>: 1 times.
<0x00002b597ae1ecf8>: 1 times.
<0x00002b58afb85b18>: 89 times.
<0x00002b58afb88548>: 1 times.
<0x00002b58afb96110>: 1 times.
<0x00002b58afb89630>: 85 times.
<0x00002b58afb8e5a0>: 1 times.
<0x00002b58afb98090>: 1 times.
<0x00002b58afbacd28>: 1 times.
<0x00002b58afb99380>: 1 times.
<0x00002b58afb9a0f8>: 1 times.
<0x00002b58afb90d48>: 1 times.
<0x00002b58a34127a8>: 4 times.
<0x00002b58b09c5138>: 29 times.
<0x00002b58afb91d18>: 1 times.
waiting on <xxx>::
<0x00002b58afbb0370>: 1 times.
<0x00002b58adae0ff8>: 1 times.
<0x00002b58adae0f48>: 1 times.
<0x00002b58adae0d38>: 1 times.
<0x00002b58afbb0518>: 1 times.
<0x00002b58adae0de8>: 1 times.
<0x00002b58af9a7ba0>: 1 times.
<0x00002b58adae0e98>: 1 times.
<0x00002b58afbaf898>: 1 times.
<0x00002b58b0930400>: 1 times.
<0x00002b58afbaff00>: 1 times.My system configuration:
jdk 1.5.0.11
tomcat 5.5.20
SuSe Linux 10.1
How can I solve this problem? Help me please!
Thanks!