Sudden and strange change in GC behaviour

Question

I see very strange GC behaviour with our application running by one of our customers, but I fail to reproduce it in lab. Most of the time the application is running just fine, and suddenly GC may start doing absolutely strange things.
Firstly, it starts running mostly major collections (normally one major collection in a few hours under the same user load)
81703.7: [GC 81703.7: [DefNew: 737664K->737664K(760704K), 0.0000446 secs]81703.7: [Tenured: 2173068K->588301K(2350848K), 13.0162284 secs] 2910732K->588301K(3111552K), 13.0167844 secs]
81730.7: [GC 81730.7: [DefNew: 737664K->319K(760704K), 0.0544614 secs] 2205198K->1467853K(3111552K), 0.0548729 secs]
81746.8: [GC 81746.8: [DefNew: 735966K->735966K(760704K), 0.0000519 secs]81746.8: [Tenured: 2347147K->580785K(2350848K), 9.9294041 secs] 3083113K->580785K(3111552K), 9.9300118 secs]
The size of used memory suggests that objects get allocated directly to Tenured Generation. Secondly, collections become much more frequent (from 150-200 secs to 10-20 sec gap between collections). All together produce an astonishing rate of memory usage (acquired memory per second) of 50-100 MB/sec (normally it is about 2MB/sec). On the other hand, major collections take much less time, from 40 sec to 6-8 sec, which is good.
I do not see anything happening on the server that could explain such a major jump in memory usage! Also, the users do not see any significant and sudden change in response time. But over time it definitely affects performance (mostly because of frequent major collections) and the server needs to restart to get back to normal.
JVM version 1.4.1._02 on Solaris 8, parameters:
-Xms3000000 �Xmx3000000 -XX:NewRatio=3
I cannot reproduce it in test environment, so I cannot try a different JVM version and need to know why it happens before changing any JVM parameters (the current set looks a bit conservative but it works fine most of the time)

Answer 1

I see very strange GC behaviour with our application
running by one of our customers, but I fail to
reproduce it in lab. Most of the time the application
is running just fine, and suddenly GC may start doing
absolutely strange things.
Firstly, it starts running mostly major collections
(normally one major collection in a few hours under
the same user load)
81703.7: [GC 81703.7: [DefNew:
737664K->737664K(760704K), 0.0000446 secs]81703.7:
[Tenured: 2173068K->588301K(2350848K), 13.0162284
secs] 2910732K->588301K(3111552K), 13.0167844 secs]
81730.7: [GC 81730.7: [DefNew:
737664K->319K(760704K), 0.0544614 secs]
2205198K->1467853K(3111552K), 0.0548729 secs]
81746.8: [GC 81746.8: [DefNew:
735966K->735966K(760704K), 0.0000519 secs]81746.8:
[Tenured: 2347147K->580785K(2350848K), 9.9294041
secs] 3083113K->580785K(3111552K), 9.9300118 secs]
As you noticed, objects are getting allocated directly in the
Tenured Generation. (As background for those of our
readers new to reading GC logs:
The occupancy of the Tenured(Old) generation following the
completion of the GC at 81730.7 is (1467853 - 319)K =
1467534 K. Now look at the Occupancy of the Tenured
generation at the start of the GC at 81746.8; it is
2347147 K. That means that between these two
collections, 879613 K (or about 870 MB) was directly
allocated into the old generation.)
More about that direct allocation in the old generation later.
But a consequence of the fact that the old generation is so full,
is that the remaining free space is insufficient to allow the worst
case promotion from the young generation to succeed. As a result
a scavenge is not attempted, but rather a full compacting GC is
done instead.
Two things now.
First, if you use a more recent version of the JVM (1.5.0_06 i believe)
and set -XX:+HandlePromotionFailure, then you need not be hobbled
by the pessimistic worst-case promotion causing a full collection
to happen, when in fact a scavenge might indeed have succeeded
and sufficed. (see
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young%20Generation%20Guarantee%7Coutline
or Google "Sun JVM Young Generation Guarantee").
Second, and coming back to the direct allocation issue, this
may be caused by one of two possible factors:
. it could be that your program is trying to allocate very large
objects in Eden that would not fit there and require
allocating directly in the much larger old generation.
Think about the kinds of data structures you have in your program.
Do you have largish hashmaps (which typically grow by doubling --
that could be one reason for a very large allocation request. Use
+PrintClassHistogram to understand the kinds of data structures
extant in your heap and their sizes; once again the document I mentioned
above or Google "HotSpot JVM PrintClassHistogram" or check out
Joe Mocker's HotSpot options list for how to use itL:
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
. it could also be that your application is making heavy use of
serialization (for example), or through some other means (perhaps
directly through JNI) executing JNI critical sections which may be
locking out GC for an extended period of time. In older JVM's this
could translate into extremely poor mutator (application) performance
until the critical section is exited. The reason is that GC is locked
out in the interim and once Eden fills up, allocations go slow-path to
the old generation.
The latter has been addressed in later JVM's; please try 1.5.0_06 for example
and let us know your experience.
Finally, it would seem as though your application needs a
large Java heap and you want to avoid excessive GC costs/pauses.
Consider running with the mostly concurrent collector especially
if you have some concurrent processing power to spare on your
deployment platform. Please refer to:
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.%20The%20Concurrent%20Low%20Pause%20Collector%7Coutline
The size of used memory suggests that objects get
allocated directly to Tenured Generation. Secondly,
collections become much more frequent (from 150-200
secs to 10-20 sec gap between collections). All
together produce an astonishing rate of memory usage
(acquired memory per second) of 50-100 MB/sec
(normally it is about 2MB/sec). On the other hand,
major collections take much less time, from 40 sec to
6-8 sec, which is good.
I do not see anything happening on the server that
could explain such a major jump in memory usage!
Also, the users do not see any significant and sudden
change in response time. But over time it definitely
affects performance (mostly because of frequent major
collections) and the server needs to restart to get
back to normal.
JVM version 1.4.1._02 on Solaris 8, parameters:
-Xms3000000 �Xmx3000000 -XX:NewRatio=3
I cannot reproduce it in test environment, so I
cannot try a different JVM version and need to know
why it happens before changing any JVM parameters
(the current set looks a bit conservative but it
works fine most of the time)