Dependencies for speed of new generation gc

Question

I have two applications, doing very similar thing (they are both routing data from one source to another). They have almost the same number of thing to process per second, they are both done with a lot of shared code. Unfortunately, I can observe big difference in new gc times between both these applications. I have attached samples from gc logs below.
I'm using CMS and non-parallel new generation gc. Both applications are running on the machines with exactly the same specs (and we get same results if we switch the applications between the machines, so it is not environment specific). As you can see from the logs,
heap size is very similar, promotion ration is almost the same, both applications are running new generation gc once every 5 seconds on average.
I'm just wondering what can be the reason for so different gc times in both cases. Amount of different pages in old generation touched between collections ? Average size of the object surviving collection ?
In both cases following jvm is used
java version "1.5.0_07"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-b03)
Java HotSpot(TM) Server VM (build 1.5.0_07-b03, mixed mode)
Application 1:
8495.565: [GC {Heap before gc invocations=380:
def new generation   total 24576K, used 22521K [0x73000000, 0x74c00000, 0x74c00000)
  eden space 20480K, 100% used [0x73000000, 0x74400000, 0x74400000)
  from space 4096K,  49% used [0x74400000, 0x745fe620, 0x74800000)
  to   space 4096K,   0% used [0x74800000, 0x74800000, 0x74c00000)
concurrent mark-sweep generation total 1019904K, used 471954K [0x74c00000, 0xb3000000, 0xf0000000)
concurrent-mark-sweep perm gen total 131072K, used 14120K [0xf0000000, 0xf8000000, 0xf8000000)
8495.566: [DefNew
Desired survivor size 2097152 bytes, new threshold 6 (max 8)
- age   1:     743624 bytes,     743624 total
- age   2:     509984 bytes,    1253608 total
- age   3:     239488 bytes,    1493096 total
- age   4:     351904 bytes,    1845000 total
- age   5:     229328 bytes,    2074328 total
- age   6:     156096 bytes,    2230424 total
: 22521K->2178K(24576K), 0.1627934 secs] 494476K->474132K(1044480K) icms_dc=5 Heap after gc invocations=381:
def new generation   total 24576K, used 2178K [0x73000000, 0x74c00000, 0x74c00000)
  eden space 20480K,   0% used [0x73000000, 0x73000000, 0x74400000)
  from space 4096K,  53% used [0x74800000, 0x74a20898, 0x74c00000)
  to   space 4096K,   0% used [0x74400000, 0x74400000, 0x74800000)
concurrent mark-sweep generation total 1019904K, used 471954K [0x74c00000, 0xb3000000, 0xf0000000)
concurrent-mark-sweep perm gen total 131072K, used 14120K [0xf0000000, 0xf8000000, 0xf8000000)
, 0.1635298 secs]Application 2:
8297.934: [GC {Heap before gc invocations=303:
def new generation   total 24576K, used 22584K [0x70000000, 0x71c00000, 0x71c00000)
  eden space 20480K, 100% used [0x70000000, 0x71400000, 0x71400000)
  from space 4096K,  51% used [0x71800000, 0x71a0e218, 0x71c00000)
  to   space 4096K,   0% used [0x71400000, 0x71400000, 0x71800000)
concurrent mark-sweep generation total 799016K, used 482200K [0x71c00000, 0xa284a000, 0xf0000000)
concurrent-mark-sweep perm gen total 131072K, used 13206K [0xf0000000, 0xf8000000, 0xf8000000)
8297.935: [DefNew
Desired survivor size 2097152 bytes, new threshold 6 (max 8)
- age   1:     955304 bytes,     955304 total
- age   2:     346240 bytes,    1301544 total
- age   3:      94104 bytes,    1395648 total
- age   4:     347336 bytes,    1742984 total
- age   5:     189976 bytes,    1932960 total
- age   6:     174000 bytes,    2106960 total
- age   7:     110744 bytes,    2217704 total
: 22584K->2165K(24576K), 0.5525575 secs] 504784K->484561K(823592K) icms_dc=5 Heap after gc invocations=304:
def new generation   total 24576K, used 2165K [0x70000000, 0x71c00000, 0x71c00000)
  eden space 20480K,   0% used [0x70000000, 0x70000000, 0x71400000)
  from space 4096K,  52% used [0x71400000, 0x7161d6e8, 0x71800000)
  to   space 4096K,   0% used [0x71800000, 0x71800000, 0x71c00000)
concurrent mark-sweep generation total 799016K, used 482396K [0x71c00000, 0xa284a000, 0xf0000000)
concurrent-mark-sweep perm gen total 131072K, used 13206K [0xf0000000, 0xf8000000, 0xf8000000)
, 0.5533990 secs]

Answer 1

What command-line options do you give the JVM in the
two cases?Almost exactly the same (except for the main class and program arguments of course).
-Xloggc:path/to/gc/file.log -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:PermSize=128m -XX:+PrintTenuringDistribution -XX:MaxTenuringThreshold=8 -XX:NewSize=28m -XX:MaxNewSize=28m -XX:SurvivorRatio=5 -XX:+TraceClassUnloading -XX:+PrintHeapAtGC -XX:+CMSIncrementalMode -XX:CMSIncrementalDutyCycleMin=5 -XX:+CMSIncrementalPacing -XX:PrintCMSStatistics=1 -XX:-UseParNewGC -XX:+ParallelRefProcEnabled
for one of the programs I have specified
-Xmx2000m -Xms1024m
for second
-Xmx2048m -Xms256m
I'll try out how the second program behaves with -Xms1024. Sorry for number of gc options, but they are leftovers from my fights with CMS/Weakreference problems.
I see that in the first snippet no promotion
happened
while in the second some small number (196 KB) of
data was promoted. This was the accident, generally number of promoted memory is very similar in both cases.
Secondly, in the second case the
heap occupancy (as a %ge of the committed heap)
was larger; this might have affected the allocation
rate
in the old gen during scavenge.This might be related to -Xms specified to very low number, I'll check how it will behave after increasing the startup size. Anyway, I would expect bigger heaps to be generally slower ?
Can you think of ways in which the applications are
different wrt the object connectivity, and
lifetimes?Not really, they are both adapters for the very same data. We have a source of realtime data in the company, which contributes it to certain distribution environment. From this place, it is read by our applications (application A) and by adapter to reuters (written in C++). There we again read the same data from reuters (application B). Yes, it is a waste, but let's not go into business requirements ;) Anyway, custom protocol used by application A is very similar to reuters and output from both applications is exactly the same - so except for the few internal structures they are doing exactly the same thing. We used to had a problem with big mutable array in one of them, but it was already fixed in _07 (and it was affecting CMS times, not new gen times).
If you can condense the differences into a simple
test
case perhaps we can get closer to the problem.I'll first try out the startup heap sizing options. I don't think I'll be able to reproduce the problem on smaller application so easily.
Also make sure to collect a sufficiently long GC trace
so that you are observing real differences, not just
large fluctations in the GC times over the duration of
the run.I have full traces from 16 hours per day for last few weeks and they show this difference all the time.
Sorry that is not much help, and more questions than
answers.No problem, you have already helped me with pointing to the heap sizes (I was completly sure they are the same). I'll let you know about the results of -Xms test.
Thanks for help.