WeakReference impact on gc times

Question

We are using JDK 5.0 on 4 CPU Sparc Solaris, in both 32 and 64 bit mode, CMS, parallel gc for new gen.
In our application, due to specific design decision, we ended up with having large number of weak references. We have a max heap of 2GB, with almost-permanent data being around 0.5GB and rest of heap being filled out by semi-transient data. New generation gc is actually doing very good job - sometimes we don't have to run CMS collection for entire day under small load. Unfortunately, when it runs, we hit a big pause during a stop-the-world phase which happens in middle of collection. 'Processing weak refs' part takes between 7-30s for us... with rest of this blocking phase taking maybe 0.3-0.5s. Funny thing is that it is often the second CMS which takes so long - first one manages to run with under 1s time for weak ref processing.
I would like to understand what is exactly happing at 'processing weak refs' phase. Is the time taken by it proportional to number of weak references which have to be cleared and enqueued,to number of still alive weak references, or to total number of them ? Is there any chance of processing them in non-blocking mode (we can afford one CPU for gc running in background, but we cannot afford pause times bigger than 2s) ?

Answer 1

We are not longer experiencing 2 minute pauses. I have enabled parallel weak ref processing and currently 2-3 second pauses are the biggest we can see. As production machine will be 50%+ more powerful, we could probably live with that - as long as it will not become worse. Unfortunately we are not able to easily reproduce it - it happens only once per few CMS collections, plus there is not easy way to force CMS collection at given time (full gc which can be forced through jconsole takes 1 minute+ and is probably very different from normal case).
As for the weak references, I know how they are allocated. We were allocating big number of weak references in bursts (1000-10000 at one time). Objects they pointed to were strongly referenced for umpteen minutes, then strong references were dropped. Somewhere during early time (most probably when they will still mostly referenced) there was CMS run, but blocking phase took around 0.3s. Few hours later, after 80%+ weak references were already possible to collect, another CMS has run. Both weak references and objects they were pointing to were not touched by anything during this time - and it could be easily 100-200k objects (with probably 3-4 unique objects referenced from each of them) + same number of weak references.
Below I have attached gc log around biggest CMS pause. Our vm flags are
-d32 -server -Xmx3000m -Xms2048m
-Xloggc:somefile/somewhere -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseParNewGC -
XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:PermSize=128m -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:MaxTenuringThreshold=8 -XX:NewSize=28m -XX:MaxNewSize=28m -XX:SurvivorRatio=5 -XX:+ParallelRefProcEnabled -XX:+TraceClassUnloading -XX:+PrintHeapAtGC
22427.997: [GC  {Heap before GC invocations=4978:
Heap
par new generation   total 24576K, used 22658K [0x34c00000, 0x36800000, 0x36800000)
  eden space 20480K, 100% used [0x34c00000, 0x36000000, 0x36000000)
  from space 4096K,  53% used [0x36000000, 0x36220b50, 0x36400000)
  to   space 4096K,   0% used [0x36400000, 0x36400000, 0x36800000)
concurrent mark-sweep generation total 2068480K, used 1912742K [0x36800000, 0xb4c00000, 0xf0400000)
concurrent-mark-sweep perm gen total 131072K, used 17676K [0xf0400000, 0xf8400000, 0xf8400000)
22427.998: [ParNew
Desired survivor size 2097152 bytes, new threshold 3 (max 8)
- age   1:     927464 bytes,     927464 total
- age   2:     787824 bytes,    1715288 total
- age   3:     439992 bytes,    2155280 total
: 22658K->2121K(24576K), 0.1776179 secs] 1935401K->1915402K(2093056K) Heap after GC invocations=4979:
Heap
par new generation   total 24576K, used 2121K [0x34c00000, 0x36800000, 0x36800000)
  eden space 20480K,   0% used [0x34c00000, 0x34c00000, 0x36000000)
  from space 4096K,  51% used [0x36400000, 0x36612788, 0x36800000)
  to   space 4096K,   0% used [0x36000000, 0x36000000, 0x36400000)
concurrent mark-sweep generation total 2068480K, used 1913280K [0x36800000, 0xb4c00000, 0xf0400000)
concurrent-mark-sweep perm gen total 131072K, used 17676K [0xf0400000, 0xf8400000, 0xf8400000)
} , 0.1788122 secs]
22428.177: [CMS-concurrent-preclean-start]
22430.743: [CMS-concurrent-preclean: 2.340/2.566 secs]
22430.743: [CMS-concurrent-abortable-preclean-start]
22430.827: [CMS-concurrent-abortable-preclean: 0.080/0.084 secs]
22430.835: [GC[YG occupancy: 16724 K (24576 K)]22430.835: [Rescan (parallel) , 0.0838017 secs]22430.919: [weak refs processing, 2.4906309 secs]22433.410: [c
lass unloading, 0.1508142 secs]22433.561: [scrub symbol & string tables, 0.0329090 secs] [1 CMS-remark: 1913280K(2068480K)] 1930005K(2093056K), 3.0276614 se
cs]
22433.865: [CMS-concurrent-sweep-start]
22434.079: [GC  {Heap before GC invocations=4979:
Heap
par new generation   total 24576K, used 22601K [0x34c00000, 0x36800000, 0x36800000)
  eden space 20480K, 100% used [0x34c00000, 0x36000000, 0x36000000)
  from space 4096K,  51% used [0x36400000, 0x36612788, 0x36800000)
  to   space 4096K,   0% used [0x36000000, 0x36000000, 0x36400000)
concurrent mark-sweep generation total 2068480K, used 1912786K [0x36800000, 0xb4c00000, 0xf0400000)
concurrent-mark-sweep perm gen total 131072K, used 17677K [0xf0400000, 0xf8400000, 0xf8400000)
22434.080: [ParNew
Desired survivor size 2097152 bytes, new threshold 3 (max 8)
- age   1:    1295336 bytes,    1295336 total
- age   2:     489792 bytes,    1785128 total
- age   3:     602880 bytes,    2388008 total
: 22601K->2345K(24576K), 0.4696339 secs] 1935388K->1915406K(2093056K) Heap after GC invocations=4980:
Heap
par new generation   total 24576K, used 2345K [0x34c00000, 0x36800000, 0x36800000)
  eden space 20480K,   0% used [0x34c00000, 0x34c00000, 0x36000000)
  from space 4096K,  57% used [0x36000000, 0x3624a400, 0x36400000)
  to   space 4096K,   0% used [0x36400000, 0x36400000, 0x36800000)
concurrent mark-sweep generation total 2068480K, used 1913061K [0x36800000, 0xb4c00000, 0xf0400000)
concurrent-mark-sweep perm gen total 131072K, used 17677K [0xf0400000, 0xf8400000, 0xf8400000)
} , 0.4708485 secs]