CMS GC, no new cycle is started, full GC after one hour

Question

On several servers we observe Full GC events about twice a day, though we configured ConcMarkSweepGC.
The machines are running HotSpot 64-Bit Server VM 1.6.0 on Linux.
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:+UseParNewGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
There are regular CMS cyclyes, from CMS-initial-mark until CMS-concurrent-reset, but sometimes there is a Full GC.
For example:
*21294.717: [CMS-concurrent-sweep: 14.191/355.020 secs]
21294.717: [CMS-concurrent-reset-start]
21297.370: [CMS-concurrent-reset: 0.098/2.653 secs]
24886.612: [GC 24886.612: [ParNew: 75602K->6908K(76672K), 0.0545010 secs] 4825260K->4759005K(6135488K) icms_dc=0 , 0.0548030 secs]
24890.489: [GC 24890.489: [ParNew: 75068K->7099K(76672K), 0.0486250 secs] 4827165K->4760420K(6135488K) icms_dc=0 , 0.0489270 secs]
24894.024: [GC 24894.024: [ParNew: 75259K->5746K(76672K), 0.0484570 secs] 4828580K->4760784K(6135488K) icms_dc=0 , 0.0487920 secs]
*24894.719: [Full GC (System) 24894.719: [CMS: 4755038K->3184682K(6058816K), 33.2352640 secs] 4769958K->3184682K(6135488K), [CMS Perm : 34379K->34079K(57380K)]
icms_dc=0 , 33.2356150 secs]
24927.955: [GC [1 CMS-initial-mark: 3184682K(6058816K)] 3184687K(6135488K), 0.0032150 secs]
24927.958: [CMS-concurrent-mark-start]
24930.683: [GC 24930.683: [ParNew: 68160K->4298K(76672K), 0.0586990 secs] 3252842K->3188980K(6135488K) icms_dc=0 , 0.0589990 secs]
24934.629: [GC 24934.629: [ParNew: 72458K->7263K(76672K), 0.0363780 secs] 3257140K->3191946K(6135488K) icms_dc=0 , 0.0366640 secs]
24936.186: [GC 24936.186: [ParNew: 75423K->4406K(76672K), 0.0469830 secs] 3260106K->3192445K(6135488K) icms_dc=0 , 0.0472680 secs]
...We don't observe any concurrent mode failures.
My observations so far are:
1. there is no CMS-initial-mark after the previous CMS-concurrent-reset.
2. the time difference between the end of the previous cycle (timestamp of CMS-concurrent-sweep) and the Full GC is always exactly 3600 seconds plus a few milliseconds.
My first suspicion was, that System.gc() is being called somewhere. But I dropped this idea, after I found the 3600 seconds pattern.
Looks as if the Full GC happens as a replacement for a skipped CMS cycle one hour after the previuos cycle.
What could be the cause for this strange behaviour?
Often a new CMS cycle starts immediately after the previous cycle has been finished (I removed the ParNew lines):
24927.955: [GC [1 CMS-initial-mark: 3184682K(6058816K)] 3184687K(6135488K), 0.0032150 secs]
24927.958: [CMS-concurrent-mark-start]
28813.444: [CMS-concurrent-mark: 19.847/3885.486 secs]
28813.444: [CMS-concurrent-preclean-start]
28829.762: [CMS-concurrent-preclean: 0.496/16.318 secs]
28829.762: [CMS-concurrent-abortable-preclean-start]
28829.919: [CMS-concurrent-abortable-preclean: 0.053/0.157 secs]
28829.920: [GC[YG occupancy: 43128 K (76672 K)]28829.920: [Rescan (parallel) , 0.0945560 secs]28830.015: [weak refs processing, 0.0084490 secs] [1 CMS-remark:
4755774K(6058816K)] 4798903K(6135488K), 0.1062900 secs]
28830.027: [CMS-concurrent-sweep-start]
29124.744: [CMS-concurrent-sweep: 9.248/294.717 secs]
29124.744: [CMS-concurrent-reset-start]
29128.108: [CMS-concurrent-reset: 0.099/3.363 secs]
29130.816: [GC [1 CMS-initial-mark: 4359831K(6058816K)] 4400863K(6135488K), 0.1016900 secs]
29130.918: [CMS-concurrent-mark-start]
30558.326: [CMS-concurrent-mark: 16.822/1427.409 secs]
30558.326: [CMS-concurrent-preclean-start]
30683.920: [CMS-concurrent-preclean: 3.595/125.594 secs]
30683.920: [CMS-concurrent-abortable-preclean-start]
30684.076: [CMS-concurrent-abortable-preclean: 0.055/0.155 secs]
30684.076: [GC[YG occupancy: 42428 K (76672 K)]30684.076: [Rescan (parallel) , 0.0966020 secs]30684.173: [weak refs processing, 0.1068570 secs] [1 CMS-remark:
4980317K(6058816K)] 5022746K(6135488K), 0.2184810 secs]
30684.296: [CMS-concurrent-sweep-start]
31095.051: [CMS-concurrent-sweep: 13.242/410.756 secs]
31095.051: [CMS-concurrent-reset-start]
31095.149: [CMS-concurrent-reset: 0.097/0.097 secs]Here we have less than 2 seconds between the two cycles.

Answer 1

The presence of the "(System)" in the full GC line means that that collection was caused by a call to System.gc(). You can try running with "-XX:+DisableExplicitGC" as tschodt suggests, but that may have detrimental effects if something thinks it needs a full collection after an hour.
Are you using something like Java RMI, which needs a collection every once in a while to maintain its distributed garbage collection algorithms. You can change the interval at which RMI calls System.gc(), and you can get it to just start a CMS cycle instead of causing a full compacting collection.