Simulate full garbage collection pauses

Hi,
I'm doing some performance tuning for our Java systems, and one thing I want to test is how the applications behave when "stop-the-world" Full GC pauses of various different durations occur in our JVMs. We have seen some problems in our systems where longer GC pauses have negative effects. However, it is difficult to recreate these conditions using the applications themselves. Does anyone know if it is possible to force a JVM to do a "stop-the-world" pause of a specified duration, that would simulate a GC?
Notes -
1) I'm not talking about forcing a GC - I know how to do this but this give no control on the duration of the GC. Appreciate might be able to work a solution by writing a program to use up X amount of memory followed by a forced GC, but hoping for something simpler!
2) We have already exhaustively tuned our JVMs to reduces frequency and duration of Full GC as much as possible but they do still need to occur occassionally. Have engaged with forums alredy on how best to do this, so no need for more info on this!
3) We are using multiple JVMs which communicate with each other (RMI, etc.), and a large part of what I'm testing is the effect on JVM A of a long GC pause in JVM B while A is communicating with B
4) JVM version is 1.5.0_u18, normally running on Solaris 9/10
Thanks in advance for any insights!
Regards,
Adrian

Hi,
The docs here:
[http://docs.sun.com/app/docs/doc/806-1367/6jalj6mv1?a=view|http://docs.sun.com/app/docs/doc/806-1367/6jalj6mv1?a=view]
seem to suggest something like what I want to do could have been achieved in Java 1.2, by sending SIGQUIT to the process. However, in Java 5, SIGQUIT just creates a thread dump. Anyone know if its possible to achive the behaviour from that link in Java 5 by any other means / signal?
Regards,
Adrian
Edited by: AdrianFitz on 05-Mar-2010 13:40

Similar Messages

Garbage Collection Pauses in a client application: Trashing.

Hi,
we have put in production a financial trading application written in Java. We are experimenting a strange behaviour of the garbage collector. We are experimenting GC pauses every 4-5 hours which raise CPU time to 100% for 30 seconds. These pauses are not-deterministic. Memory consumption of the entire application is very low (about 30MB).
This is a strange beheaviour we have detected in the log file of GC:
5020.027: [GC 27847K->14659K(63936K), 0.0086360 secs]
5020.036: [Inc GC 14659K->24546K(63936K), 0.0149416 secs]
5020.107: [GC 27842K->14658K(63936K), 0.0086947 secs]
5020.116: [Inc GC 14658K->24546K(63936K), 0.0094716 secs]
5020.181: [GC 27842K->14658K(63936K), 0.0086846 secs]
5020.190: [Inc GC 14658K->24546K(63936K), 0.0095778 secs]
5020.255: [GC 27842K->14658K(63936K), 0.0102155 secs]
5020.266: [Inc GC 14658K->24546K(63936K), 0.0084659 secs]
5020.335: [GC 27842K->14658K(63936K), 0.0088606 secs]
5020.344: [Inc GC 14658K->24554K(63936K), 0.0084514 secs]
as you can see in one second the GC and the IncGC are called many times. WHY?
We have read articles and tried all various known settings. But we didn't find any solution and at the moment we don't know how to move.
Thank you for your hint
Piero

Which settings have you tried? By the looks of things you've not set any of your JVM memory variables.
Try bumping up your maximum heap space using -Xmx.
If you are running on a nice server try using -Xmx256m -Xmx256m -Xmn64m, these should help for a start.
Also why are you using the "incremental garbage collector" when you have such a tiny application as you can see a full garbage collection is only taking 0.01 seconds.
Also try reading this http://java.sun.com/docs/hotspot/gc/

Can I force full garbage collection?

Hi, my program is memory bound , as users load more files more memory is required , if the user decides to load a very large number of files they will eventually run out of memory. So I am trying to detect when there is less 15% of heap memory left, then force garbage collection and if it cant free up more than 15% of the heap I will stop the user from loading any more files. But the problem is though I call System.gc() to try and force a full garbage collect it rarely
retrieve enough memory to get below the 15% limit. But using the Yourkit Profiler I can select the Force GarbageCollection option and this always manages to free up more memory to get the figure under the 15% limit. in support of this I found that sometimes my program stop me loading more files when there is still quite a bit available.
So my questions are.
1. I know System.gc() is only hint to garbage collect, but the docs imply it only replys after the garbage collection (if any) has been done, is this right or do I have to wait.
2. Is there any way to Force complete Garbage Collectionas profiler appears to do.
3. is there a VM option I could set instead to force the JVM to completely garbage collect at say 83% so that if I then polled that 85% of heap was being used I would know that it really was, and I wouldnt need to bother trying to garbage collect further. (Im using Suns 1.6. JVM on Windows and Linux, and Apples 1.5 or 1.6 JVM on Macs)
public static void checkMemoryWhilstLoadingFiles() throws LowMemoryException
        MemoryUsage mu = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage();
        //No max defined future proofing
        if(mu.getMax()==-1)
            return;
        if (mu.getUsed() > (mu.getMax() * 0.85f))
            MainWindow.logger.warning("Memory low:" + mu);
            System.gc();
            MainWindow.logger.warning("Memory low gc1:" + ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
            System.gc();
            MainWindow.logger.warning("Memory low gc2:" + ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
            System.gc();
            MainWindow.logger.warning("Memory low gc3:" + ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());
            mu = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage();
            if (mu.getUsed() > (mu.getMax() * 0.85f))
                MainWindow.logger.severe("Memory too low:" + mu);
                throw new LowMemoryException("Running out of memory:"+mu.getUsed());
            else
                 MainWindow.logger.warning("Memory usage reduced to:" + mu);
    }thanks for any help Paul
Edited by: paultaylor on 27-Jun-2008 11:10

On all of the current Sun HotSpot JVM's, calling System.gc() will cause a full compacting collection. Unless you have -XX:+DisableExplicitGC on your command line, in which case the call is a noop. Or if you are running the mostly-concurrent collector (-XX:+UseConcMarkSweepGC) and have the -XX:+ExplicitGCInvokesConcurrent flag on your command line, in which case calling System.gc() will start a concurrent collection (and the calling thread will block until the cycle is finished).
But calling System.gc() isn't enough to recover all the space that might be recovered. For example, System.gc() will identify objects that are unreferenced but need to have their finalize() methods called before their space becomes available again. So one call to System.gc() won't recover their space. Those finalize() methods need some cycles to run in, so back-to-back (or back-to-back-to-back :-) calls to System.gc() won't help. If you use a lot of finalize() methods, you should leave a lot of time for the finalize() methods to run between the calls to System.gc(). (Better would be to convert your code to use WeakReferences and run your own reference processing queues, and then you could tell when you were done processing references. But that's real work.) Some people try calling System.runFinalization() and wait for that to return, but that has at least two failure modes (details left to the reader).
In addition, there are details like: if there is still 15% of the heap free, then we won't aggressively clear SoftReferences when you call System.gc(). We might if you waited until the heap was full and we collected it on our own, since we know how much free space there will be after a collection at the point where we are choosing which SoftReferences to clear, and use that to decide how aggressively to clear SoftReferences.
There is no method to force the collector to do a compacting collection at, say 85% full. There is an option to have the mostly-concurrent collector start a collection cycle that way. But there's no way to find out if a collection cycle is running.
You are skating on the edge of the qualities of service offered by the different collectors in the various JVM's available. That weakens your ability to "write once, run anywhere".

Full Garbage Collection Problem

Hi All,
We are working on NetWeaver Application Server JAVA 7.0
I am getting an error message in of the the EWA reports for JAVA system. The red alert says as below:
The maximum ratio of full garbage collections to total garbage collections in the reported interval was higher than 90%.
In order to solve the above problem, I increased Heap Memory for all JAVA Server nodes to 3072 (earlier it was 2048 for all the server nodes). However, still I am getting same error in EWA report.
Can any one help me in further analysing and solving the above problem?
Your help is appreciated.

Here are links to some of the tools. I have worked with [IBM GC for IBM JVM|http://www.ibm.com/developerworks/java/library/j-ibmtools2/index.html]. You may have to try others that can read Sun JVM's GC log.
http://www.tagtraum.com/gcviewer.html
http://www.yourkit.com/overview/index.jsp
https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER
http://java.sun.com/performance/jvmstat/visualgc.html

Full garbage collection issue, not releasing/flagging memory

I have the following problem running on a multi-cpu windows server with Java 1.4.2_05 using WebLogic 8.1:
During a lifecyle of the web application (under load, but not to heavy) memory usage seems ok and garbage collection is called regularly. Suddenly, the used heap starts to rize very fast and after a while, even a full garbage collection cylce, does not release any memory anymore.
I am sure that, from our coding, we release memory ok, and normally we should only use about 5 to 10 mb for each user max (with0 normal defnew garbage collections).
I tried changing the garbage collection parameters, but this does not solve the problem. Best scenario was with the concurrent collector and I got this output at +/- the end:
[GC 100202K->93511K(115628K), 0.0091472 secs]
[GC 148480K->139612K(163808K), 0.0225914 secs]
[Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor289]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor290]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor273]
153750K->133006K(164064K), 1.2434402 secs]
[GC 148939K->137948K(203264K), 0.0223085 secs]
[GC 188789K->177116K(203264K), 0.0180729 secs]
[Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor312]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor322]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor309]
189788K->170264K(203264K), 1.1851945 secs]
[Full GC 203228K->203227K(203264K), 1.2876122 secs]
[Full GC 203263K->203233K(203264K), 1.3354548 secs]
[Full GC 203263K->203258K(203264K), 1.2873518 secs]
<Jan 17, 2007 9:40:40 AM EST> <Error> <HTTP> <BEA-101017> <[ServletContext(id=33114655,name=console,context-path=/console)] Root cause of ServletException.
java.lang.OutOfMemoryError
>
[Full GC 203263K->203233K(203264K), 1.2814516 secs]
[Full GC 203233K->203231K(203264K), 1.6029044 secs]
[Full GC 203263K->203242K(203264K), 1.3081352 secs]
<Jan 17, 2007 9:41:51 AM EST> <Emergency> <WebLogicServer> <BEA-000210> <The WebLogic Server is no longer listening for connections.>
[Full GC 203263K->203247K(203264K), 1.3161194 secs]
[Full GC 203263K->203249K(203264K), 1.2954988 secs]
[Full GC 203263K->203247K(203264K), 1.6423404 secs]
<Jan 17, 2007 9:41:57 AM EST> <Alert> <WebLogicServer> <BEA-000218> <Server shutdown has been requested by <WLS Kernel>>
[Full GC 203263K->203250K(203264K), 1.3161025 secs]
Another strange item is: I maximized the amount of memory it uses to 512m with the Xmx parameter, I am almost sure that that one is used, but it never gets higher than 203M? Does anyone know why this is?
Another strange item: the monitoring in the weblogic code indicates 32MB of usage (relative memory usage seems to be ok, but the quanity indication is just plain wrong) with 15 threads running.
This problem does not exist when using JBoss 4.0.2 or 4.0.3 (standard j2ee settings).
If anyone has an idea or can help me, I would appreciate it very very much. :)

Hi ,
Is this issue resolved ?
we are facing same problem.
1. We have checked the CPU and memory utilization everything is normal
2. GC logs showing FULL GC calls continuously
3. After restart the resin server system is working normally.
Environment detail
Resin ./resin-pro-3.0.18 on suse Linux
Java JDK1.4.2_08
Please suggest

How to specify when Full Garbage Collections occur in the Old Generation

Hi. We seem to be having a problem with a number of JVMs (1.5.0_17-b04) that run a component of a Document Management application. This component stores a large amount of information in caches which reside in the Old Generation. Although these cache sizes can be somewhat controlled by the application, they are currently taking about 85% of the Old Generation space. Fortunately, very few objects get tenured into the Old Generation - they all are cleaned up in the New Generation space.
The problem we are seeing is that with the Old Generation at 85% full, there are constant full GC's occurring. Since the caches cannot be removed, the system frantically tries to remove objects that can't be removed.
We have three solutions in mind. The first is to increase the memory allocation to the Old Generation so that the caches take a smaller percentage of the available memory allocation. The second would be to decrease the size of the caches; but this is set more by the number of documents in the application and cannot be made much smaller.
The third solution is to configure the JVM so that Garbage Collections in the Old Generation do not occur until the memory is more than a specific percentage of memory in the Old Generation. We would then set this percentage to be higher than the amount of memory being used by the caches.
So, is it possible to tell the JVM to only run a Full GC when the memory in the Old Generation is greater than a specific value (say 85% full)?
Thanks for your help.
Andre Fischer.

afischer wrote:
The third solution is to configure the JVM so that Garbage Collections in the Old Generation do not occur until the memory is more than a specific percentage of memory in the Old Generation. We would then set this percentage to be higher than the amount of memory being used by the caches.
So, is it possible to tell the JVM to only run a Full GC when the memory in the Old Generation is greater than a specific value (say 85% full)?Switch to the CMS collector.
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=86

Reducing full Garbage Collection frequency.

I've been trying to improve the performance while inserting a large number of records to an embedded H2 database. Monitoring memory useage suggests that it's being used rather innefficiently. A lot of the objects created by H2 seem to find their way into "tenured" space before being freed. Full mark-and-sweap garbage collections are occuring every couple of seconds, despite the fact that only about 10% of the available heap is occupied.
Any advise on tuning the garbage collector to improve throughput in this case?

I think if surviving objects max out the to-space (part of the young generation) the remaining objects are copied into the tenured generation.
You can get more info via:
-XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGC -Xloggc:<filename>

Full Garbage Collection

Hi Friends,
I'm using weblogic workflow for my project. Last night i got one error, once i gone throw the bea ...i come to know that the error will comes because of "once the application calls webservice and the webservice intern calls the EJB stateless or stateful will fail". But my application is not using any sort of webservices. So i try to find the problem finally i found tht the problem is because of Garbage collection of Heap Size. It is taking 3.8508577 sec's. I feel in this time the JVM thread gets heighest priority and it is killing the application thread which is to be executed as usal.
Can u guide me how to catch this exception so tht my application won;t get affected. The actual error says like this which is related to Garbage Collection...[Full GC 313152K -> 105060K (1004928K), 3.8508577 secs]. I'm using JDK 1.4.
Thaks & Regards
[email protected]

Replies in this thread.

Garbage collection pauses screw up my cluster. JVM tuning suggestions?

I am running Sun's 64 bit JVM 1.7.0. Everything hums along nicely until my storage enabled nodes all show GC log entries like these and then the cluster experiences timeouts. This happens like once a day at most. Suggestions? Thinking maybe add more storage enabled JVMs? Any JVM settings I should use to prevent this? What can determine the cause? I don't think the machine's CPU is maxed out at the time.
Thanks,
Andrew
2012-02-02T09:09:15.585-0600: 61965.351: [GC 61965.351: [ParNew: 915634K->5291K(1023552K), 0.0396299 secs] 1143514K->233236K(3299264K), 0.0406305 secs] [Times: user=1.26 sys=0.00, real=0.04 secs]
2012-02-02T09:09:22.816-0600: 61972.582: [GC 61972.583: [ParNew: 915115K->4454K(1023552K), 0.0497440 secs] 1143060K->233468K(3299264K), 0.0510981 secs] [Times: user=0.86 sys=0.02, real=0.05 secs]
2012-02-02T09:09:29.647-0600: 61979.413: [GC 61979.413: [ParNew: 914278K->2750K(1023552K), 0.0141793 secs] 1143292K->232548K(3299264K), 0.0152009 secs] [Times: user=0.44 sys=0.00, real=0.02 secs]
2012-02-02T09:09:37.432-0600: 61987.198: [GC 61987.199: [ParNew: 912574K->2926K(1023552K), 0.0145958 secs] 1142372K->232786K(3299264K), 0.0156003 secs] [Times: user=0.45 sys=0.00, real=0.02 secs]
2012-02-02T09:09:45.213-0600: 61994.979: [GC 61994.980: [ParNew: 912750K->3264K(1023552K), 0.0133675 secs] 1142610K->233225K(3299264K), 0.0144490 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2012-02-02T09:09:50.709-0600: 62000.474: [GC 62000.475: [ParNew: 913088K->2920K(1023552K), 0.0169685 secs] 1143049K->232971K(3299264K), 0.0179272 secs] [Times: user=0.31 sys=0.00, real=0.02 secs]
2012-02-02T09:09:57.938-0600: 62007.704: [GC 62007.705: [ParNew: 912744K->4014K(1023552K), 0.0367154 secs] 1142795K->234128K(3299264K), 0.0377584 secs] [Times: user=0.36 sys=0.00, real=0.04 secs]
2012-02-02T09:10:04.134-0600: 62013.899: [GC 62013.899: [ParNew: 913838K->4306K(1023552K), 0.0179481 secs] 1143952K->234465K(3299264K), 0.0188570 secs] [Times: user=0.28 sys=0.00, real=0.02 secs]
2012-02-02T09:10:10.223-0600: 62019.988: [GC 62019.989: [ParNew: 914130K->4890K(1023552K), 0.0258524 secs] 1144289K->235119K(3299264K), 0.0267208 secs] [Times: user=0.37 sys=0.00, real=0.03 secs]
2012-02-02T09:10:19.538-0600: 62029.304: [GC 62029.305: [ParNew: 914714K->106275K(1023552K), 0.4447269 secs] 1144943K->336555K(3299264K), 0.4456875 secs] [Times: user=3.46 sys=0.14, real=0.45 secs]
2012-02-02T09:10:28.095-0600: 62037.860: [GC 62037.861: [ParNew: 1016099K->113728K(1023552K), 2.3496060 secs] 1246379K->627933K(3299264K), 2.3506549 secs] [Times: user=11.48 sys=1.48, real=2.35 secs]
2012-02-02T09:10:40.675-0600: 62050.440: [GC 62050.441: [ParNew: 1023552K->113728K(1023552K), 1.6754442 secs] 1537757K->946051K(3299264K), 1.6765925 secs] [Times: user=36.85 sys=0.37, real=1.68 secs]
2012-02-02T09:10:52.540-0600: 62062.306: [GC 62062.306: [ParNew: 1023552K->113728K(1023552K), 1.6465925 secs] 1855875K->1262979K(3299264K), 1.6475185 secs] [Times: user=38.24 sys=0.61, real=1.65 secs]
2012-02-02T09:10:54.190-0600: 62063.955: [GC [1 CMS-initial-mark: 1149251K(2275712K)] 1276188K(3299264K), 0.2430027 secs] [Times: user=0.25 sys=0.00, real=0.24 secs]
2012-02-02T09:10:54.434-0600: 62064.199: [CMS-concurrent-mark-start]
2012-02-02T09:10:55.375-0600: 62065.140: [CMS-concurrent-mark: 0.941/0.941 secs] [Times: user=10.44 sys=0.25, real=0.94 secs]
2012-02-02T09:10:55.375-0600: 62065.140: [CMS-concurrent-preclean-start]
2012-02-02T09:10:56.231-0600: 62065.996: [CMS-concurrent-preclean: 0.836/0.856 secs] [Times: user=1.78 sys=0.00, real=0.86 secs]
2012-02-02T09:10:56.231-0600: 62065.996: [CMS-concurrent-abortable-preclean-start]
2012-02-02T09:11:00.905-0600: 62070.669: [CMS-concurrent-abortable-preclean: 4.665/4.673 secs] [Times: user=9.75 sys=0.03, real=4.67 secs]
2012-02-02T09:11:00.906-0600: 62070.671: [GC[YG occupancy: 626192 K (1023552 K)]62070.672: [Rescan (parallel) , 0.4593515 secs]62071.132: [weak refs processing, 0.8817154 secs]62072.014: [scrub string table, 0.0006451 secs] [1 CMS-remark: 1149251K(2275712K)] 1775444K(3299264K), 1.7319597 secs] [Times: user=15.60 sys=0.00, real=1.73 secs]
2012-02-02T09:11:02.639-0600: 62072.404: [CMS-concurrent-sweep-start]
2012-02-02T09:11:04.088-0600: 62073.853: [CMS-concurrent-sweep: 1.443/1.449 secs] [Times: user=4.63 sys=0.05, real=1.45 secs]
2012-02-02T09:11:04.088-0600: 62073.853: [CMS-concurrent-reset-start]
2012-02-02T09:11:04.101-0600: 62073.866: [CMS-concurrent-reset: 0.013/0.013 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2012-02-02T09:11:09.312-0600: 62079.077: [GC 62079.077: [ParNew: 1023552K->113728K(1023552K), 1.6178150 secs] 1587225K->979876K(3299264K), 1.6187922 secs] [Times: user=37.58 sys=0.11, real=1.62 secs]
2012-02-02T09:11:21.172-0600: 62090.937: [GC 62090.937: [ParNew: 1023552K->113728K(1023552K), 1.5327921 secs] 1889700K->1243425K(3299264K), 1.5335996 secs] [Times: user=34.29 sys=0.14, real=1.53 secs]
2012-02-02T09:11:33.305-0600: 62103.070: [GC 62103.070: [ParNew: 1023552K->113728K(1023552K), 1.6349678 secs] 2153249K->1475423K(3299264K), 1.6361176 secs] [Times: user=36.40 sys=0.17, real=1.64 secs]
2012-02-02T09:11:45.480-0600: 62115.245: [GC 62115.246: [ParNew: 1023552K->113728K(1023552K), 1.6566701 secs] 2385247K->1744596K(3299264K), 1.6579350 secs] [Times: user=37.52 sys=0.25, real=1.66 secs]
2012-02-02T09:11:47.140-0600: 62116.905: [GC [1 CMS-initial-mark: 1630868K(2275712K)] 1744677K(3299264K), 0.2085259 secs] [Times: user=0.22 sys=0.00, real=0.21 secs]
2012-02-02T09:11:47.350-0600: 62117.115: [CMS-concurrent-mark-start]
2012-02-02T09:11:48.481-0600: 62118.245: [CMS-concurrent-mark: 1.130/1.130 secs] [Times: user=12.03 sys=0.02, real=1.13 secs]
2012-02-02T09:11:48.481-0600: 62118.245: [CMS-concurrent-preclean-start]
2012-02-02T09:11:49.191-0600: 62118.955: [CMS-concurrent-preclean: 0.702/0.710 secs] [Times: user=1.51 sys=0.00, real=0.71 secs]
2012-02-02T09:11:49.191-0600: 62118.955: [CMS-concurrent-abortable-preclean-start]
2012-02-02T09:11:53.016-0600: 62122.780: [CMS-concurrent-abortable-preclean: 3.819/3.825 secs] [Times: user=8.44 sys=0.05, real=3.83 secs]
2012-02-02T09:11:53.017-0600: 62122.781: [GC[YG occupancy: 637511 K (1023552 K)]62122.782: [Rescan (parallel) , 0.6066033 secs]62123.389: [weak refs processing, 0.8883109 secs]62124.278: [scrub string table, 0.0005759 secs] [1 CMS-remark: 1630868K(2275712K)] 2268379K(3299264K), 1.8602409 secs] [Times: user=16.88 sys=0.02, real=1.86 secs]
2012-02-02T09:11:54.878-0600: 62124.643: [CMS-concurrent-sweep-start]
2012-02-02T09:11:57.491-0600: 62127.255: [CMS-concurrent-sweep: 2.610/2.613 secs] [Times: user=7.57 sys=0.16, real=2.61 secs]
2012-02-02T09:11:57.491-0600: 62127.256: [CMS-concurrent-reset-start]
2012-02-02T09:11:57.498-0600: 62127.262: [CMS-concurrent-reset: 0.007/0.007 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2012-02-02T09:12:01.137-0600: 62130.902: [GC 62130.902: [ParNew: 1023552K->113728K(1023552K), 1.4807261 secs] 1678232K->1015472K(3299264K), 1.4816808 secs] [Times: user=26.47 sys=0.00, real=1.48 secs]
2012-02-02T09:12:13.113-0600: 62142.877: [GC 62142.878: [ParNew: 1023552K->113728K(1023552K), 1.5627711 secs] 1925296K->1258829K(3299264K), 1.5635313 secs] [Times: user=34.26 sys=0.00, real=1.56 secs]
2012-02-02T09:12:25.167-0600: 62154.931: [GC 62154.932: [ParNew: 1023552K->113728K(1023552K), 1.5751999 secs] 2168653K->1510600K(3299264K), 1.5762551 secs] [Times: user=31.75 sys=0.00, real=1.58 secs]
2012-02-02T09:12:36.964-0600: 62166.729: [GC 62166.730: [ParNew: 1023552K->113728K(1023552K), 1.6056739 secs] 2420424K->1768193K(3299264K), 1.6065643 secs] [Times: user=36.02 sys=0.08, real=1.61 secs]
2012-02-02T09:12:38.573-0600: 62168.337: [GC [1 CMS-initial-mark: 1654465K(2275712K)] 1768195K(3299264K), 0.1967081 secs] [Times: user=0.20 sys=0.00, real=0.20 secs]
2012-02-02T09:12:38.770-0600: 62168.535: [CMS-concurrent-mark-start]
2012-02-02T09:12:39.611-0600: 62169.376: [CMS-concurrent-mark: 0.841/0.841 secs] [Times: user=9.24 sys=0.00, real=0.84 secs]
2012-02-02T09:12:39.611-0600: 62169.376: [CMS-concurrent-preclean-start]
2012-02-02T09:12:40.036-0600: 62169.801: [CMS-concurrent-preclean: 0.419/0.425 secs] [Times: user=0.84 sys=0.02, real=0.42 secs]
2012-02-02T09:12:40.036-0600: 62169.801: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2012-02-02T09:12:46.001-0600: 62175.765: [CMS-concurrent-abortable-preclean: 5.955/5.964 secs] [Times: user=12.00 sys=0.08, real=5.96 secs]
2012-02-02T09:12:46.003-0600: 62175.767: [GC[YG occupancy: 770945 K (1023552 K)]62175.767: [Rescan (parallel) , 0.7886971 secs]62176.557: [weak refs processing, 0.8179398 secs]62177.375: [scrub string table, 0.0008280 secs] [1 CMS-remark: 1654465K(2275712K)] 2425410K(3299264K), 1.8989263 secs] [Times: user=25.40 sys=0.02, real=1.90 secs]
2012-02-02T09:12:47.903-0600: 62177.667: [CMS-concurrent-sweep-start]
2012-02-02T09:12:50.676-0600: 62180.440: [CMS-concurrent-sweep: 2.772/2.774 secs] [Times: user=8.20 sys=0.06, real=2.77 secs]
2012-02-02T09:12:50.676-0600: 62180.441: [CMS-concurrent-reset-start]
2012-02-02T09:12:50.696-0600: 62180.460: [CMS-concurrent-reset: 0.019/0.019 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2012-02-02T09:12:52.272-0600: 62182.037: [GC 62182.037: [ParNew: 1023552K->113728K(1023552K), 1.5954584 secs] 1584704K->951316K(3299264K), 1.5963712 secs] [Times: user=34.26 sys=0.00, real=1.60 secs]
2012-02-02T09:13:34.402-0600: 62224.165: [GC 62224.166: [ParNew: 1023552K->113728K(1023552K), 0.9755846 secs] 1861140K->1084527K(3299264K), 0.9767018 secs] [Times: user=19.19 sys=0.02, real=0.98 secs]
2012-02-02T09:16:04.628-0600: 62374.391: [GC 62374.391: [ParNew: 1023552K->113517K(1023552K), 0.1955335 secs] 1994351K->1173747K(3299264K), 0.1966082 secs] [Times: user=2.12 sys=0.05, real=0.20 secs]
2012-02-02T09:18:44.395-0600: 62534.156: [GC 62534.157: [ParNew: 1023341K->19351K(1023552K), 0.0506139 secs] 2083571K->1079581K(3299264K), 0.0517891 secs] [Times: user=0.37 sys=0.00, real=0.05 secs]
2012-02-02T09:21:20.270-0600: 62690.030: [GC 62690.030: [ParNew: 929175K->8239K(1023552K), 0.0405564 secs] 1989405K->1068470K(3299264K), 0.0417057 secs] [Times: user=0.76 sys=0.00, real=0.04 secs]
2012-02-02T09:23:54.367-0600: 62844.125: [GC 62844.126: [ParNew: 918063K->5778K(1023552K), 0.0497713 secs] 1978294K->1066008K(3299264K), 0.0508845 secs] [Times: user=0.58 sys=0.00, real=0.05 secs]
2012-02-02T09:26:33.522-0600: 63003.279: [GC 63003.279: [ParNew: 915602K->5421K(1023552K), 0.0524503 secs] 1975832K->1065651K(3299264K), 0.0536713 secs] [Times: user=0.59 sys=0.02, real=0.06 secs]
2012-02-02T09:29:13.270-0600: 63163.025: [GC 63163.026: [ParNew: 915245K->5390K(1023552K), 0.0317213 secs] 1975475K->1065620K(3299264K), 0.0330578 secs] [Times: user=0.25 sys=0.00, real=0.03 secs]
2012-02-02T09:31:58.001-0600: 63327.754: [GC 63327.755: [ParNew: 915214K->5524K(1023552K), 0.0617393 secs] 1975444K->1065894K(3299264K), 0.0629008 secs] [Times: user=1.05 sys=0.00, real=0.06 secs]
2012-02-02T09:34:43.454-0600: 63493.206: [GC 63493.207: [ParNew: 915348K->5600K(1023552K), 0.0376009 secs] 1975718K->1066058K(3299264K), 0.0388287 secs] [Times: user=0.36 sys=0.00, real=0.04 secs]
2012-02-02T09:37:15.601-0600: 63645.352: [GC 63645.353: [ParNew: 915424K->6656K(1023552K), 0.0273727 secs] 1975882K->1067278K(3299264K), 0.0285435 secs] [Times: user=0.56 sys=0.00, real=0.03 secs]
2012-02-02T09:39:41.873-0600: 63791.623: [GC 63791.623: [ParNew: 916480K->8588K(1023552K), 0.0239936 secs] 1977102K->1069403K(3299264K), 0.0252618 secs] [Times: user=0.75 sys=0.00, real=0.03 secs]
2012-02-02T09:42:02.276-0600: 63932.025: [GC 63932.025: [ParNew: 918412K->8408K(1023552K), 0.0259162 secs] 1979227K->1069313K(3299264K), 0.0269622 secs] [Times: user=0.34 sys=0.02, real=0.03 secs]
2012-02-02T09:44:22.990-0600: 64072.737: [GC 64072.738: [ParNew: 918232K->8777K(1023552K), 0.0208236 secs] 1979137K->1069755K(3299264K), 0.0220036 secs] [Times: user=0.41 sys=0.00, real=0.02 secs]

Today i used
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:-CMSConcurrentMTEnabled
-XX:+CMSScavengeBeforeRemark
and had the same problem with all 3 storage enabled nodes simultaneously when I closed the Extend app. The GC logs from those storage nodes were nearly identical with each showing three "abort preclean" messages like below before eventually stabilizing again. CPU usage for the storage node JVMs was up a bit on the machine at the time then eventually came down. Extend proxy node CPU usage never goes up very high and GC times are always good there.
Could it be because the Extend app has many CQCs running and when the app disconnects without shutting them down properly their results queue up in the storage enabled nodes? Something the Extend proxy can't pass along is getting added to old space?
9:04:29.313-0600: 34201.081: [GC 34201.081: [ParNew: 914605K->4950K(1023552K), 0.0469807 secs] 1057777K->148216K(3299264K), 0.0480525 secs] [Times: user=0.78 sys=0.03, real=0.05 secs]
9:04:35.742-0600: 34207.509: [GC 34207.509: [ParNew: 914774K->4559K(1023552K), 0.0317232 secs] 1058040K->147868K(3299264K), 0.0324390 secs] [Times: user=0.48 sys=0.00, real=0.03 secs]
9:04:44.806-0600: 34216.573: [GC 34216.574: [ParNew: 914383K->113728K(1023552K), 1.4924395 secs] 1057692K->348748K(3299264K), 1.4934226 secs] [Times: user=7.52 sys=0.81, real=1.49 secs]
9:04:56.498-0600: 34228.265: [GC 34228.265: [ParNew: 1023552K->113728K(1023552K), 1.3332836 secs] 1258572K->634502K(3299264K), 1.3341394 secs] [Times: user=28.58 sys=0.44, real=1.34 secs]
9:05:09.845-0600: 34241.612: [GC 34241.612: [ParNew: 1023552K->113728K(1023552K), 1.5617413 secs] 1544326K->920121K(3299264K), 1.5628521 secs] [Times: user=33.43 sys=0.34, real=1.56 secs]
9:05:24.569-0600: 34256.335: [GC 34256.336: [ParNew: 1023552K->113728K(1023552K), 2.3253881 secs] 1829945K->1186221K(3299264K), 2.3263253 secs] [Times: user=33.95 sys=0.48, real=2.33 secs]
9:05:39.812-0600: 34271.579: [GC 34271.580: [ParNew: 1023552K->113728K(1023552K), 1.6817535 secs] 2096045K->1453026K(3299264K), 1.6827555 secs] [Times: user=33.60 sys=0.20, real=1.68 secs]
9:05:41.498-0600: 34273.264: [GC [1 CMS-initial-mark: 1339298K(2275712K)] 1453026K(3299264K), 0.2348391 secs] [Times: user=0.22 sys=0.02, real=0.24 secs]
9:05:41.733-0600: 34273.500: [CMS-concurrent-mark-start]
9:05:44.732-0600: 34276.498: [CMS-concurrent-mark: 2.999/2.999 secs] [Times: user=6.49 sys=0.05, real=3.00 secs]
9:05:44.732-0600: 34276.498: [CMS-concurrent-preclean-start]
9:05:45.839-0600: 34277.606: [CMS-concurrent-preclean: 1.095/1.107 secs] [Times: user=2.25 sys=0.00, real=1.11 secs]
9:05:45.839-0600: 34277.606: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 9:05:50.866-0600: 34282.633: [CMS-concurrent-abortable-preclean: 5.021/5.027 secs] [Times: user=10.14 sys=0.05, real=5.03 secs]
9:05:50.868-0600: 34282.634: [GC[YG occupancy: 809255 K (1023552 K)]9:05:50.868-0600: 34282.635: [GC 34282.635: [ParNew: 809255K->113728K(1023552K), 1.0828740 secs] 2148553K->1611793K(3299264K), 1.0835767 secs] [Times: user=19.41 sys=0.06, real=1.08 secs]
34283.719: [Rescan (parallel) , 0.1498045 secs]34283.869: [weak refs processing, 1.0258216 secs]34284.896: [scrub string table, 0.0006202 secs] [1 CMS-remark: 1498065K(2275712K)] 1611793K(3299264K), 2.7794183 secs] [Times: user=24.56 sys=0.06, real=2.78 secs]
9:05:53.648-0600: 34285.415: [CMS-concurrent-sweep-start]
9:05:56.112-0600: 34287.879: [CMS-concurrent-sweep: 2.464/2.464 secs] [Times: user=7.38 sys=0.03, real=2.46 secs]
9:05:56.112-0600: 34287.879: [CMS-concurrent-reset-start]
9:05:56.120-0600: 34287.887: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
9:06:06.204-0600: 34297.970: [GC 34297.971: [ParNew: 1023552K->113728K(1023552K), 1.5001175 secs] 1845197K->1139597K(3299264K), 1.5011371 secs] [Times: user=31.31 sys=0.00, real=1.50 secs]
9:06:20.044-0600: 34311.810: [GC 34311.811: [ParNew: 1023552K->113728K(1023552K), 1.6438234 secs] 2049421K->1389674K(3299264K), 1.6446616 secs] [Times: user=36.97 sys=0.03, real=1.64 secs]
9:06:34.063-0600: 34325.829: [GC 34325.830: [ParNew: 1023552K->113728K(1023552K), 2.0298132 secs] 2299498K->1648212K(3299264K), 2.0308665 secs] [Times: user=34.10 sys=0.03, real=2.03 secs]
9:06:36.096-0600: 34327.862: [GC [1 CMS-initial-mark: 1534484K(2275712K)] 1648282K(3299264K), 0.2109338 secs] [Times: user=0.22 sys=0.00, real=0.21 secs]
9:06:36.308-0600: 34328.074: [CMS-concurrent-mark-start]
9:06:39.105-0600: 34330.871: [CMS-concurrent-mark: 2.797/2.797 secs] [Times: user=6.02 sys=0.00, real=2.80 secs]
9:06:39.105-0600: 34330.871: [CMS-concurrent-preclean-start]
9:06:39.795-0600: 34331.561: [CMS-concurrent-preclean: 0.686/0.690 secs] [Times: user=1.40 sys=0.00, real=0.69 secs]
9:06:39.795-0600: 34331.561: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 9:06:46.623-0600: 34338.389: [CMS-concurrent-abortable-preclean: 6.820/6.828 secs] [Times: user=13.85 sys=0.03, real=6.83 secs]
9:06:46.624-0600: 34338.390: [GC[YG occupancy: 891686 K (1023552 K)]9:06:46.624-0600: 34338.391: [GC 34338.391: [ParNew: 891686K->113728K(1023552K), 1.2772255 secs] 2426171K->1846761K(3299264K), 1.2780695 secs] [Times: user=25.66 sys=0.11, real=1.28 secs]
34339.669: [Rescan (parallel) , 0.1456700 secs]34339.815: [weak refs processing, 0.6135547 secs]34340.431: [scrub string table, 0.0005247 secs] [1 CMS-remark: 1733033K(2275712K)] 1846761K(3299264K), 2.3580749 secs] [Times: user=30.44 sys=0.11, real=2.36 secs]
9:06:48.983-0600: 34340.749: [CMS-concurrent-sweep-start]
9:06:51.895-0600: 34343.660: [CMS-concurrent-sweep: 2.911/2.911 secs] [Times: user=7.71 sys=0.06, real=2.91 secs]
9:06:51.895-0600: 34343.660: [CMS-concurrent-reset-start]
9:06:51.941-0600: 34343.707: [CMS-concurrent-reset: 0.047/0.047 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
9:07:02.441-0600: 34354.207: [GC 34354.207: [ParNew: 1023552K->113728K(1023552K), 1.5969685 secs] 1805173K->1136718K(3299264K), 1.5980247 secs] [Times: user=33.85 sys=0.00, real=1.60 secs]
9:07:18.155-0600: 34369.921: [GC 34369.922: [ParNew: 1023552K->113728K(1023552K), 2.2864633 secs] 2046542K->1381538K(3299264K), 2.2874742 secs] [Times: user=33.74 sys=0.02, real=2.29 secs]
9:07:33.522-0600: 34385.288: [GC 34385.288: [ParNew: 1023552K->113728K(1023552K), 1.6890223 secs] 2291362K->1632772K(3299264K), 1.6898157 secs] [Times: user=35.18 sys=0.01, real=1.69 secs]
9:07:35.213-0600: 34386.979: [GC [1 CMS-initial-mark: 1519044K(2275712K)] 1632799K(3299264K), 0.2011283 secs] [Times: user=0.20 sys=0.00, real=0.20 secs]
9:07:35.415-0600: 34387.181: [CMS-concurrent-mark-start]
9:07:37.810-0600: 34389.576: [CMS-concurrent-mark: 2.395/2.395 secs] [Times: user=5.63 sys=0.00, real=2.39 secs]
9:07:37.810-0600: 34389.576: [CMS-concurrent-preclean-start]
9:07:38.623-0600: 34390.389: [CMS-concurrent-preclean: 0.808/0.813 secs] [Times: user=1.70 sys=0.05, real=0.81 secs]
9:07:38.623-0600: 34390.389: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 9:07:45.370-0600: 34397.135: [CMS-concurrent-abortable-preclean: 6.742/6.747 secs] [Times: user=13.76 sys=0.11, real=6.75 secs]
9:07:45.371-0600: 34397.136: [GC[YG occupancy: 950924 K (1023552 K)]9:07:45.372-0600: 34397.137: [GC 34397.137: [ParNew: 950924K->113728K(1023552K), 1.3829715 secs] 2469968K->1894060K(3299264K), 1.3840623 secs] [Times: user=31.28 sys=0.00, real=1.38 secs]
34398.522: [Rescan (parallel) , 0.1888848 secs]34398.711: [weak refs processing, 0.3561537 secs]34399.068: [scrub string table, 0.0004774 secs] [1 CMS-remark: 1780332K(2275712K)] 1894060K(3299264K), 2.1091730 secs] [Times: user=36.55 sys=0.00, real=2.11 secs]
9:07:47.482-0600: 34399.247: [CMS-concurrent-sweep-start]
9:07:50.500-0600: 34402.265: [CMS-concurrent-sweep: 3.017/3.018 secs] [Times: user=8.63 sys=0.33, real=3.02 secs]
9:07:50.500-0600: 34402.265: [CMS-concurrent-reset-start]
9:07:50.507-0600: 34402.273: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
9:09:58.395-0600: 34530.160: [GC 34530.160: [ParNew: 1023552K->113438K(1023552K), 0.4065570 secs] 1764220K->964374K(3299264K), 0.4077345 secs] [Times: user=4.63 sys=0.03, real=0.41 secs]
9:12:06.040-0600: 34657.802: [GC 34657.803: [ParNew: 1023262K->27778K(1023552K), 0.0334396 secs] 1874198K->878714K(3299264K), 0.0345056 secs]-Andrew

Garbage Collector pausing my Server for MINUTES at a time

Garbage Collection pauses are really crippling my server. Here is a grep of Full GCs from the log file. (The lines following each Full GC have a timestamp, yymmdd:hhmmss.sss)
[Full GC 876186K->169083K(222400K), 28.4152794 secs]
011120:004809.413:
[Full GC 883850K->214246K(259008K), 68.2294388 secs]
011120:033449.558: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 209.2 MB/900.0 MB (Free: 43.7 MB, Allocated: 252.9 MB)
[Full GC 881484K->300898K(347200K), 73.3787317 secs]
011120:045927.680: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 293.8 MB/900.0 MB (Free: 45.2 MB, Allocated: 339.1 MB)
[Full GC 886225K->269194K(317824K), 79.8212023 secs]
011120:055800.166: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 262.9 MB/900.0 MB (Free: 47.5 MB, Allocated: 310.4 MB)
[Full GC 890405K->261774K(338112K), 104.6192760 secs]
011120:064350.312: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 255.6 MB/900.0 MB (Free: 74.5 MB, Allocated: 330.2 MB)
[Full GC 893451K->306762K(361024K), 134.4052782 secs]
011120:073523.335: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 299.6 MB/900.0 MB (Free: 53.0 MB, Allocated: 352.6 MB)
[Full GC 896034K->286275K(350016K), 103.4689894 secs]
011120:081636.710: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 279.6 MB/900.0 MB (Free: 62.2 MB, Allocated: 341.8 MB)
[Full GC 882755K->260923K(310272K), 129.6787672 secs]
011120:093812.612: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 254.8 MB/900.0 MB (Free: 48.2 MB, Allocated: 303.0 MB)
You can see that between 12:48am and 9:38am, the Server is pausing for a combined total of TWELVE MINUTES! Often for 2 whole minutes at a time, during which the Server is completely unresponsive.
I have read Sun's GC tuning guide at http://java.sun.com/docs/hotspot/gc/index.html but it is not much help:
It says "Pauses can be minimized by using a small young generation and incremental collection, at the expense of throughput."
Well, I AM using incremental collection, and the young generation is the default size, which is quite small I believe. (32 MB max?)
It also says "Unless you have problems with pauses, try granting as much memory as possible to the JVM. The default size (64MB) is often too small."
Well I DO have problems with pauses, but I HAVE to allocate a ton of memory. The Server must support hundreds of simultaneous users, each with hundreds of K's of personal, dynamic, temporary data.
It also says "Unless you find problems with excessive major collection or pause times, grant plenty of memory to the young generation. The default MaxNewSize (32MB) is generally too small."
I AM having problems with excessive major collections and pause times, so according to this I should NOT grant more memory to young generation (eden).
What am I left with? Nothing! I am totally out to lunch here. The box this is running on is a dual P3-600 with a gig of RAM and Red Hat 6.2, with Sun JRE 1.3.1_02. My startup command line options are:
java -server -Xincgc -verbose:gc -Xms256M -Xmx900M
What should I do? The Tuning guide essentially says to screw around with generation sizes, but none of its specific tips seem to apply to me! This is a production server and I am worried about making things worse by playing around, so I was hoping someone out there might have some experience with this and concrete advice before I just start messing around with the memory map. Should I increase the young generation size? Decrease it? Something else altogether? HELP!

With default settings, I am getting horrible performance, under peak load, 25%-50% of server time is being spent with GC, approx. 11,000 minor GCs per hour! And a Full GC every hour or two which takes up several minutes. On average, I am losing 13.09 minutes per hour due to minor GC, and 54.5 seconds to major GC.
I tried your settings, Chuck, and they seemed to be working beautifully at first... minor GCs were now occuring once every few minutes for a couple seconds, instead of 3 times per second for a tenth of a second. Minor GC were now taking 1.06 minutes per hour instead of 13.09, more than a 1300% improvement!
HOWEVER...
After 7 hours, the Server started getting REALLY slow for no apparent reason. It seemed almost as if reading from sockets was suddenly becoming extremely time intensive. Then I got a HUGE minor GC that took 25.6 seconds. [GC 507410K->478034K(551296K), 25.6328459 secs], then a couple minutes later, the Server CRASHED with this output:
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x40573aa7
Function name=malloc
Library=/lib/libc.so.6
Current Java thread:
Dynamic libraries:
011121:231738.194: Search ( pizzala, lzh ) returned 50 results (28 ms)
08048000-0804c000 r-xp 00000000 08:06 376904 /usr/local/java/jre1.3.1_01/bin/i386/native_threads/java
0804c000-0804d000 rw-p 00003000 08:06 376904 /usr/local/java/jre1.3.1_01/bin/i386/native_threads/java
40000000-40013000 r-xp 00000000 08:06 311298 /lib/ld-2.1.3.so
40013000-40014000 rw-p 00012000 08:06 311298 /lib/ld-2.1.3.so
40015000-40016000 r--p 00000000 08:06 2424834 /usr/share/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES
40016000-40017000 r--p 00000000 08:06 2408452 /usr/share/locale/en_US/LC_MONETARY
40017000-40018000 r--p 00000000 08:06 2408454 /usr/share/locale/en_US/LC_TIME
40018000-40023000 r-xp 00000000 08:06 311344 /lib/libpthread-0.8.so
40023000-4002a000 rw-p 0000a000 08:06 311344 /lib/libpthread-0.8.so
4002b000-40034000 r-xp 00000000 08:06 1081477 /usr/local/java/jre1.3.1_01/lib/i386/native_threads/libhpi.so
40034000-40035000 rw-p 00008000 08:06 1081477 /usr/local/java/jre1.3.1_01/lib/i386/native_threads/libhpi.so
40035000-403b0000 r-xp 00000000 08:06 2769091 /usr/local/java/jre1.3.1_01/lib/i386/server/libjvm.so
403b0000-403b1000 ---p 0037b000 08:06 2769091 /usr/local/java/jre1.3.1_01/lib/i386/server/libjvm.so
403b1000-404fd000 rw-p 0037b000 08:06 2769091 /usr/local/java/jre1.3.1_01/lib/i386/server/libjvm.so
40515000-40517000 r-xp 00000000 08:06 311314 /lib/libdl-2.1.3.so
40517000-40519000 rw-p 00001000 08:06 311314 /lib/libdl-2.1.3.so
4051a000-40607000 r-xp 00000000 08:06 311305 /lib/libc-2.1.3.so
40607000-4060b000 rw-p 000ec000 08:06 311305 /lib/libc-2.1.3.so
4060f000-40621000 r-xp 00000000 08:06 311318 /lib/libnsl-2.1.3.so
40621000-40623000 rw-p 00011000 08:06 311318 /lib/libnsl-2.1.3.so
40625000-40641000 r-xp 00000000 08:06 311316 /lib/libm-2.1.3.so
40641000-40642000 rw-p 0001b000 08:06 311316 /lib/libm-2.1.3.so
40642000-40676000 r-xp 00000000 08:06 704575 /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so
40676000-40682000 rw-p 00033000 08:06 704575 /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so
40684000-40695000 r-xp 00000000 08:06 1081479 /usr/local/java/jre1.3.1_01/lib/i386/libverify.so
40695000-40697000 rw-p 00010000 08:06 1081479 /usr/local/java/jre1.3.1_01/lib/i386/libverify.so
40697000-406b8000 r-xp 00000000 08:06 1081480 /usr/local/java/jre1.3.1_01/lib/i386/libjava.so
406b8000-406ba000 rw-p 00020000 08:06 1081480 /usr/local/java/jre1.3.1_01/lib/i386/libjava.so
406bb000-406cf000 r-xp 00000000 08:06 1081481 /usr/local/java/jre1.3.1_01/lib/i386/libzip.so
406cf000-406d2000 rw-p 00013000 08:06 1081481 /usr/local/java/jre1.3.1_01/lib/i386/libzip.so
406d2000-41400000 r--s 00000000 08:06 1081510 /usr/local/java/jre1.3.1_01/lib/rt.jar
4142d000-416d2000 r--s 00000000 08:06 1081511 /usr/local/java/jre1.3.1_01/lib/i18n.jar
416d2000-416e8000 r--s 00000000 08:06 1081498 /usr/local/java/jre1.3.1_01/lib/sunrsasign.jar
77a9f000-77ab5000 r--p 00000000 08:06 2408451 /usr/share/locale/en_US/LC_CTYPE
77ab5000-77abd000 r--p 00000000 08:06 2408450 /usr/share/locale/en_US/LC_COLLATE
77abd000-77abe000 r--p 00000000 08:06 2408453 /usr/share/locale/en_US/LC_NUMERIC
77abe000-77abf000 r-xp 00000000 08:06 1622117 /usr/lib/gconv/ISO8859-1.so
77abf000-77ac0000 rw-p 00000000 08:06 1622117 /usr/lib/gconv/ISO8859-1.so
77ac1000-77ac9000 r-xp 00000000 08:06 311336 /lib/libnss_files-2.1.3.so
77ac9000-77aca000 rw-p 00007000 08:06 311336 /lib/libnss_files-2.1.3.so
77b37000-77b40000 r-xp 00000000 08:06 1081484 /usr/local/java/jre1.3.1_01/lib/i386/libnet.so
77b40000-77b41000 rw-p 00008000 08:06 1081484 /usr/local/java/jre1.3.1_01/lib/i386/libnet.so
77b41000-77b45000 r-xp 00000000 08:06 1081513 /usr/local/java/jre1.3.1_01/lib/i386/libNBIO.so
77b45000-77b46000 rw-p 00003000 08:06 1081513 /usr/local/java/jre1.3.1_01/lib/i386/libNBIO.so
77b46000-77b4f000 r-xp 00000000 08:06 311342 /lib/libnss_nisplus-2.1.3.so
77b4f000-77b51000 rw-p 00008000 08:06 311342 /lib/libnss_nisplus-2.1.3.so
77b51000-77b59000 r-xp 00000000 08:06 311340 /lib/libnss_nis-2.1.3.so
77b59000-77b5b000 rw-p 00007000 08:06 311340 /lib/libnss_nis-2.1.3.so
77b5b000-77b5e000 r-xp 00000000 08:06 311334 /lib/libnss_dns-2.1.3.so
77b5e000-77b5f000 rw-p 00002000 08:06 311334 /lib/libnss_dns-2.1.3.so
77b5f000-77b6b000 r-xp 00000000 08:06 311346 /lib/libresolv-2.1.3.so
77b6b000-77b6c000 rw-p 0000b000 08:06 311346 /lib/libresolv-2.1.3.so
Local Time = Wed Nov 21 23:17:38 2001
Elapsed Time = 26581
# The exception above was detected in native code outside the VM
# Java VM: Java HotSpot(TM) Server VM (1.3.1_01 mixed mode)
# An error report file has been saved as hs_err_pid20055.log.
# Please refer to the file for further information.
Yummy. Signal 11's really make my day fun :(. I tried restarting the Server, but it crashed again in the same way 15 minutes later. Then I switched back to default settings and it has run fine all weekend.
The Sig11 makes me think either our C lib is buggy (our Linux distro is getting a bit old), or we have a physical memory problem. Odd that it only shows up when I tried your new settings though...

Exorbitant Garbage Collection Times

Once a day, at around 4-4:30am, our application experiences a single Full GC with taking an exorbitant amount of time. A Full GC kicks in and pauses all threads for 5 to 30 minutes. Full GC's regularly occur at other times and complete in 5-20 seconds. Anyone seen this or any have any ideas about this?
Example of normal GC that occurs frequently throughout the day:
[Full GC[Unloading class $Proxy136]
4632722K->1616289K(6141952K), 14.9400110 secs]
Example of ridiculously long GC that occurs daily at 4-4:30am:
[Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor455]
5456231K->2017269K(6141952K), 492.0402160 secs]
Here's our VM params:
-server -XX:-OmitStackTraceInFastThrow -Xincgc -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:MaxGCPauseMillis=20000 -XX:+DisableExplicitGC -XX:NewSize=1500m -XX:SurvivorRatio=18 -verbose:gc
-Xms6000M -Xmx6000M
We're running Java 1.6 on Linux, Redhat 3.

Just experienced a 498 second (yikes!) garbage collection pause. The entry from the log is below (we added -XX:+PrintGCTimeStamps -XX:+PrintGCDetails and removed -verbose:gc). I will also send the entire log to the email address you have provided.
Note that we actually reduced our Xmx to 4000M (from 6000M) despite this machine having 8gig physical ram in the hopes that this would prevent any paging, and it seems to have drastically improved the situation - these exorbitant pauses now occur only very seldomly.
This is the full command we're justing to start the JBoss JVM:
/proj/bo/apps/jdk/production/bin/java -Dprogram.name=run.sh -server -XX:-OmitSta
ckTraceInFastThrow -Xincgc -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:MaxGCPauseMillis=20000 -XX:+DisableExplicitGC
-XX:NewSize=1500m -XX:SurvivorRatio=18 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Djava.endorsed.dirs=/var/local/jboss/jboss-c
urrent/lib/endorsed -Dsystem.state=production -Dcom.hb.common.api.util.Environment.app.name=Hermes -Djboss.partition.name=as20.
nyc.hcmny.com -Dorg.jboss.security.SecurityAssociation.ThreadLocal=true -Doracle.net.tns_admin=/var/opt/oracle/etc -Djnp.discov
eryPort=1103 -classpath /var/local/jboss/jboss-current/bin/run.jar:/proj/bo/apps/jdk/production/lib/tools.jar -Xms4000M -Xmx400
0M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8134 -Dcom.sun.management.jmxremote.authenticate=true -Dc
om.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/proj/bo/jmxremote.password org.jboss.Main -
cdefault
This is the exact JRE we're using:
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode)
Here's the 498 second GC pause:
103999.802: [GC 103999.802: [ParNew (promotion failed): 1459200K->1459200K(1459200K), 478.7897900 secs]104478.592: [CMS104483.552: [CMS-concurrent-mark: 5.170/491.280 secs]
(concurrent mode failure)[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor605]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor866]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor870]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor887]
[Unloading class sun.reflect.GeneratedMethodAccessor267]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor864]
[Unloading class sun.reflect.GeneratedConstructorAccessor148]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor861]
[Unloading class sun.reflect.GeneratedMethodAccessor268]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor606]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor862]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor886]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor706]
[Unloading class sun.reflect.GeneratedConstructorAccessor145]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor865]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor350]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor860]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor868]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor863]
[Unloading class sun.reflect.GeneratedConstructorAccessor147]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor867]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor869]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor185]
[Unloading class sun.reflect.GeneratedConstructorAccessor125]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor871]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor707]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor708]
: 2517456K->2073715K(2560000K), 19.7399910 secs] 3927185K->2073715K(4019200K) icms_dc=5 , 498.5297810 secs]
     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
     at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
     at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1747)
     at java.io.ObjectOutputStream.<init>(ObjectOutputStream.java:226)
21:37:07,931 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=as20.nyc.hcmny.com/163214, BranchQual=, localId=163214] timed out. status=STATUS_ACTIVE
2007-09-12 21:37:08

Garbage collection Java Virtual Machine : Hewlett-Packard Hotspot release 1.3.1.01

"Hi,
I try and understand the mechanism of garbage collection of the Java Virtual Machine : Hewlett-Packard Hotspot release 1.3.1.01.
There is description of this mechanism in the pdf file : "memory management and garbage collection" available at the paragraph "Java performance tuning tutorial" at the page :
http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1607,00.html
Regarding my question :
Below is an extract of the log file of garbage collections. This extract has 2 consecutive garbage collections.
(each begins with "<GC:").
<GC: 1 387875.630047 554 1258496 1 161087488 0 161087488 20119552 0 20119552
334758064 238778016 335544320
46294096 46294096 46399488 5.319209 >
<GC: 5 387926.615209 555 1258496 1 161087488 0 161087488 0 0 20119552
240036512 242217264 335544320
46317184 46317184 46399488 5.206192 >
There are 2 "full garbage collections", one of reason "1" and one of reason "5".
For the first one "Old generation After " =238778016
For the second "Old generation After " =238778016
Thus, "Old generation Before garbage collection" of the second is higher than "Old generation After garbage collection". Why?
I expected all objects to be allocated in the "Eden" space. And therefore I did not expect to s

I agree but my current Hp support is not very good on JVM issues.
Rob Woollen <[email protected]> wrote:
You'd probably be better off asking this question to HP.
-- Rob
Martial wrote:
The object of this mail is the Hewlett-Packard 1.3.1.01 Hotspot JavaVirtual Machine
release and its garbage collection mechanism.
I am interested in the "-Xverbosegc" option for garbage collectionmonitoring.
I have been through the online document :
http://www.hp.com/products1/unix/java/infolibrary/prog_guide/java1_3/hotspot.html#-Xverbosegc
I would like to find out more about the garbage collection mechanismand need
further information to understand the result of the log file generatedwith the
"-Xverbosegc"
For example here is an extract of a garbage collection log file generatedwith
Hewlett-Packard Hotspot Java Virtual Machine. Release 1.3.1.01.
These are 2 consecutive rows of the files :
<GC: 5 385565.750251 543 48 1 161087488 0 161087488 0 0 20119552 264184480255179792
335544320 46118384 46118384 46137344 5.514721 >
<GC: 1 385876.530728 544 1258496 1 161087488 0 161087488 20119552 020119552 334969696
255530640 335544320 46121664 46106304 46137344 6.768760 >
We have 2 full garbage collections, one of Reason 5 and the next oneof Reason
1.
What happened between these 2 garbage collections as we got : "Oldgeneration
After" of row 2 is higher than "Old generation Before" of row 1? Iexpected Objects
to be initially allocated in eden and so we could not get "old generation2modified
between the end of one garbage collection and before the next one.
Could you please clarify this issue and/or give more information aboutgarbage
collection mechanisms with the Hewlett-Packard Hotspot Java VirtualMachine. Release
1.3.1.01.

EWA Alert A- high ratio of full garbage

Hi All,
I have been receiving "RED" EWA report for pure JAVA stack from past 3 weeks. Earlier also I had got it and then the heap memory size was 1024. I had increased to 3GB and then I started getting a "YELLO" EWA report alert. However, from past 3 weeks I have been getting a red EWA report for full Garbage Collection problem.
I had gone throug some of the threads here and found some usefule info like, Paul mentioned in below link:
[EWA Alert A- high ratio of garbage collection|EarlyWatch Alert A - high ratio of full garbage;
How have gone to th dev_server0 file and saw some of the garbage collections there. I am quite unable to get the details mentioned there like:
1.
Mon Dec 12 07:43:12 2011
796020.432: [GC 796020.432: [ParNew: 319604K->28169K(320128K),
0.0588451 secs] 2671315K->2382219K(3116672K), 0.0589239 secs]
2.
Thu Dec 08 10:40:24 2011
461123.817: [Full GC 461123.817: [Tenured
Thu Dec 08 10:40:25 2011
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor150]
[Unloading class sun.reflect.GeneratedMethodAccessor543]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor220]......
I think the first one is the small garbage collections and the latter one is the "Full GC" as the name itself says.---Correct me if I am wrong.
Can any one help me in understanding the above details?
What should I do to remove this from my EWA alert?
Regards,
Faisal

Hi,
I have gone through the server nodes , std_server<n>.out (I have 3 server nodes) and found that on today's date, there are Full GC written in the logs multiple times. But can any one help me in diagnosing this and reaching out to the cause of this problem?
Regards
Faisal

Low Latency Garbage Collection

Hi,
What is the best garbage collector configuration for a low latency system?
Multi-threaded server process running on a multi-core box (say 8 cores). Reading and writing messages to the network, no disk. Want all messages to be processed in as timely a manner as possible wrt their individual latencies. Want to avoid some messages being delayed by a garbage collection pause. What will give me the 'fairest' latency time amongst the messages?
I should add, Java SE 1.6 recent version.
Thanks in advance for you suggestions.
Rupert Smith
Edited by: rupertlssmith on Dec 2, 2009 10:17 AM

The simplest approach is don't create any objects in your critical path. (Or keep them to a minimum) Javolution and Joda-time have a libraries which helps.
You can try the concurrent mark sweep or G1 collectors to see if they help.
You can get the GC low enough that you will see other delays such as the SMC or network jitter or memory bank effects instead.
i.e. If your server is simple enough, you can see more jitter from the hardware/OS than Java itself.
Edited by: Peter__Lawrey on 02-Dec-2009 20:45

Garbage Collection Mysteries

I have an application that consumes huge amounts of heap at time, on the order of 100 MB. I added some code to call System.gc() when I dispose of the window that is responsible for making such huge allocations. My understanding is that System.gc() is supposed to do a full garbage collection, but when I am watching the heap in the Eclipse profiler it does not seem to reclaim very much. In almost ever case, when I click the garbage collect button in the profiler it seems to do a much better job of collecting a lot.
What is the difference between explicitly invoking the garbage collector from the Eclipse profiler and calling System.gc() in code when the window is disposed?
How can I get my code to automatically make the garbage collector work so well?
Cheers, Eric

jschell wrote:
Might note however that calling gc() is unlikely to do anything to make your application better.Ain't it the truth. In fact it may even make it worse. Sane applications should not need manual GC calls - the runtime does what it is supposed to do itself and if it doesn't you should investigate and fix the problem that is preventing it from doing what it is supposed to do. First making sure that there is in fact an issue of course, perhaps the app simply needs 100mb of heap space at some point in the lifetime of your application. It isn't exactly a HUGE amount of memory, especially in a Java or .NET VM environment.

Simulate full garbage collection pauses

Similar Messages

Maybe you are looking for