Long pause of ParNew without Promotion failed.
I encounter a same problem like " [Long pauses |http://forums.sun.com/thread.jspa?threadID=5240437] ", no "promotion failed" tip but ParNew cost more than 20s, the following is the gc log's snapshot:
my jdk version is the latest JDK6u21 on RHEL5.1/IBM x3850 with 16core.
I try to use -X::+UseMemBar, but no effect. anybody could help me?
1913.579: [GC 1913.579: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15459032 bytes, 15459032 total
- age 2: 10545904 bytes, 26004936 total
: 433430K->35393K(458752K), 0.1056850 secs] 6509320K->6122752K(10420224K), 0.1061990 secs] [Times: user=0.83 sys=0.01, real=0.11 secs]
Heap after GC invocations=3850 (full 10):
par new generation total 458752K, used 35393K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.1082700 seconds
CMSCollector shouldConcurrentCollect: 1913.685
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.3237750 seconds
CMSCollector shouldConcurrentCollect: 1935.690
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
CMSCollector shouldConcurrentCollect: 1937.692
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Total time for which application threads were stopped: 23.8250250 seconds
CMSCollector shouldConcurrentCollect: 1937.834
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000880 seconds
Total time for which application threads were stopped: 0.0016130 seconds
CMSCollector shouldConcurrentCollect: 1937.836
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000580 seconds
Total time for which application threads were stopped: 0.0004200 seconds
CMSCollector shouldConcurrentCollect: 1937.837
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0019560 seconds
{Heap before GC invocations=3850 (full 10):
par new generation total 458752K, used 428609K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
1937.839: [GC 1937.839: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15389248 bytes, 15389248 total
- age 2: 10280752 bytes, 25670000 total
: 428609K->33832K(458752K), 0.0985530 secs] 6515968K->6131457K(10420224K), 0.0990430 secs] [Times: user=0.77 sys=0.01, real=0.10 secs]
Heap after GC invocations=3851 (full 10):
par new generation total 458752K, used 33832K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.0999980 seconds
CMSCollector shouldConcurrentCollect: 1937.939
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0446810 seconds
Total time for which application threads were stopped: 0.0048440 seconds
CMSCollector shouldConcurrentCollect: 1937.988
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.1273720 seconds
{Heap before GC invocations=3851 (full 10):
{Heap before GC invocations=3851 (full 10):
par new generation total 458752K, used 427048K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37808K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Same problems occur with some of our server instances of a large server backend with huge heapsizes. Hardware is 16 x86 cores on Windows at one site. Although we dont't see "promotion failed" messages.
-Djava.awt.headless=true
-Djava.rmi.dgc.leaseValue=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dfile.encoding=UTF-8
-Xmx11000m
-Xms7000m
-Xmn3500m
-XX:PermSize=768m
-XX:MaxPermSize=768m
-XX:+UseParNewGC
-XX:+DisableExplicitGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:-UseLargePages
-XX:SoftRefLRUPolicyMSPerMB=1
-XX:SurvivorRatio=1
-XX:InitialTenuringThreshold=15
-Xshare:off
-verbose:class
-verbose:gc
-Xloggc:gc.log
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintCMSInitiationStatistics
-XX:PrintCMSStatistics=1
-XX:+PrintGCTaskTimeStamps
-XX:+PrintTenuringDistribution
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=1234
The following two options are currently disabled to exclude problems with concurrent CMS perm size collections:
-XX:+CMSClassUnloadingEnabled
-XX:+CMSIncrementalMode
This extract of the GC log shows one pause of 40 seconds at 2010/11/19 08:35:30 during the ParNew phase which normally runs for only 0.05 to 0.1 seconds.
INFO | jvm 1 | 2010/11/19 08:34:45 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:34:47 | CMSCollector shouldConcurrentCollect: 143805.306
INFO | jvm 1 | 2010/11/19 08:34:47 | time_until_cms_gen_full 1857957.8130999
INFO | jvm 1 | 2010/11/19 08:34:47 | free=425866008
INFO | jvm 1 | 2010/11/19 08:34:47 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:34:47 | promotion_rate=203.795
INFO | jvm 1 | 2010/11/19 08:34:47 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:34:47 | occupancy=0.8942350
INFO | jvm 1 | 2010/11/19 08:34:47 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:34:47 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:34:49 | 2010-11-19T08:34:49.332+0100: 143807.135: [GC 143807.135: [ParNew
INFO | jvm 1 | 2010/11/19 08:35:30 | Desired survivor size 671088640 bytes, new threshold 15 (max 15)
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 1: 102644048 bytes, 102644048 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 2: 30772776 bytes, 133416824 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 3: 27784600 bytes, 161201424 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 4: 10277736 bytes, 171479160 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 5: 1696 bytes, 171480856 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 6: 60584 bytes, 171541440 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 7: 1968 bytes, 171543408 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 8: 2200 bytes, 171545608 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 9: 81752 bytes, 171627360 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 10: 153928 bytes, 171781288 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 11: 48 bytes, 171781336 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 12: 136 bytes, 171781472 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 13: 10104 bytes, 171791576 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 14: 41344 bytes, 171832920 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 15: 96 bytes, 171833016 total
INFO | jvm 1 | 2010/11/19 08:35:30 | : 1410396K->197458K(2621440K), 40.7184593 secs] 4926671K->3713735K(6553600K), 40.7185248 secs]CMSCollector shouldConcurrentCollect: 143847.854
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | [Times: user=1.25 sys=0.36, real=40.71 secs]
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.854
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | WrapperManager: The timer fell behind the system clock by 40700ms.
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.865
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.865
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.871
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
Similar Messages
-
Long pauses during ParNew garbage collection Please Help !
Hi,
We are running a server application on an large machine (~120 CPU, ~380 GB Memory).
After running 1 or 2 hours we suddenly get exorbitant application pause times during garbage collection and a massive cpu usage from the java vm
We are running on Java 6 (64Bit) with 6GB Heap.
Concurrent garbage collection is turned on using the parameters:
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:CMSInitiatingOccupancyFraction=80
-XX:+DisableExplicitGC
We turned on verbose garbage collection and are getting the following output:
1. Normal operation:
Application time: 217.4656792 seconds
3180.905: [GC 3180.906: [ParNew
Desired survivor size 20119552 bytes, new threshold 4 (max 4)
- age 1: 2843824 bytes, 2843824 total
- age 2: 2577128 bytes, 5420952 total
- age 3: 5742024 bytes, 11162976 total
- age 4: 625672 bytes, 11788648 total
: 329531K->15764K(353920K), 0.1484379 secs] 2435799K->2122105K(3392144K), 0.1492386 secs]
Total time for which application threads were stopped: 0.1886810 seconds
2. The Problem:
Application time: 2.8858445 seconds
5008.433: [GC 5008.434: [ParNew
Desired survivor size 20119552 bytes, new threshold 2 (max 4)
- age 1: 15837712 bytes, 15837712 total
- age 2: 12284416 bytes, 28122128 total
: 348338K->39296K(353920K), 138.5317715 secs] 2487779K->2192551K(3392144K), 138.5327383 secs]
Total time for which application threads were stopped: 138.5778558 seconds
Application time: 2.9764564 seconds
5149.957: [GC 5149.957: [ParNew
Desired survivor size 20119552 bytes, new threshold 2 (max 4)
- age 1: 9483176 bytes, 9483176 total
- age 2: 14499344 bytes, 23982520 total
: 353920K->39296K(353920K), 231.5110574 secs] 2507175K->2204546K(3392144K), 231.5121011 secs]
Total time for which application threads were stopped: 231.5257754 seconds
Application time: 2.7932907 seconds
5384.277: [GC 5384.278: [ParNew
Desired survivor size 20119552 bytes, new threshold 4 (max 4)
- age 1: 10756376 bytes, 10756376 total
- age 2: 9135888 bytes, 19892264 total
: 353920K->28449K(353920K), 256.2065591 secs] 2519170K->2207651K(3392144K), 256.2076388 secs]
Total time for which application threads were stopped: 256.2221463 seconds
I can't find any significant differences in the log between fast and long running garbage collections.
I urgently need help in solving this problem !
What can I do ?After the exciting reply I did get on my question, we did some more investigations on the problem and it seems that we finally found the solution to our problem.
The number of garbage collection threads used by the virtual machine defaults to the number of cpus of the machine.
This is ok for small machines or machines where the main load is produced by the java application itself.
In our environment the main load is not produced by the java application but oracle database processes.
When java tries to do it's garbage collection using 120 threads (# CPU) on the machine which is already overloaded by non java processes, the thread synchronization seems to produce an exorbitant overhead.
My theory is that spin locking is used on memory access, causing threads to spin while waiting for other blocking threads not getting cpu because of the heavy load on the system.
The solution is now to limit the number of garbage collection threads.
We did that on the first try by setting -XX:ParallelGCThreads=8
For over one day with heavy load no long GC pauses were experienced any more. -
ParNew (promotion failed) and long time stopped !
I use the latest JDK6u21 and I found a strange problem, the JVM will stop after 1 hours of my server start. after view the gc log, it said ParNew (promotion failed), but the memory is enough.
before promotion failed, I found the variable contiguous_available decreased to 3682040, but other case it is very big, and I used the "-XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=2 " for CMS memory fragment...
anybody could give me a light? I am very appreciate for that!
*1. The following is my jvm parameter:*
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+UseMembar -XX:PrintCMSStatistics=3 -XX:+PrintCMSInitiationStatistics -XX:+PrintAdaptiveSizePolicy -XX:+PrintCompilation -XX:+PrintConcurrentLocks -XX:+PrintGCTaskTimeStamps -XX:+PrintSharedSpaces -XX:+PrintTenuringDistribution -XX:+PrintVMQWaitTime -XX:+PrintHeapAtGC -Xms10g -Xmx10g -Xmn1512m -XX:SurvivorRatio=6 -XX:MaxTenuringThreshold=2 -XX:ParallelGCThreads=15 -XX:MaxPermSize=96m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:MaxGCPauseMillis=150 -XX:CMSAbortablePrecleanWaitMillis=100 -XX:ParallelCMSThreads=8 -XX:CMSInitiatingOccupancyFraction=65 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=2 -XX:GCTimeRatio=19 -Dsun.rmi.dgc.client.gcInterval=7200000 -Dsun.rmi.dgc.server.gcInterval=7200000 -XX:+UseFastAccessorMethods -XX:-UseFastEmptyMethods -XX:+UseFastJNIAccessors -Xnoclassgc -XX:+DisableExplicitGC
by the way, I had change the -Xmn to 512m, but same problem..
*2. The gc log of ParNew (promotion failed):*
4393.505: [GC 4393.505: [ParNew
Desired survivor size 99090432 bytes, new threshold 2 (max 2)
- age 1: 46429016 bytes, 46429016 total
- age 2: 34759512 bytes, 81188528 total
: 1273754K->126139K(1354752K), 0.2438560 secs] 7058007K->5941399K(10292224K), 0.2443570 secs] [Times: user=1.72 sys=0.00, real=0.24 secs]
Total time for which application threads were stopped: 0.2466460 seconds
CMSCollector shouldConcurrentCollect: 4393.750
time_until_cms_gen_full 99.3452089
free=3197145264
contiguous_available=3682040
promotion_rate=2.86619e+07
cms_allocation_rate=0
occupancy=0.6506605
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0001620 seconds
4393.751: [GC [1 CMS-initial-mark: 5815259K(8937472K)] 5941797K(10292224K), 0.3126700 secs] [Times: user=0.31 sys=0.00, real=0.32 secs]
Total time for which application threads were stopped: 0.3144440 seconds
4394.064: [CMS-concurrent-mark-start]
Application time: 0.6382660 seconds
4404.367: [GC 4404.368: [ParNew
Desired survivor size 99090432 bytes, new threshold 2 (max 2)
- age 1: 49937360 bytes, 49937360 total
- age 2: 30650624 bytes, 80587984 total
: 1254716K->132309K(1354752K), 0.2603790 secs] 7080972K->5991636K(10292224K), 0.2609090 secs] [Times: user=1.74 sys=0.01, real=0.26 secs]
Total time for which application threads were stopped: 0.2631700 seconds
Application time: 0.7369730 seconds
{Heap before GC invocations=3976 (full 37):
par new generation total 1354752K, used 1293525K [0x00002aaaae200000, 0x00002aab0ca00000, 0x00002aab0ca00000)
eden space 1161216K, 100% used [0x00002aaaae200000, 0x00002aaaf5000000, 0x00002aaaf5000000)
from space 193536K, 68% used [0x00002aaaf5000000, 0x00002aaafd1355e0, 0x00002aab00d00000)
to space 193536K, 0% used [0x00002aab00d00000, 0x00002aab00d00000, 0x00002aab0ca00000)
concurrent mark-sweep generation total 8937472K, used 5719921K [0x00002aab0ca00000, 0x00002aad2e200000, 0x00002aad2e200000)
concurrent-mark-sweep perm gen total 64956K, used 38890K [0x00002aad2e200000, 0x00002aad3216f000, 0x00002aad34200000)
*4405.367: [GC 4405.368: [ParNew (promotion failed)*
Desired survivor size 99090432 bytes, new threshold 2 (max 2)
- age 1: 44406944 bytes, 44406944 total
- age 2: 34887616 bytes, 79294560 total
: 1293525K->1255442K(1354752K), 3.3235350 secs]4408.691: [CMS4430.835: [CMS-concurrent-sweep: 24.401/28.241 secs] (CMS-concurrent-sweep yielded 3 times)
[Times: user=48.33 sys=1.42, real=28.24 secs]
*(concurrent mode failure) (concurrent mode failure): 5749663K->2538850K(8937472K), 22.2083950 secs] 7013447K->3794292K(10292224K), [CMS Perm : 38890K->38890K(64956K)], 25.5325540 secs] [Times: user=26.98 sys=0.00, real=25.53 secs]*
Heap after GC invocations=3977 (full 38):
par new generation total 1354752K, used 1255442K [0x00002aaaae200000, 0x00002aab0ca00000, 0x00002aab0ca00000)
eden space 1161216K, 100% used [0x00002aaaae200000, 0x00002aaaf5000000, 0x00002aaaf5000000)
from space 193536K, 48% used [0x00002aab00d00000, 0x00002aab06904a50, 0x00002aab0ca00000)
to space 193536K, 68% used [0x00002aaaf5000000, 0x00002aaafd1355e0, 0x00002aab00d00000)
concurrent mark-sweep generation total 8937472K, used 2538850K [0x00002aab0ca00000, 0x00002aad2e200000, 0x00002aad2e200000)
concurrent-mark-sweep perm gen total 64956K, used 38890K [0x00002aad2e200000, 0x00002aad3216f000, 0x00002aad34200000)
Total time for which application threads were stopped: 25.5346940 seconds
Application time: 1.6390800 seconds
{Heap before GC invocations=3977 (full 38):
par new generation total 1354752K, used 1354751K [0x00002aaaae200000, 0x00002aab0ca00000, 0x00002aab0ca00000)
eden space 1161216K, 100% used [0x00002aaaae200000, 0x00002aaaf5000000, 0x00002aaaf5000000)
from space 193536K, 99% used [0x00002aab00d00000, 0x00002aab0c9fff50, 0x00002aab0ca00000)
to space 193536K, 68% used [0x00002aaaf5000000, 0x00002aaafd1355e0, 0x00002aab00d00000)
concurrent mark-sweep generation total 8937472K, used 2538850K [0x00002aab0ca00000, 0x00002aad2e200000, 0x00002aad2e200000)
concurrent-mark-sweep perm gen total 64956K, used 38891K [0x00002aad2e200000, 0x00002aad3216f000, 0x00002aad34200000)
4432.541: [Full GC 4432.541: [CMS: 2538850K->2254294K(8937472K), 29.0664770 secs] 3893601K->2254294K(10292224K), [CMS Perm : 38891K->38867K(64956K)], 29.0668970 secs]CMSCollector shouldConcurrentCollect: 4461.608
time_until_cms_gen_full 203.1005006
free=6843573736
contiguous_available=6843442664
promotion_rate=3.01731e+07
cms_allocation_rate=0
occupancy=0.2522295
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
}another abnormal case without "promotion failed" tip, but cost long time of app stopped:
11958.151: [GC 11958.151: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15735688 bytes, 15735688 total
- age 2: 9981368 bytes, 25717056 total
: 436769K->30885K(458752K), 0.1082290 secs] 7131287K->6734723K(10420224K), 0.1088060 secs] [Times: user=0.82 sys=0.02, real=0.11 secs]
Total time for which application threads were stopped: 0.1130550 seconds
Application time: 0.1259410 seconds
Total time for which application threads were stopped: 25.8654900 seconds
Application time: 0.0003140 seconds
Total time for which application threads were stopped: 0.0035920 seconds
Application time: 0.0005900 seconds
Total time for which application threads were stopped: 0.0018320 seconds
Application time: 0.0002240 seconds
Total time for which application threads were stopped: 25.7993410 seconds
Application time: 0.0002000 seconds
Total time for which application threads were stopped: 0.0035800 seconds
Application time: 0.0059850 seconds
Total time for which application threads were stopped: 0.0031670 seconds
Application time: 0.0754810 seconds
12010.152: [GC 12010.152: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 21472096 bytes, 21472096 total
- age 2: 9853168 bytes, 31325264 total
: 424101K->48164K(458752K), 0.1014030 secs] 7092494K->6726096K(10420224K), 0.1018690 secs] [Times: user=0.88 sys=0.01, real=0.10 secs]
Total time for which application threads were stopped: 0.1081710 seconds
Application time: 0.1441860 seconds
Total time for which application threads were stopped: 0.0064210 seconds
Application time: 0.0013600 seconds
Total time for which application threads were stopped: 0.0151980 seconds
Application time: 0.0033530 seconds
Total time for which application threads were stopped: 0.0236910 seconds
Application time: 0.0139520 seconds
12010.491: [GC 12010.491: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 26804960 bytes, 26804960 total
- age 2: 14022000 bytes, 40826960 total
: 441380K->56783K(458752K), 0.1109360 secs] 7099837K->6724826K(10420224K), 0.1113860 secs] [Times: user=1.04 sys=0.02, real=0.12 secs]
Total time for which application threads were stopped: 0.1172810 seconds
Application time: 0.1822510 seconds
12010.792: [GC 12010.792: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 21406568 bytes, 21406568 total
- age 2: 19288808 bytes, 40695376 total
: 449999K->65536K(458752K), 0.1353970 secs] 7089876K->6720230K(10420224K), 0.1358610 secs] [Times: user=1.16 sys=0.01, real=0.14 secs]
Total time for which application threads were stopped: 0.1424450 seconds
Application time: 0.1772370 seconds
this jdk version is java version "1.6.0_20" -
Periodic GC pauses (promotion failed)
We are using the folling JVM options on a moderately loaded production server (running a servlet engine):
-server -Xms2500m -Xmx2500m -XX:LargePageSizeInBytes=2m -XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:-CMSParallelRemarkEnabled -XX:NewSize=384m -XX:MaxNewSize=384m -XX:MaxTenuringThreshold=3 -XX:SurvivorRatio=8 -XX:PermSize=64m -XX:MaxPermSize=64m -XX:CMSInitiatingOccupancyFraction=55
Periodically, young generation collections will take 5-15s rather than the usual 100ms due to failed promotions:
42528.169: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs]
42529.143: [GC 42529.143: [ParNew: 353920K->39296K(353920K), 0.2909090 secs] 1694024K->1452600K(2520704K), 0.2911660 secs]
42533.178: [GC[YG occupancy: 196817 K (353920 K)]42533.178: [Rescan (non-parallel) 42533.178: [grey object rescan, 0.0293920 secs]42
533.208: [root rescan, 0.1727150 secs], 0.2022200 secs]42533.380: [weak refs processing, 0.0097820 secs] [1 CMS-remark: 1413304K(216
6784K)] 1610122K(2520704K), 0.2121640 secs]
42533.390: [CMS-concurrent-sweep-start]
42534.823: [CMS-concurrent-sweep: 1.432/1.432 secs]
42534.823: [CMS-concurrent-reset-start]
42534.851: [CMS-concurrent-reset: 0.028/0.028 secs]
42536.572: [GC 42536.572: [ParNew: 353920K->2080K(353920K), 0.1246980 secs] 1450967K->1137843K(2520704K), 0.1249020 secs]
42546.733: [GC 42546.733: [ParNew: 316704K->3604K(353920K), 0.0599980 secs] 1452467K->1139368K(2520704K), 0.0602080 secs]
42555.527: [GC 42555.527: [ParNew: 318228K->5034K(353920K), 0.0609210 secs] 1453992K->1140797K(2520704K), 0.0611270 secs]
42564.154: [GC 42564.154: [ParNew: 319658K->5214K(353920K), 0.0780650 secs] 1455421K->1142411K(2520704K), 0.0782850 secs]
42571.444: [GC 42571.444: [ParNew: 319838K->5514K(353920K), 0.0649090 secs] 1457035K->1144161K(2520704K), 0.0651220 secs]
42576.714: [GC 42576.714: [ParNew (promotion failed): 320138K->320138K(353920K), 0.2365970 secs]42576.951: [CMS: 1139969K->1120688K(
2166784K), 9.2214860 secs] 1458785K->1120688K(2520704K), 9.4584090 secs]
42587.017: [GC 42587.017: [ParNew: 314624K->34454K(353920K), 0.1340880 secs] 1435312K->1205264K(2520704K), 0.1344800 secs]
42588.700: [GC 42588.701: [ParNew: 349078K->23091K(353920K), 0.1186190 secs] 1519888K->1227195K(2520704K), 0.1190990 secs]
42588.861: [GC [1 CMS-initial-mark: 1204104K(2166784K)] 1227628K(2520704K), 0.0134160 secs]
42588.875: [CMS-concurrent-mark-start]
42593.001: [CMS-concurrent-mark: 4.126/4.127 secs]
42593.001: [CMS-concurrent-preclean-start]
42593.001: [CMS-concurrent-preclean: 0.000/0.000 secs]
42593.259: [CMS-concurrent-abortable-preclean-start]
42593.259: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs]
Any suggestions on how to avoid these pauses? Would increasing the total heap (Xmx) or reducing CMSInitiatingOccupancyFraction help? We did see the same issues when using a significantly smaller young generation (128m, with MaxTenuringThreshold set to 0).
Also, occaisionally a "grey object rescan" takes 5-10s:
152810.810: [GC 152810.811: [ParNew: 320276K->6864K(353920K), 0.0684100 secs] 1549279K->1235867K(2520704K), 0.0687690 secs]
152810.885: [GC[YG occupancy: 8070 K (353920 K)]152810.885: [Rescan (non-parallel) 152810.885: [grey object rescan, 7.8413190 secs]152818.727: [root rescan, 0.0196880 secs], 7.8611820 secs]152818.747: [weak refs processing, 0.0090520 secs] [1 CMS-remark: 1229003K(2166784K)] 1237074K(2520704K), 7.8703580 secs]
What exactly is this and is it stop-the-world or concurrent? Can it be avoided or at least reduced?
Any help/suggestions are highly appreciated. Thanks!Thanks, we tried this but no luck.
We used the following settings:
-server -Xms2000m -Xmx3500m -XX:LargePageSizeInBytes=2m -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCs
BeforeCompaction=0 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseAdaptiveSizePolicy -XX:M
axGCPauseMillis=200 -XX:PermSize=96m -XX:MaxPermSize=96m -XX:CMSInitiatingOccupancyFraction=60
Also the adaptive size policy is selected, the young generation stayed at 20MB:
18239.813: [GC 18239.813: [ParNew: 21114K->0K(21184K), 0.3406570 secs] 1995672K->1976591K(3583936K), 0.3410450 secs]
18240.323: [GC 18240.323: [ParNew: 21116K->0K(21184K), 0.3173180 secs] 1997708K->1978403K(3583936K), 0.3176800 secs]
18240.818: [GC 18240.818: [ParNew: 21120K->0K(21184K), 0.3158010 secs] 1999523K->1980380K(3583936K), 0.3161750 secs]
18241.270: [GC 18241.270: [ParNew: 21120K->0K(21184K), 0.3192580 secs] 2001500K->1982506K(3583936K), 0.3196310 secs]
18241.870: [GC 18241.870: [ParNew: 21120K->0K(21184K), 0.3169500 secs] 2003626K->1984487K(3583936K), 0.3173320 secs]
18242.482: [GC 18242.482: [ParNew: 21112K->0K(21184K), 0.3169740 secs] 2005600K->1986487K(3583936K), 0.3173510 secs]
18243.146: [GC 18243.146: [ParNew: 21118K->0K(21184K), 0.3067560 secs] 2007606K->1988292K(3583936K), 0.3071490 secs]
18243.680: [GC 18243.680: [ParNew: 21120K->0K(21184K), 0.3058310 secs] 2009412K->1990478K(3583936K), 0.3062120 secs]
Without +UseConcMarkSweep, it worked as expected (but resulted in longer GC pause times when full collections occured).
Is this a bug or are my settings not correct? -
ParNew long pauses in Tomcat 6 on T5120.
I am running some simmulated user load for 50 concurent users on a Java application running inside Tomcat 6 on Sun's T5120 (8 processor 8G RAM) on Solaris 10. The utilization of the application seens to be ok (90-95%), but sometimes JVM goes into long GC pauses that are not acceptable for in my case. The longest pauses I can afford is about 10 seconds. The only other thing that's running on the machine is apache (mod_jk) that forwards the requests to the Tomcat process.
My GC Settings:
GC_VERBOSE_OPTS="-XX:+PrintTenuringDistribution -Xloggc:${CATALINA_HOME}/logs/gc_${TSTAMP}.log"
GC_VERBOSE="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime $GC_VERBOSE_OPTS"
MEMORY_SETTINGS="-Xms6912m -Xmx6912m -XX:NewSize=1728m -XX:MaxNewSize=1728m -XX:SurvivorRatio=8 -Xss256k -d64 -XX:PermSize=192M -XX:MaxPermSize=192M"
GC_SETTINGS=" -XX:+DisableExplicitGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -X:ParallelGCThreads=6
The output of verbose GC just before it goes into very long 55 second pause:
76824.094: [GC 76824.094: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age 1: 28658536 bytes, 28658536 total
- age 2: 78258856 bytes, 106917392 total
- age 3: 47935016 bytes, 154852408 total
: 1592576K->152602K(1592576K), 0.7859117 secs] 4605449K->3213877K(6900992K), 0.7864152 secs]
Total time for which application threads were stopped: 0.7876342 seconds
Application time: 17.9823540 seconds
Total time for which application threads were stopped: 0.0037981 seconds
Application time: 0.0002300 seconds
76842.868: [Full GC 76842.868: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age 1: 16586944 bytes, 16586944 total
- age 2: 24580024 bytes, 41166968 total
- age 3: 77453264 bytes, 118620232 total
- age 4: 41331496 bytes, 159951728 total
: 1568283K->157516K(1592576K), 0.5095165 secs] 4629559K->3218791K(6900992K), 0.5099791 secs]
Total time for which application threads were stopped: 0.5109060 seconds
Application time: 21.7442444 seconds
76865.132: [GC 76865.132: [ParNew
Desired survivor size 163027352 bytes, new threshold 5 (max 31)
- age 1: 70949544 bytes, 70949544 total
- age 2: 325496 bytes, 71275040 total
- age 3: 16995328 bytes, 88270368 total
- age 4: 51961024 bytes, 140231392 total
- age 5: 39539328 bytes, 179770720 total
: 1573196K->176896K(1592576K), 4.8568739 secs] 4634471K->3402338K(6900992K), 4.8573679 secs]
Total time for which application threads were stopped: 4.8667158 seconds
Application time: 0.0000997 seconds
76870.002: [GC [1 CMS-initial-mark: 3225442K(5308416K)] 3425594K(6900992K), 1.2304280 secs]
Total time for which application threads were stopped: 1.2421993 seconds
76871.233: [CMS-concurrent-mark-start]
Application time: 3.9732836 seconds
76875.212: [GC 76875.212: [ParNew
Desired survivor size 163027352 bytes, new threshold 5 (max 31)
- age 1: 89811184 bytes, 89811184 total
- age 2: 47431856 bytes, 137243040 total
- age 3: 229704 bytes, 137472744 total
- age 4: 10895976 bytes, 148368720 total
- age 5: 31371840 bytes, 179740560 total
: 1592576K->176896K(1592576K), 19.5701529 secs] 4818018K->3616356K(6900992K), 19.6394327 secs]
*Total time for which application threads were stopped: 19.6456021 seconds*
Application time: 5.3914509 seconds
76900.622: [GC 76900.622: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age 1: 73583152 bytes, 73583152 total
- age 2: 73057080 bytes, 146640232 total
- age 3: 32981616 bytes, 179621848 total
- age 4: 98336 bytes, 179720184 total
- age 5: 12880 bytes, 179733064 total
: 1592576K->176896K(1592576K), 54.7521974 secs] 5032036K->3921814K(6900992K), 54.7527833 secs]
*Total time for which application threads were stopped: 55.1387374 seconds*
Application time: 5.3171414 seconds
76960.700: [GC 76960.700: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age 1: 66359848 bytes, 66359848 total
- age 2: 62758584 bytes, 129118432 total
- age 3: 50477568 bytes, 179596000 total
: 1592576K->176896K(1592576K), 7.1058428 secs] 5337494K->4095887K(6900992K), 7.1063802 secs]
*Total time for which application threads were stopped: 7.1077796 seconds*
Application time: 12.7908379 seconds
Total time for which application threads were stopped: 0.0042312 seconds
Application time: 0.3358069 seconds
76980.938: [GC 76980.939: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age 1: 69012896 bytes, 69012896 total
- age 2: 56309432 bytes, 125322328 total
- age 3: 54304696 bytes, 179627024 total
: 1592576K->176896K(1592576K), 2.5164799 secs] 5512702K->4233158K(6900992K), 2.5169843 secs]
Total time for which application threads were stopped: 2.5178621 seconds
Application time: 14.8490191 seconds
Total time for which application threads were stopped: 0.0070005 seconds
Application time: 0.0003056 seconds
76998.312: [Full GC 76998.313: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age 1: 38875304 bytes, 38875304 total
- age 2: 63941104 bytes, 102816408 total
- age 3: 56090408 bytes, 158906816 total
: 1592576K->156275K(1592576K), 1.1885482 secs] 5648848K->4265573K(6900992K), 1.1891583 secs]
Total time for which application threads were stopped: 1.1899499 seconds
Application time: 8.8337996 seconds
I would appreciate some help tunning GC patameters to eliminate the long pauses.
Edited by: MikhailPDX on May 6, 2008 11:53 AMI saw this posting before about pauses and I don't think I am having the same issue since I am using Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_15-b04 Java HotSpot(TM) 64-Bit Server VM) where the mentioned pauses bug was supposedly fixed.
Edited by: MikhailPDX on May 7, 2008 10:30 AM -
How can I play online videos without many long pauses during play?
I have current flash and 7. Lately, all online videos have many long pauses throughout play. They used to play continuously through. Flash is enabled.
How can I solve this?To help troubleshoot we'll need the following system information:
Operating system
Browser
Flash Player version
When reporting issues with video or audio, it's also helpful to get your system hardware and driver details. Instructions for finding this information can be found here:
Windows
Mac
Finally, sometimes video and audio problems are caused at a lower level and not directly related to Flash Player. I recommend trying both of the links below to see how they perform. If the problem exists with both, then Flash Player is most likely not the culprit as the HTML5 video link does not use Flash Player when playing. You can verify the use of HTML5 by right clicking the HTML5 video and looking for the words "About HTML5" at the bottom of the context menu.
HTML5 video
Non-HTML5 video -
Hi,
I'm trying to tune the JVM options for a JAVA app that requires low pauses. After playing around with several diferent combinations I still can not get a setup that will not give an occasional very long pause.
Here's an example of a long pause from the stdout log:
2007-11-24 07:44:29| occupancy=0.6494621
2007-11-24 07:44:29| initiatingOccupancy=0.6800000
2007-11-24 07:44:32| CMSCollector shouldConcurrentCollect: 64459.941
2007-11-24 07:44:32| time_until_cms_gen_full 1986.2853957
2007-11-24 07:44:32| free=4410787008
2007-11-24 07:44:32| contiguous_available=1888036816
2007-11-24 07:44:32| promotion_rate=1.98556e+06
2007-11-24 07:44:32| cms_allocation_rate=0
2007-11-24 07:44:32| occupancy=0.6494621
2007-11-24 07:44:32| initiatingOccupancy=0.6800000
2007-11-24 07:44:32| Total time for which application threads were stopped: *46.1601130 seconds*
2007-11-24 07:44:32| CMSCollector shouldConcurrentCollect: 64461.180
2007-11-24 07:44:32| time_until_cms_gen_full 1986.2853957
2007-11-24 07:44:32| free=4410787008
2007-11-24 07:44:32| contiguous_available=1888036816
2007-11-24 07:44:32| promotion_rate=1.98556e+06
2007-11-24 07:44:32| cms_allocation_rate=0
2007-11-24 07:44:32| occupancy=0.6494621
2007-11-24 07:44:32| initiatingOccupancy=0.6800000
2007-11-24 07:44:32| CMSCollector shouldConcurrentCollect: Total time for which application threads were stopped: 0.0048
4
The pause of 46 seconds is not good for the application. It would be OK to have an occasional 5 sec pause but not 46 seconds.
I've been through likely all JVM tunning papers, blogs and previus postings to this forum.
BTW I know that some of the JVM options I'm setting are already the default but it helps me keep track of which options I tried.
Can anyone help?
Here are some details:
Platform: Linux 2.6.9-42.0.10.ELsmp #1 SMP Tue Feb 27 09:40:21 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
16GB of physical memory, 2 x dual core CPUs AMD 1.86GHz
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
JVM options:
-Xbatch -XX:-DontCompileHugeMethods -XX:CMSMaxAbortablePrecleanTime=10000
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=68
-XX:+UseParNewGC -XX:SurvivorRatio=15
-XX:+CMSConcurrentMTEnabled -XX:ParallelGCThreads=4 -XX:+CMSParallelSurvivorRemarkEnabled -XX:InitialTenuringThreshold=1 -XX:MaxTenuringThreshold=8 -XX:TargetSurvivorRatio=90 -XX:ThreadPriorityPolicy=0 -XX:+AggressiveOpts -XX:CMSTriggerRatio=65 -XX:+AlwaysActAsServerClassMachine -XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:PrintCMSStatistics=1 -XX:+PrintCMSInitiationStatistics
-XX:+PrintGCApplicationStoppedTime -verbose:gc -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+UseTLAB
-Xms13000m -Xmx13000m -XX:MaxNewSize=1000m -XX:NewSize=1000m -XX:ThreadStackSize=128I encounter same problem, no "promotion failed" tip but ParNew cost more than 20s, the following is the gc log's snapshot:
my jdk version is the latest JDK6u21 on RHEL5.1/IBM x3850 with 16core.
I try to use -X::+UseMemBar, but no effect.
1913.579: [GC 1913.579: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15459032 bytes, 15459032 total
- age 2: 10545904 bytes, 26004936 total
: 433430K->35393K(458752K), 0.1056850 secs] 6509320K->6122752K(10420224K), 0.1061990 secs] [Times: user=0.83 sys=0.01, real=0.11 secs]
Heap after GC invocations=3850 (full 10):
par new generation total 458752K, used 35393K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.1082700 seconds
CMSCollector shouldConcurrentCollect: 1913.685
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.3237750 seconds
CMSCollector shouldConcurrentCollect: 1935.690
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
CMSCollector shouldConcurrentCollect: 1937.692
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Total time for which application threads were stopped: 23.8250250 seconds
CMSCollector shouldConcurrentCollect: 1937.834
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000880 seconds
Total time for which application threads were stopped: 0.0016130 seconds
CMSCollector shouldConcurrentCollect: 1937.836
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000580 seconds
Total time for which application threads were stopped: 0.0004200 seconds
CMSCollector shouldConcurrentCollect: 1937.837
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0019560 seconds
{Heap before GC invocations=3850 (full 10):
par new generation total 458752K, used 428609K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
1937.839: [GC 1937.839: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15389248 bytes, 15389248 total
- age 2: 10280752 bytes, 25670000 total
: 428609K->33832K(458752K), 0.0985530 secs] 6515968K->6131457K(10420224K), 0.0990430 secs] [Times: user=0.77 sys=0.01, real=0.10 secs]
Heap after GC invocations=3851 (full 10):
par new generation total 458752K, used 33832K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.0999980 seconds
CMSCollector shouldConcurrentCollect: 1937.939
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0446810 seconds
Total time for which application threads were stopped: 0.0048440 seconds
CMSCollector shouldConcurrentCollect: 1937.988
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.1273720 seconds
{Heap before GC invocations=3851 (full 10):
{Heap before GC invocations=3851 (full 10):
par new generation total 458752K, used 427048K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37808K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000) -
Problem with long pauses in Tenured collection
We have a Swing application running on Windows XP, which gets long pauses when Tenured objects are collected after a running couple of days.
System:
Windows XP
Intel Pentium M 1.8 ghz
512 MB ram
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03)
JVM Arguments:
-XX:+ForceTimeHighResolution -Xloggc:gc.log -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xms16m -Xmx384m
The first Tenured collections after 1 hour:
3520.317: [GC 3520.317: [DefNew: 2414K->242K(2432K), 0.0035340 secs]3520.320: [Tenured: 32111K->15691K(32140K), 0.3034778 secs] 34405K->15691K(34572K), 0.3072704 secs]
3673.867: [GC 3673.868: [DefNew: 2335K->125K(2432K), 0.0028552 secs]3673.870: [Tenured: 32042K->18786K(32140K), 0.2668128 secs] 34312K->18786K(34572K), 0.2699225 secs]
3839.103: [GC 3839.103: [DefNew: 2423K->156K(2432K), 0.0028918 secs]3839.106: [Tenured: 32126K->22544K(32140K), 0.3026440 secs] 34341K->22544K(34572K), 0.3058143 secs]Pretty harmless so far.
After 1 day:
85332.923: [GC 85332.924: [DefNew: 6528K->703K(6528K), 0.0122671 secs]85332.936: [Tenured: 86929K->48880K(87032K), 0.6243351 secs] 92687K->48880K(93560K), 0.6369188 secs]
97305.405: [GC 97305.405: [DefNew: 6493K->91K(6528K), 0.0041855 secs]97305.410: [Tenured: 86755K->55828K(86776K), 0.5189188 secs] 92662K->55828K(93304K), 0.5233905 secs]
97844.653: [GC 97844.653: [DefNew: 7040K->768K(7040K), 0.0142342 secs]97844.668: [Tenured: 93393K->57002K(93432K), 0.5293686 secs] 99611K->57002K(100472K), 0.5438841 secs]The time between two Tenured collection just took a jump now to about 3 hours. The gc length is about half a second.
After 3 days:
256894.994: [GC 256894.994: [DefNew: 13938K->1166K(14720K), 0.0080440 secs]256895.002: [Tenured: 195455K->114421K(195696K), 0.8638032 secs] 208915K->114421K(210416K), 0.8721174 secs]
258670.232: [GC 258670.232: [DefNew: 14327K->1600K(14720K), 0.0273586 secs]258670.259: [Tenured: 196045K->117861K(196104K), 0.9146393 secs] 209062K->117861K(210824K), 0.9422603 secs]
261577.464: [GC 261577.464: [DefNew: 14144K->584K(14848K), 0.0078192 secs]261577.472: [Tenured: 196705K->121486K(196820K), 0.8976985 secs] 210412K->121486K(211668K), 0.9058215 secs]The Tenured collection interval is about 2-3 hours now. Each collection is about 1 second.
Then we get some peaks after 4 days:
413096.848: [GC 413096.848: [DefNew: 20126K->997K(21504K), 0.0067539 secs]413096.855: [Tenured: 285819K->160167K(285828K), 2.9034254 secs] 305888K->160167K(307332K), 2.9103817 secs]
472214.414: [GC 472214.414: [DefNew: 22388K->1227K(23616K), 0.0095498 secs]472214.424: [Tenured: 313891K->178781K(313992K), 22.2893423 secs] 336110K->178781K(337608K), 22.3386254 secs]
485465.045: [GC 485465.045: [DefNew: 23616K->2423K(23616K), 0.0404118 secs]485465.086: [Tenured: 315925K->203606K(316040K), 9.0069023 secs] 336994K->203606K(339656K), 9.0477205 secs]
492248.783: [GC 492248.783: [DefNew: 24461K->24461K(25536K), 0.0000111 secs]492248.783: [Tenured: 338665K->201218K(339344K), 6.5526445 secs] 363126K->201218K(364880K), 6.5985010 secs]
533898.023: [GC 533898.023: [DefNew: 25913K->25913K(27200K), 0.0000150 secs]533898.024: [Tenured: 337387K->205940K(363008K), 3.2169493 secs] 363301K->205940K(390208K), 3.2172333 secs]
546566.675: [GC 546566.675: [DefNew: 27199K->27199K(27200K), 0.0000103 secs]546566.675: [Tenured: 338319K->227086K(363008K), 26.2682151 secs] 365519K->227086K(390208K), 26.2684060 secs]
562738.779: [GC 562738.780: [DefNew: 26081K->26081K(27200K), 0.0000104 secs]562738.780: [Tenured: 336929K->230971K(363008K), 7.9569240 secs] 363010K->230971K(390208K), 7.9571524 secs]The average time between each young collection is approx. 20 seconds.
7 full gc's have been performed:
3.877: [Full GC 3.877: [Tenured: 1759K->1831K(15168K), 0.0801147 secs] 2351K->1831K(16320K), [Perm : 12287K->12287K(12288K)], 0.0802154 secs]
6.281: [Full GC 6.281: [Tenured: 5071K->4056K(15168K), 0.1245083 secs] 6018K->4056K(16320K), [Perm : 16383K->16383K(16384K)], 0.1245978 secs]
184.854: [Full GC 184.854: [Tenured: 12628K->10437K(15168K), 0.2133769 secs] 12995K->10437K(16320K), [Perm : 20480K->20480K(20480K)], 0.2135845 secs]
5216.782: [Full GC 5216.782: [DefNew: 3008K->101K(3008K), 0.0044835 secs] 35049K->32886K(42340K), 0.0046464 secs]
198022.892: [Full GC 198022.892: [DefNew: 11520K->790K(11520K), 0.0110744 secs] 117670K->108219K(164672K), 0.0112750 secs]
254208.982: [Full GC 254208.982: [DefNew: 13765K->844K(14720K), 0.0059109 secs] 151946K->139025K(210056K), 0.0061115 secs]
566774.751: [Full GC 566774.751: [DefNew: 25680K->1611K(27200K), 0.0103055 secs] 274385K->250385K(390208K), 0.0105140 secs]The long pauses are not acceptable at all. Only pauses under 1 second are tolerable.
I know there is at least one memory leak in the logged software, where probably 50.000-100.000 ThreadGroups are produced each day and not collected (but only with some configurations).
So, after browsing around, I have found out that UseConcMarkSweepGC could reduce the long pauses. I tried it together with CMSIncrementalMode, but the JVM running at one of our customers crashed after 2 hours while using both options together. Apparently there where some issues with the incremental collector, which was fixed in 1.6. I was forced to remove the VM arguments immediately, but will probably use UseConcMarkSweepGC later after some thorough testing.
I have tried find out which other VM options I could use to tune the application, but I'm quite new to this and the amount of options are a bit overwhelming at the moment. I thought I could ask here if someone know some obvious option values (like NewRatio,SurvivorRatio etc.)>
>
Doesn't a "Full Collection" always collect all generations (i.e. also the perm generation)? It's always been my understanding, that a "Full Collection" is some kind of "irregular" collection which only occurs if the JVM has "really bad need" for memory (e.g. when the concurrent phase of a CMS collection fails, when perm runs full or if someone calls System.gc()). So, as you write earlier, the first 3 collections labeld "Full GC" are real full GCs since they collect all generations (including perm). The latter four collections shouldn't be labeld "Full GC" (as you write). Yes, a full collection collects all the generations including the
perm generation.
In a stop-the-world collector such as the one you're using, a full
collection is not an unusual collection. They just happen less
frequenctly. Objects that survive a minor collection (collection
of the young gen) a few times will be promoted (copied to) the
tenured generation. The idea is that such objects may be
long lived so should be promoted to save the cost of handling
them during a minor collection. Eventually the tenured
generation gets full and then it gets collected.
And yes and yes. The first 3 are full collections and the
latter 4 are labelled incorrectly as full collections.
>
However, there are a lot of collections with "DefNew" followed by "Tenured" (the first three sections in the original post) which are regular collections where the promotion of objects from new to old caused the old generation to run full which triggers a "Tenured collection". But these collections (according to my understanding) are not full collections, but only regular tenured collections. (Sorry, I don't want to be picky about this. I guess that's even what you meant in your post...)I can see how this can be unclear. A "Tenured" collection
is a full collection. It also collects the perm gen. And
in fact it also collects the young gen again. This is how it
is implemented in the hotspot JVM. It can be implemented
differently in other JVM's.
The situation where a "DefNew" is followed by a "Tenured"
as in your logs is a bit unusual I think. The minor collection
is reducing the amount of used space in young generation.
DefNew: 13938K->1166K(14720K)
The decrease from 13938K to 1166K. There is then a collection
of the tenured gen.
Tenured: 195455K->114421K(195696K)
I've seen this when large objects (a significant fraction of
the young gen) are being allocated. The allocation fails in
the young gen, a minor collection is done but the space freed
in the young gen is still not large enough to allocate the object
so a collection of the tenured generation is done and the large
object is allocated directly out of the tenured gen. In the numbers
above after the minor collection about 13M of free space is
available in the young gen. Are you allocating objects larger
that 13M? -
How to solve the problem of "promotion failed"
Hello,
I am new to JVM tuning, could anybody help me and explain how to solve a problem of "promotion failed" as following? (Java 1.6 and the JVM options: -XX:+UseConcMarkSweepGC -XX:NewSize=128m -XX:MaxPermSize=256m ):
[GC 96800.304: [ParNew (promotion failed): 287552K->287552K(287552K), 0.8694459 secs]96801.174: [CMS: 5229568K->457193K(5971968K), 19.7176579 secs] 5499429K->457193K(6259520K), [CMS Perm : 66540K->66458K(110904K)], 20.5878599 secs] [Times: user=22.50 sys=0.03, real=20.59 secs]
bevor: 96755.515: [GC 96755.515: [ParNew: 287552K->31936K(287552K), 0.1900583 secs] 5491052K->5243813K(6259520K), 0.1905515 secs] [Times: user=2.21 sys=0.00, real=0.19 secs]
after: 96885.774: [GC 96885.774: [ParNew: 255484K->31936K(287552K), 0.0808239 secs] 712678K->490703K(6259520K), 0.0812894 secs] [Times: user=0.97 sys=0.01, real=0.08 secs]
my questions:
1. why did the promotion failed when the old generation apears to have enough space (5229568+287552=5517120 < 5971968)? muss "promotion failed" by all means be handelt?
2. [CMS: 5229568K->457193K(5971968K), 19.7176579 secs]: is this full GC and is the time of 19.7176579 really pause (stop the world) time?
3. [CMS Perm : 66540K->66458K(110904K)], 20.5878599 secs]: why ist Permanent space collected too? dit it take a time of 20.5878599-19.7176579=0.870202 ? this action apears to be unnessesary (66540K->66458K), is es possible to disable such aktion?
4. how to solve the problem? I read a post of fixing such problem with XX:SurvivorRatio=4 -XX:CMSInitiatingOccupancyFraction=75, But this did not work for my case.
could this problem be due to the fragmentation of old generation space? could the reduce size of PLAB improving reuse of small chunks like -XX:OldPLABSize=16 help in my case? what ist the default value of OldPLABSize of JAVA 6 ? is there any side-effect for the reduce OldPLABSize?
thanks a lot for any help!
TianLooks like your old generation has become fragmented and the old space is almost full, or maybe a promoted object is too huge, and there is not enough continue space. Try following:
1> increase the size of old generation.
2> If increase the size of old generation doesnt work , you may try to use G1 algorithm, it may reduce the fragment problem of old generation.
Thanks
Amrit -
CMS collector taking too long pauses due to fragmentation
we are using Weblogic 10gR3 servers with JDK 160_23 for ODSI application and using CMS collector for garbage collection. But we are seeing ParNew (promotion failed) due to fragmentation and ending up CMS having more than 30 seconds stop the world pauses every 12-13 hours. other than this normally CMS takes only 0.03 - 0.05 seconds of application pauses. Here is the JVM arguments we are using and the GC logs that has ParNew - promotion failed.
/opt/oracle/10gR3/jdk160_23/jre/bin/java -Dweblogic.Name=member3MS1 -Djava.security.policy=/opt/oracle/10gR3/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.management.server=http://wdcsn443a.sys.cigna.com:7001 -Djava.library.path=/opt/oracle/10gR3/jdk160_23/jre/lib/sparc/client:/opt/oracle/10gR3/jdk160_23/jre/lib/sparc:/opt/oracle/10gR3/jdk160_23/jre/../lib/sparc:/opt/oracle/10gR3/patch_wlw1030/profiles/default/native:/opt/oracle/10gR3/patch_wls1030/profiles/default/native:/opt/oracle/10gR3/patch_cie670/profiles/default/native:/opt/oracle/10gR3/patch_aldsp1030/profiles/default/native:/opt/oracle/10gR3/patch_wlw1030/profiles/default/native:/opt/oracle/10gR3/patch_wls1030/profiles/default/native:/opt/oracle/10gR3/patch_cie670/profiles/default/native:/opt/oracle/10gR3/patch_aldsp1030/profiles/default/native:.:/opt/oracle/10gR3/wlserver_10.3/server/native/solaris/sparc:/opt/oracle/10gR3/wlserver_10.3/server/native/solaris/sparc/oci920_8:/opt/oracle/10gR3/wlserver_10.3/server/native/solaris/sparc:/opt/oracle/10gR3/wlserver_10.3/server/native/solaris/sparc/oci920_8:/opt/oracle/10gR3/wlserver_10.3/server/native/solaris/sparc:/opt/oracle/10gR3/wlserver_10.3/server/native/solaris/sparc/oci920_8:/usr/jdk/packages/lib/sparc:/lib:/usr/lib -Djava.class.path=/opt/oracle/10gR3/user_projects/lib/commons-lang-2.4.jar:/opt/oracle/10gR3/user_projects/lib/log4j-1.2.15.jar:/opt/oracle/10gR3/modules/com.bea.common.configfwk_1.2.0.0.jar:/opt/oracle/10gR3/modules/com.bea.core.xquery.beaxmlbeans-interop_1.3.0.0.jar:/opt/oracle/10gR3/modules/com.bea.core.xquery.xmlbeans-interop_1.3.0.0.jar:/opt/oracle/10gR3/modules/com.bea.core.binxml_1.3.0.0.jar:/opt/oracle/10gR3/modules/com.bea.core.sdo_1.1.0.0.jar:/opt/oracle/10gR3/modules/com.bea.core.xquery_1.3.0.0.jar:/opt/oracle/10gR3/modules/com.bea.alsb.client_1.1.0.0.jar:/opt/oracle/10gR3/modules/com.bea.common.configfwk.wlinterop_10.3.0.0.jar:/opt/oracle/10gR3/patch_wss110/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/oracle/10gR3/patch_wls1001/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/oracle/10gR3/patch_cie650/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/oracle/10gR3/patch_aldsp320/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/oracle/10gR3/jdk160_23/lib/tools.jar:/opt/oracle/10gR3/wlserver_10.3/server/lib/weblogic_sp.jar:/opt/oracle/10gR3/wlserver_10.3/server/lib/weblogic.jar:/opt/oracle/10gR3/modules/features/weblogic.server.modules_10.0.1.0.jar:/opt/oracle/10gR3/modules/features/com.bea.cie.common-plugin.launch_2.1.2.0.jar:/opt/oracle/10gR3/wlserver_10.3/server/lib/webservices.jar:/opt/oracle/10gR3/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/opt/oracle/10gR3/modules/net.sf.antcontrib_1.0b2.0/lib/ant-contrib.jar:/opt/oracle/10gR3/modules/features/aldsp.server.modules_3.2.0.0.jar:/opt/oracle/10gR3/odsi_10.3/lib/ld-server-core.jar:/opt/oracle/10gR3/wlserver_10.3/common/eval/pointbase/lib/pbclient51.jar:/opt/oracle/10gR3/wlserver_10.3/server/lib/xqrl.jar:/opt/oracle/10gR3/user_projects/lib/db2jcc.jar:/opt/oracle/10gR3/user_projects/lib/db2jcc_license_cisuz.jar:/opt/oracle/10gR3/properties -Dweblogic.system.BootIdentityFile=/opt/oracle/10gR3/user_projects/domains/DataFabricDomain/servers/member3MS1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -Xms2048m -Xmx2048m -Xmn640m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:-UseBiasedLocking -XX:ParallelGCThreads=16 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/logs/oracle/10gR3/DataFabricDomain/ManagedServer/member3agc.log_* -da -Dplatform.home=/opt/oracle/10gR3/wlserver_10.3 -Dwls.home=/opt/oracle/10gR3/wlserver_10.3/server -Dweblogic.home=/opt/oracle/10gR3/wlserver_10.3/server -Dwli.home=/opt/oracle/10gR3/wlserver_10.3/integration -Daldsp.home=/opt/oracle/10gR3/odsi_10.3 -Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl -Dweblogic.management.discover=false -Dweblogic.management.server=http://wdcsn443a.sys.cigna.com:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=true -Dweblogic.ext.dirs=/opt/oracle/10gR3/patch_wss110/profiles/default/sysext_manifest_classpath:/opt/oracle/10gR3/patch_wls1001/profiles/default/sysext_manifest_classpath:/opt/oracle/10gR3/patch_cie650/profiles/default/sysext_manifest_classpath:/opt/oracle/10gR3/patch_aldsp320/profiles/default/sysext_manifest_classpath -Dweblogic.system.BootIdentityFile=/opt/oracle/10gR3/user_projects/domains/DataFabricDomain/security/boot.properties -DDB2_USE_LEGACY_TOP_CLAUSE=true -Dlog4j.configuration=file:/opt/oracle/10gR3/user_projects/domains/DataFabricDomain/properties/log4j.xml -Ddeploymentsite=prod -DLOG4J_LEVEL=WARN -DLOG4J_ROOT=/logs/oracle/10gR3/DataFabricDomain -DLOG4J_NODENAME=member3a weblogic.Server
48461.245: [GC 48461.245: [*ParNew (promotion failed)*: 559017K->551408K(589824K), 1.1880458 secs]48462.433: [CMS: 1294242K->895754K(1441792K), 28.3698618 secs] 1852617K->895754K(2031616K), [CMS Perm : 122026K->120411K(262144K)], 29.5587684 secs] [Times: user=29.93 sys=0.04, real=29.56 secs]
Total time for which application threads were stopped: 29.5661221 seconds
109007.379: [GC 109007.380: [ParNew: 531521K->8922K(589824K), 0.0181922 secs] 1805634K->1283302K(2031616K), 0.0187539 secs] [Times: user=0.22 sys=0.01, real=0.02 secs]
Total time for which application threads were stopped: 0.0285263 seconds
Application time: 33.9224151 seconds
Total time for which application threads were stopped: 0.0086703 seconds
Application time: 8.5028806 seconds
109049.842: [GC 109049.842: [ParNew: 533210K->8861K(589824K), 0.0181380 secs] 1807590K->1283332K(2031616K), 0.0187288 secs] [Times: user=0.22 sys=0.01, real=0.02 secs]
Total time for which application threads were stopped: 0.0283473 seconds
Application time: 42.6375077 seconds
109092.508: [GC 109092.508: [ParNew: 533149K->8811K(589824K), 0.0161865 secs] 1807620K->1283418K(2031616K), 0.0167544 secs] [Times: user=0.19 sys=0.00, real=0.02 secs]
Total time for which application threads were stopped: 0.0264697 seconds
109122.582: [GC 109122.583: [*ParNew (promotion failed)*: 533099K->532822K(589824K), 1.2159460 secs]109123.799: [CMS: 1274986K->928935K(1441792K), 30.2900798 secs] 1807706K->928935K(2031616K), [CMS Perm : 127780K->126922K(262144K)], 31.5070045 secs] [Times: user=31.72 sys=0.04, real=31.51 secs]
Total time for which application threads were stopped: 31.5171276 seconds
Even though we cannot avoid fragmentation, what would be the best way to reduce these stop the world pauses?
Edited by: user12844507 on Mar 31, 2011 6:19 AM
Edited by: user12844507 on Mar 31, 2011 6:46 AMThe problem appears to be that the CMS work best if it can start before it is forced to start. The -XX:CMSInitiatingOccupancyFraction= determines at what point it should start before it is full. However it appear that this is too high, ie. you are creating objects too fast and it is running out of space before it finishes.
In particular you have "533099K->532822K(589824K)" which indicated to me you are filling the eden space with medium term lived objects very quickly. (More than 1/2 GB of them)
I would try to increase the young generation space until it appears to be too large. I would try "-XX:NewSize=2g -mx3g" to give it a much large younger generation space. This will stop some medium lived object being promopted and flooding the tenured space (which then have to be cleaned up and result in fragmentation)
Perhaps you have enough memory to try larger sizes. I use "-XX:NewSize=7g -mx8g" and I have no objects being prompted after startup.
BTW -mx == -Xmx
You might find this interesting http://blogs.sun.com/jonthecollector/entry/when_the_sum_of_the -
Hallo,
I got my first macbook some days ago and I see some strange behavior that I hope somebody knows and could explain.
The first boot out of the box was ok till I come to "user picture" setup window. From there, with camera active, the computer has started to take some long pauses: I often got the round rolling cursor and I just have to wait... but I don't know wait what. Anyway I have completed the setup and got OS X running. Unfortunally it was starting to do the same on the OS too; I have made a couple of restarts but nothing was changing.
So I take the OS X dvd and I made a new fresh installation of the os.
Things went much better from there...but not always.
Sometimes the round rolling curson comes out and I have to "wait for something". I have tried to use the monitor utility but I didn't find any service wasting cpu or memory when that happens.
I was thinking about hdd movement protection, but it happens when the notebook is on the desk too...
I was thinking about power saving shuting down the hdd when not in use, but I cannot hear start/stop noise and it takes too much time.
So I really don't understand what could be.
A couple of times things went more badly: I start to see the "wait cursor", than all starts to become slower...and finally the cursor hangs too, I cannot do anything.. just listen the fan running faster. I have tried the key combination to kill services but it doesn't work. So I just had to press power button for some seconds
Do anybody have some idea or suggestion about understanding this issue?
By my side I could give these more datails:
- I have installed the last version of parallels desktop;
- I have upgraded the RAM from 1 GB to 2GB, but the problem was the same before and after;
- I have run the hardware test form OS X DVD and no error was found.
thank youI have moved some steps ahead:
- I see that by changing network properties the locks frequency was changing, expecially when disabling wirelss connection;
- I have removed the HDD and replaced it with another one where I have reinstalled OS X while the computer was connected to the network (and to the Internet) by a wired connection. Using this new installation and using basic wireless settings, leaving default values and a computer to computer ad hoc wireless connection all has worked well for more then one day. I was able to do all without any pause, or the only pauses were due to hdd heavy usage by virtual machine installation under Fusion. OS X update was not installed.
After this "heavy" use of the notebook with this new hdd and no pauses at all, I have switched back to the original hdd: the computer have immediately started to slow down and leave me wait and wait...
To be really sure I have now booted in XP using bootcamp partition and I have installed and executed the Fujitsu HDD Diagnostic Tool with Extended Drive test. No errors were found on the drive.
I am now going to reinstall OS X on the original drive using wired network and default wireless setting to check if this will be enought to have the system work well like in the test hdd.
If this happens like I suppose and hope I am going to think that there are heavy problems with the wireless software on OS X. I will let you know.
Not really a very easy and fast start in using Mac computer and OS X... for now windows and pc are still better... macbook is nicer sure. -
How do I pause a download without a pause button?
I am downloading Adobe CS6 using Akamai NetSession Downloader but no pause button shows as illustrated in the Akamai Download Manager FAQ. How can I pause the download without a pause button? If I turn off my computer will the download fail?
Thanks! Sandra
Aaaah, Still no button but, while slowly moving the cursor around the area again, the hand lit up this time. And, though nothing showed where the cursor was, when I clicked, the download paused. And when I clicked again (over nothing) it started again. Whew!Hi,
Follow these steps:
1. Open the form
2. Select the File > Save as PDF form... menu item
3. When prompted, select the "Don't Collect Responses" option
The generated PDF will not contain a submit button.
Regards,
Brian -
Hi. How do I get to the start of a really long text message conversation without having to scroll through and load earlier messages repeatedly? Thanks
click on the top of the screen (the top bar/clock sectionish). Hope i helped
-
Windows 2012 Verification of prerequisites for Domain Controller promotion failed
Windows 2012 Verification of prerequisites for Domain Controller promotion failed and gave the below error(In computer management local group and user option is not there as suggested by a solution!)
"Verification of prerequisites for Domain Controller promotion failed. The local Administrator account becomes the domain Administrator account when you create a new domain. The new domain cannot be created because the local Administrator account password
does not meet requirements.
Currently, the local Administrator password is blank, which might lead to security issues. We recommend that you press Ctrl+Alt+Delete, use the net user command-line tool, or use Local Users and Groups to set a strong password for the local Administrator
account before you create the new domain."OK, the reason you see this error is because when you set up and configured your Windows R2 environment you may have logged into the OS with an account other than Administrator. So, if you created your log in account named Bob, this is throwing off the Server.
So, hit Ctrl-Alt-Delete, and look who you are logged in as, and then change the account you are logging in as and use the local Administrator account. What you may find is that the default Admin account password has not been set.
Check that out and see if that is what you are experiencing.
Best wishes -
Since I updated my Creative Cloud desktop App to its last version, my files are no longer synchronized. I received the "fail to synch files" and "server error" messages.
Hi, Jeff.
I'm not on a network and I didn't change anything on my secutiy setups, so I got in touch to the customer support. They checked my computer and found nothing wrong, so they uploded some log files to analyse the case. I'm waiting for a answer.
Thanks for the tips.
Maybe you are looking for
-
How do I uninstall a pirated version and load a good version from your site.
I thought that I was on an official Mozilla/Firefox website but apparently I was not. The FF I loaded immediately disabled pop-up control. Every second there were adds popping up from all sides. As fast as I got rid of them they reappeared. It looked
-
Hi I have downloaded 6 songs from Soundcloud. iCloud doesn't recognise them and i have the 'Waiting' iCloud symbol. How can i change those icons as i can't put any of the songs into a playlist?
-
How can I give an iBook I have finished to a daughter across the country?
iPad
-
Convert fireworks *.PNG to Dreamweaver xhtml file???
I have a fireworks *.png image which I would like to export into Dreamweaver MX. The fireworks artwork was provided by the designer to use in a Dreamweaver based website. This *.png file containes menues, images with their respective placeholders, te
-
Applying GPO Packs on Server 2012 R2 with MDT 2013
Hi All, I'm having a problem applying a GPO Pack to Server 2012 R2 via MDT 2013 LTI offline media. I'm using a standard server task sequence. I've copied my GPO Pack files (created with local GPO export) to the following directory: DeploymentShare\Te