CMS Rescan taking way too long - sometimes

I am running with the following environment:
Java HotSpot(TM) 64-Bit Server VM Version 1.5.0_14-b03 from Sun Microsystems Inc. on Linux Version 2.6.9-55.0.2.ELsmp on amd64 (quad processor)
My JVM args are:
-Xms12000m -Xmx12000m -XX:+UseConcMarkSweepGC -XX:+PrintGC XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Our app is a service which loads a lot of data and creates a lot of object (thus the 12G heap). Ordinarily, the CMS algorithm works quite nicely, we don't notice lengthy pauses, even on a rescan. However, today, we saw a big rescan that paused for 200 seconds.
I have put excerpts from the GC logging below and highlighted the problem in bold. Note, we were able to go from 8:26 AM until around 2:46 PM before the really long rescan hit, and it's not clear why it would have taken so long! During the CMS stages, there were cases of several GCs seen. I just put one and then put ... in this post to keep it short. If you need the entire thing let me know.
13030.340: [GC [1 CMS-initial-mark: 6125443K(12245440K)] 6126297K(12287936K), 0.0034960 secs]
13030.344: [CMS-concurrent-mark-start]
13030.462: [GC 13030.462: [ParNew: 42432K->0K(42496K), 0.0289210 secs] 6167875K->6130845K(12287936K), 0.0291580 secs]
First batch:
13046.113: [CMS-concurrent-mark: 13.399/15.768 secs]
13046.113: [CMS-concurrent-preclean-start]
13047.579: [CMS-concurrent-preclean: 1.457/1.466 secs]
13047.579: [CMS-concurrent-abortable-preclean-start]
13147.139: [GC 13147.139: [ParNew: 42432K->0K(42496K), 0.0215060 secs] 6517978K->6478076K(12287936K), 0.0217410 secs]
13537.285: [CMS-concurrent-abortable-preclean: 183.481/489.705 secs]
13537.285: [GC[YG occupancy: 21222 K (42496 K)]13537.285: [Rescan (parallel) , 0.0096050 secs]13537.295: [weak refs processing, 0.1354430 secs] [1 CMS-remark: 6478076K(12245440K)] 6499298K(12287936K), 0.2196620 secs]
13537.505: [CMS-concurrent-sweep-start]
13543.140: [CMS-concurrent-sweep: 5.631/5.635 secs]
13543.140: [CMS-concurrent-reset-start]
13543.312: [CMS-concurrent-reset: 0.172/0.172 secs]
13857.150: [GC 13857.150: [ParNew: 42432K->0K(42496K), 0.0259620 secs] 2243371K->2200951K(12287936K), 0.0262160 secs]
Next batch:
13898.547: [GC [1 CMS-initial-mark: 2367861K(12245440K)] 2368713K(12287936K), 0.0026800 secs]
13898.550: [CMS-concurrent-mark-start]
13898.651: [GC 13898.651: [ParNew: 42432K->0K(42496K), 0.0227580 secs] 2410293K->2373310K(12287936K), 0.0229820 secs]
13907.339: [CMS-concurrent-mark: 7.155/8.789 secs]
13907.339: [CMS-concurrent-preclean-start]
13907.389: [GC 13907.389: [ParNew: 42432K->0K(42496K), 0.0278740 secs] 2805295K->2771872K(12287936K), 0.0281080 secs]
13911.561: [CMS-concurrent-preclean: 3.124/4.222 secs]
13911.561: [CMS-concurrent-abortable-preclean-start]
13911.637: [CMS-concurrent-abortable-preclean: 0.072/0.076 secs]
13911.638: [GC[YG occupancy: 37493 K (42496 K)]13911.638: [Rescan (parallel) , 0.0244660 secs]13911.662: [weak refs processing, 0.0019580 secs] [1 CMS-remark: 3010929K(12245440K)] 3048423K(12287936K), 0.0266320 secs]
13911.665: [CMS-concurrent-sweep-start]
13911.677: [GC 13911.677: [ParNew: 42432K->0K(42496K), 0.0257240 secs] 3053238K->3019738K(12287936K), 0.0259860 secs]
13917.256: [CMS-concurrent-sweep: 4.651/5.592 secs]
13917.256: [CMS-concurrent-reset-start]
13917.348: [CMS-concurrent-reset: 0.091/0.091 secs]
13917.574: [GC 13917.574: [ParNew: 42432K->0K(42496K), 0.0445350 secs] 2894255K->2860507K(12287936K), 0.0447740 secs]
Next batch:
13917.620: [GC [1 CMS-initial-mark: 2860507K(12245440K)] 2861359K(12287936K), 0.0026680 secs]
13917.623: [CMS-concurrent-mark-start]
13917.827: [GC 13917.827: [ParNew: 42432K->0K(42496K), 0.0226630 secs] 2902939K->2863049K(12287936K), 0.0228990 secs]
13925.670: [CMS-concurrent-mark: 7.171/8.048 secs]
13925.670: [CMS-concurrent-preclean-start]
13925.800: [GC 13925.800: [ParNew: 42432K->0K(42496K), 0.0197400 secs] 2977148K->2936020K(12287936K), 0.0199660 secs]
13926.922: [CMS-concurrent-preclean: 1.012/1.251 secs]
13926.922: [CMS-concurrent-abortable-preclean-start]
13927.065: [CMS-concurrent-abortable-preclean: 0.135/0.143 secs]
13927.066: [GC[YG occupancy: 27276 K (42496 K)]13927.066: [Rescan (parallel) , 0.0128830 secs]13927.079: [weak refs processing, 0.0056630 secs] [1 CMS-remark: 2945981K(12245440K)] 2973257K(12287936K), 0.0187590 secs]
13927.085: [CMS-concurrent-sweep-start]
13927.163: [GC 13927.163: [ParNew: 42432K->0K(42496K), 0.0226630 secs] 2985423K->2945599K(12287936K), 0.0228920 secs]
13934.091: [CMS-concurrent-sweep: 6.277/7.007 secs]
13934.091: [CMS-concurrent-reset-start]
13934.165: [GC 13934.165: [ParNew: 42432K->0K(42496K), 0.0187220 secs] 2181328K->2139713K(12287936K), 0.0189510 secs]
13934.202: [CMS-concurrent-reset: 0.091/0.111 secs]
13934.308: [GC 13934.308: [ParNew: 42432K->0K(42496K), 0.0190990 secs] 2182145K->2140530K(12287936K), 0.0193180 secs]
Next batch:
17931.953: [GC [1 CMS-initial-mark: 3128945K(12245440K)] 3128945K(12287936K), 0.0029070 secs]
17931.956: [CMS-concurrent-mark-start]
17932.075: [GC 17932.075: [ParNew: 42432K->0K(42496K), 0.0188230 secs] 3171377K->3135061K(12287936K), 0.0190740 secs]
17950.186: [CMS-concurrent-mark: 16.599/18.230 secs]
17950.186: [CMS-concurrent-preclean-start]
17950.398: [GC 17950.398: [ParNew: 42432K->0K(42496K), 0.0175640 secs] 3622038K->3583691K(12287936K), 0.0178560 secs]
17952.374: [CMS-concurrent-preclean: 2.040/2.188 secs]
17952.374: [CMS-concurrent-abortable-preclean-start]
17952.416: [CMS-concurrent-abortable-preclean: 0.039/0.041 secs]
17952.417: [GC[YG occupancy: 37468 K (42496 K)]17952.417: [Rescan (parallel) , 0.0152990 secs]17952.432: [weak refs processing, 0.0108270 secs] [1 CMS-remark: 3602857K(12245440K)] 3640325K(12287936K), 0.0314730 secs]
17952.448: [CMS-concurrent-sweep-start]
17952.482: [GC 17952.482: [ParNew: 42432K->0K(42496K), 0.0184580 secs] 3645198K->3607387K(12287936K), 0.0187530 secs]
17954.512: [CMS-concurrent-sweep: 1.940/2.064 secs]
17954.513: [CMS-concurrent-reset-start]
17954.603: [CMS-concurrent-reset: 0.090/0.090 secs]
17954.890: [GC 17954.890: [ParNew: 42432K->0K(42496K), 0.0179710 secs] 3001972K->2965101K(12287936K), 0.0182600 secs]
Final, problematic batch: 206 second rescan!!!
22786.693: [GC [1 CMS-initial-mark: 3509569K(12245440K)] 3509613K(12287936K), 0.0028590 secs]
22786.696: [CMS-concurrent-mark-start]
22787.039: [GC 22787.040: [ParNew: 40489K->0K(42496K), 0.0198730 secs] 3550058K->3518243K(12287936K), 0.0201500 secs]
22787.445: [GC 22787.445: [ParNew: 42432K->0K(42496K), 0.0377070 secs] 3560675K->3535588K(12287936K), 0.0379940 secs]
22787.805: [GC 22787.806: [ParNew: 41936K->0K(42496K), 0.0276640 secs] 3577524K->3559012K(12287936K), 0.0279380 secs]
22796.052: [CMS-concurrent-mark: 9.267/9.356 secs]
22796.052: [CMS-concurrent-preclean-start]
22796.286: [CMS-concurrent-preclean: 0.234/0.234 secs]
22796.286: [CMS-concurrent-abortable-preclean-start]
22878.302: [GC 22878.303: [ParNew: 42432K->0K(42496K), 0.0585960 secs] 3601444K->3570279K(12287936K), 0.0589470 secs]
23227.292: [CMS-concurrent-abortable-preclean: 161.943/431.006 secs]
*23227.293: [GC[YG occupancy: 21261 K (42496 K)]23227.293: [Rescan (parallel) , 206.0731410 secs]23433.366: [weak refs processing, 0.0137160 secs] [1 CMS-remark: 3570279K(12245440K)] 3591541K(12287936K), 206.0871880 secs]*
23433.380: [CMS-concurrent-sweep-start]
2008-02-14 14:57:15,031 WARN [MonitorTimerThread] LM1: OFFLINE
23436.204: [CMS-concurrent-sweep: 2.824/2.824 secs]
23436.204: [CMS-concurrent-reset-start]
23436.293: [CMS-concurrent-reset: 0.089/0.089 secs]
2008-02-14 14:57:51,160 INFO [RequestConsumer:10001-8] Handling Report Request: Description=[REGENERATE] Context=[a213976@MMKA850246] Date=[02/13/2008]
2008-02-14 14:57:51,163 INFO [RequestConsumer:10001-8] Generating Report Request: Description=[REGENERATE] Context=[a213976@MMKA850246] Date=[02/13/2008]
23469.633: [GC 23469.633: [ParNew: 42432K->0K(42496K), 0.0240360 secs] 2923658K->2888635K(12287936K), 0.0243860 secs]

Can you file a bug with Sun under
product: java
category: hotspot
subcategory: garbage_collector
and include the complete logs?
Or submit them through your Sun support contact and ask them to
create the bug?

Similar Messages

Installing OS X on a new hard drive taking way too long?

I switched out my old 250 GB hard drive that shipped with my MacBook Pro for a new 640 GB WesternDigit drive. I'm trying to use a Mavericks installer USB to install on my newly formatted hard drive, but it's taking way too long - I mean, I left it going overnight (about 7.5 hours), and when I came down, it wasn't even half done. Is there anything I can do here? I'd like to use my laptop. Thank you very much for any suggestions.

Do you have a internet connection? Even with a USB installer that if good and functioning properly you need an internet connection to Authenticate the install with your Apple ID.
If your old drive is still functioning you can connect it to your Mac with a SATA to USB adapter and use the Recovery HD on that original drive to reinstall Mavericks on the new drive.
If the drive is no longer functioning and you have no way of getting to the Apple internet download for Mavericks then you will need to start with an install of the original version of OS X that came on your Mac when new then update and or upgrade to Snow Leopard 10.6.8 and then use the Mac App Store to reinstall Mavericks. There is not other way to do this. So save yourself some time and trouble and start fresh.
If your original drive is still functioning and it had Mavericks installed on it you can also CLONE that drive to the new one once you connect it to your mac with some type of SATA to USB adapter or enclosure.
If the Mavericks USB installer you have was made from a Mac that came with Mavericks pre-installed on it that will not work on your Mac.

It's taking way too long for my messages to load.

Starting on Tuesday this week (10/14) it is taking way too long for my messages to load from my Gmail account. Once the message finally shows up in the Inbox, it's taking up to 10-15 min for the message to load where I can actually view the message. I'm going to have to move to another email application if this can't be resolved which is very unfortunate as I've enjoyed using Thunderbird up until now.

I guess I'm at least happy to know it's not just me. I've had to abandon Thunderbird for now. I guess I'll keep an eye out for future updates.

Firefox is taking way too long to load pages. I have a very fast internet connection but firefox takes forever to load websites. At first it worked fine and very fast but now very slow??

firefox is taking way too long to load pages. I have a very fast internet connection but firefox takes forever to load websites. At first it worked fine and very fast but now very slow??

Hi there!
do you test other browsers?
i suggest to make a clean install of newer version of Firefox like v8.0.

Taking way too long to iPhone for iOS 5

Taking too long to sinc iPhone for ios5

What it has to do is install the framworks and then reload all of your data onto the iPhone. It took me 2 hours to install on my iPad but it took 15 min for my iPhone

OSX installation (and reloading images) taking way too long

Hi everyone.
I've noticed a problem lately: installing OSX (either my original Tiger DVD or my new Leopard DVD) is taking way longer than it should. The same thing also happens when I try to revert back to an older disk image.
I tried installing Leopard yesterday and it said it was going to take 5 hours. Initially I thought it was because the disk I was using was partitioned, so today I have wiped the disk completely, restored it to one partition, and disconnected all my external drives and all but the primary internal ones. That seems to have dropped the install time to 3 hours.
The first thing I did yesterday was try to restore an older image of my Tiger installation. That one was about 60GB in size and, even after 8 hours, the installation had not predicted an actual remaining install time, but from the progress bar, it looked like it would be at least 2 days.
Why in the world would this be happening and has anyone run into it before? All the system checks I ran the other day seemed to indicate things were fine. So am I missing something?
Thanks.
-N

Check the S.M.A.R.T. status of the HDs.
Run DU and repair the HDs if needed.
It's also possible that your optical drive is dying and/or your Tiger CD and Leopard DVDs are dirty or scratched.
A typical Leopard install takes under 1 hour. Anything over 2 indicates some major issues.

Slideshow Taking WAY too long to load

I have a slideshow posted at
http://olyrents.com/slideshow.htm
My client is saying it takes too long to load, and I am
finding that it is loading all of the photos first before you can
scroll through and view them. It's just not functioning properly.
I wonder if someone could look at the actionscript and give
me any suggestions? Maybe there's something else going on?
I have attached the script from the original fla file.
Thanks!

That adds up to somewhere in the realm of 5MB, which isn't
going to be a quick load time, though it might be visually workable
if you separate the loading from the viewing.
One thing you might consider is doing away with the preloader
visual. With 65 images being interupted by 65 preloader screens, it
kinda defeats the potential aethetics of the images and may lead
one to think it's taking a long time to load them as a result.

Query taking way too long

Hello,
Can somebody please help.
I am wanting to change this query so that it does not take too long. How can I do it please?
select count(*), t.tel_no, t.prospect_id, a.ACCOUNT_NO, a.account_title
from tel_nos t, account a
where t.tel_type = 'S'
AND t.prospect_id = a.prospect_id
AND SYSDATE BETWEEN t.start_date AND NVL(t.end_date, SYSDATE)
group by t.tel_no, t.prospect_id, a.ACCOUNT_NO, a.account_title
having count(*) > 1
Thanks in advance.

Tuning is a science not a parlour guessing game. It's hard for us to diagnose the problem you have here as you have given us almost no information. For instance, are you using rule based or cost based optimizer? How big are these two tables? What indexes do they have? What percentage of rows are you expecting the query to return?
Tuning is such a big area that it's hard to know where to begin. I suggest you start at the Oracle docs. Chapters 1, 4 and 5 will set you going.
http://download-west.oracle.com/docs/cd/A87860_01/doc/server.817/a76992/toc.htm
Having said all that you may well find this runs faster:
SELECT count(*), t.tel_no, t.prospect_id, a.account_no, a.account_title
FROM   account a, tel_nos t
WHERE t.tel_type = 'S'
AND    sysdate BETWEEN t.start_date AND nvl(t.end_date, sysdate)
AND    t.prospect_id = a.prospect_id
GROUP BY t.tel_no, t.prospect_id, a.account_no, a.account_title
HAVING count(*) > 1
/I leave it to you to work out why (it's nothing to do with clearer layout).
Cheers, APC

Mail taking way too long to get to my iphone...Rogers network Toronto

Today, on more than one occasion I sent an email from my yahoo account to my .mac account. The email arrived within 2 minutes. I was online the entire time to verify. My iphone, failed to be notified within 30 minutes!! In actual fact, i don;t know how long it would have taken to get the email. The only way I managed to retrieve the email on my iphone was to select the mail application on the iphone and select "inbox". This forces the software to check for mail....This isn't PUSH at all.
Is anyone else experiencing this.?
Thanks
B

the push services are only as good as the pusher I believe. I know my yahoo mail doesn't get pushed very well. Sometimes it's instant other times it's not. My girlfriend's email on a microsoft exchange server activesync is pushed pretty consistently. Given some of the hangups with the mobileme services, I'm not too surprised that the push isn't working great. It's really a mobilme issue I think, and I will let others with the service chime in with their experiences.

HT1689 It is taking way too long to download my purchase, what's going on?

My tv program download is taking too long

Your TV program may be huge ... like a gigabyte or so. That could take a very long time to download, depending on your WiFi connection speed and quality.

BUNDLE_EXECUTE TAKING WAY TOO LONG TIME. PLEASE HELP

Hi Experts,
Please see that after we ugrade our SEM 4 to SEM 6 along with BW 3.5 to BI 7, I am performing Post Upgrade Testing in Sandbox.
In this process when I tried executing a Global Planning Sequence in Back Ground by Right Clicking the GPS I wish to run and selected Execute PS in Background Option.
But when I am looking at it's status in Job overview, it is running for ever. I still wonder that how come it is taking so long time and I have no clue.
The same GPS when I tried executing the ssame way in DEV and QA servers where the Upgrade is still not happened it is not taking more than 80 Seconds.
So Kindly guide me through this issue. Highly appreciate your valuble inputs.
Thanks and Regards,
BIP

Hi ,
We are facing the same issue in our new parallel systems (SEM 3.2 to SEM 6 along with BW 3.1 to BI 7). Please let us know how you solved the issue.
Our Planning Sequence has just one copy function.
The same GPS when I tried executing in old systems in SEM 3.2 and BW 3.1, it is working fine.
Any solution?
Thanks,
H

Updating 3GS from 3.1.2 to 6.1.3 taking way too long. Is there a better and faster way?

How long does it take to update an original 3GS to 6.1.3? I followed the iTune instructions and it has been updating for hours already. Is there a better and faster way to do this?

SO if I want to represent the London Underground
network as a Graph how would I be able to do it?By writing classes of your own or using some one else's. I don't think that the collections framework has out-of-the-box support for graphs.... So try google first.
The implementation details depend on the API you are going to use, of course...
from A to B 5 min
from B to C 4 min
from A to C 9 min
this information is redundant, the third info should be omitted. <nit-picking>
No it's not - What if there's a special, more direct route from A to C? Also, getting from A to B to C will usually take more time than the time from A to B plus the time from B to C. The trains have to stop at B, don't they?
</nit-picking>
Maybe this approach is more complex, but it solves the problem Have I understood correctly - isn't what you describe just another way to implement a graph?

Shuffle taking way too long to sync

after updating to itunes 8.o, all of the three shuffles that use that itunes library take forever to sync-even one song or podcast. what happened?? does anyone know how to fix this??

Even though the Shuffle is listed as USB 2.0 compatible it only seems to sync at USB 1.1 speeds. I've had a nano and a shuffle and the nano syncs WAY faster than shuffle (no conversion to 128kbps AAC is taking place)
Message was edited by: ajstahl

OS X Mountain Lion...taking way too long

I purchased OS X Mountain Lion for my iMac. iMac meets all the requirements MAC OS X 10.7.4. It's been 22 hours and it isn't doing anything, sitting at 1.86 GB of 4.37 GB. Why is the download taking so long? Is there something else I should be doing?

Thank you for your reply...I do believe the WiFi connection in our neighborhood could be the factor here...congestion.
Browsing has never been an issue...but downloading something this large obviously is.
I have paused the download and resumed...since I posted earlier, it is now at 2.22 GB of 4.37 GB ... moving VERY slowly.
Not sure what to do at this point...I will call the support line during wrkg hrs. See if they can cancel my online pymt and I'm assuming I can purchase the OS X Mountain Lion at Best Buy???
The neighborhood is waking up soon...and more congestion will be interfering soon.

JDBC Connection taking way too long

Hi,
I have an odd problem which has just started to happen. I am creating a very simple JDBC connection to the MS SQL Server database using the Microsoft JDBC driver. This has always worked for me in the past and has never caused any problems. Now, however, creating a connection to my database is taking about 20 seconds! This is usually a sub-second operation. I have tried to narrow down the cause, but everything I have tried has failed. Here are all the things I have tried so far:
1. Using a different JDBC driver (several, including the ODBC bridge)
2. Create a test app which just has my connection code and run this from different machines on the network under different JDKs
3. Re-installed by SQL Server database
4. Connected using IP address rather than machine name (in case it was a DNS issue of some sort)
5. Connected using different login accounts
6. Vertified connectivity using telnet (port 1433)
7. Verified connectivity via ODBC in an ASP page
If I telnet to the SQL Server service I can connection without any delay. I can also connect immediately using an ODBC connection via an ASP page, so I don't think it's a problem with my database. I just seems that connectivity from Java land of woefully slow. I have never had this problem before, it's only started happening recently. The only thing that has changed recently is that I signed up for a new ADSL provider, but I can't see how this could have any bearing as all the machines are internal and connected by a 100Mbps router!
Here's the code I am using to test...
long current = System.currentTimeMillis();
Class.forName("com.microsoft.jdbc.sqlserver.SQLServerDriver").newInstance();
Connection connection = DriverManager.getConnection("jdbc:microsoft:sqlserver://<server-ip>:1433;User=<usr>;Password=<pwd>", null);
System.out.println(System.currentTimeMillis() - current);
connection.close();This gives me times of about 20 seconds.
Help!

You are right.. but the fact that:
a) I can telnet and connect instantly
b) The jdbc-odbc bridge connects immediately
.. tells me that the network is ok.. at least for native functions. The only consistency I can find is that native (windows) functions connect without any issue, but pure Java functions experience problems. It may well be an issue with my network, but it seems that the issue is isolated to the Java environment (guessing here).
I tested this theory by simply opening a socket connection from Java, but has not problems (connected immediately).
I tried switching on the debug messages in the DriverManager (DriverManager.setLogWriter(writer))... this didn't tell me anything.
... The ONLY thing I can think of trying is disconnecting from my ADSL router and trying a simple cross-over cable from one machine to the other.
Thanks for your help though.

CMS Rescan taking way too long - sometimes

Similar Messages

Maybe you are looking for