2.6.18-emission1: Pathological Euphoria

E-Mission Sources is based on the Lockless Pagecache by Nick Piggin, GenPatches, and CK.
The main goal of this patchset is to stay updated with gentoo-sources releases,
offer latest features, decrease latencies, decrease overhead, and improve interactivity. Another
goal of E-Mission Sources is stability. We boot test each release and run for an extended amount of time.
/* Notes */
New version of genpatches, more Thinkpad related patches, added grsecurity for security conscious users, and alot of cleanups and bug fix patches.
All problems you have been having with -emission4 and -emission5 are cleared up with this release.
2.6.17-emission6: Shock Treatment
/* Patch List */
# Latest Linux Kernel Patch
patch-2.6.17.7
patch-2.6.17.8-rc1
# Latest Gentoo Patches
1700_sparc-obp64-naming.patch
1705_sparc-U1-hme-lockup.patch
2000_ethtool-oops.patch
2300_usb-insufficient-power.patch
4000_deprecate-sk98lin.patch
4005_bcm4319.patch
4010_pcnet-cs-te-cf100.patch
4015_forcedeth-new-ids.patch
4020_asix-88178.patch
4025_r8169-new-id.patch
4030_tg3-5787.patch
4035_sky2-v1.5.patch
4100_vt8251-sata.patch
4105_dm-bbr.patch
4110_nvidia-mcp61.patch
4115_nvidia-sata-new.patch
4120_ahci-nvidia-mcp65.patch
4125_nvidia-ide-new.patch
4130_jmicron-ahci.patch
4135_promise-pdc2037x.patch
4200_fbsplash-0.9.2-r5.patch
4205_vesafb-tng-1.0-rc2.patch
4206_vesafb-tng-mtrr.patch
4300_squashfs-3.0.patch
4400_speakup-20060618.patch
4401_speakup-serio.patch
4405_alpha-sysctl-uac.patch
# Latest -ck
sched-implement-smpnice-2.6.17.patch
sched-revise_smt_nice_locking.patch
2.6.17-smpnice-staircase-16.patch
sched-staircase16_interactive_tunable.patch
sched-staircase16_compute_tunable.patch
sched-range.patch
sched-iso-4.5.patch
track_mutexes-1.patch
sched-idleprio-1.9.patch
sched-limit_policy_changes.patch
defaultcfq.diff
cfq-ioprio_inherit_rt_class.patch
cfq-iso_idleprio_ionice.patch
hz-default_1000.patch
hz-no_default_250.patch
sched-add-above-background-load-function.patch
mm-swap_prefetch-32.patch
swsusp-rework-memory-shrinker-rev-2.patch
mm-convert_swappiness_to_mapped.patch
mm-lots_watermark.diff
mm-kswapd_inherit_prio-1.patch
mm-prio_dependant_scan-1.patch
mm-background_scan-1.patch
mm-idleprio_prio.patch
mm-decrease_minimum_dirty_ratio.patch
mm-set_zero_dirty_ratio.patch
mm-filesize_dependant_lru_cache_add.patch
kconfig-expose_vmsplit_option.patch
# Fixes from -mm
sleazy-fpu-feature-i386-support.patch
sleazy-fpu-feature-x86_64-support-2.patch
2.6.17-mm-mutex-conversion.patch
2.6.17-mm1-rollup-cleanup-patches.patch
tty-remove-include-of-screen_infoh-from-ttyh-fix.patch
tty-remove-include-of-screen_infoh-from-ttyh-fix-fix.patch
kernel-sysc-doesnt-need-inith.patch
the-scheduled-removal-of-some-oss-drivers.patch
zlib_inflate-update.patch
# Latest Reiser4 Patch from Namesys
reiser4-for-2.6.17-3.patch
# IOSched Improvements by Jens Axboe
iosched-rollup-2.6.17.4-2.patch
# Thinkpad related
hdaps_protect.20060430.patch
04-ibm_acpi-fan-for-2.6.17.patch
05-ibm_acpi-thermal-for-2.6.17.patch
linux-phc-0.2.6-kernel-vanilla-2.6.17.patch
10-tp_smapi-0.27-for-2.6.17.patch
# Unionfs 1.3
unionfs-1.3.patch
# Suspend2 2.2.7.4
suspend2-2.2.7.4-2.6.17-r2.patch
# Realtime Linux Security Model
realtime-lsm.patch
# patches from -tiny
cpu-support-rollup.patch
# Menuconfig Themes/Support
# make MENUCONFIG_COLOR=blackbg menuconfig
# make MENUCONFIG_COLOR=classic menuconfig
lxdialog-refractor-color-support.patch
lxdialog-color-themes-blackbg.patch
lxdialog-color-themes-classic.patch
daconfig-2.3.1-2.6.17-rc4
usb-mouse-polling.patch
name-tag.patch
# Hardened
grsecurity-2.1.9-2.6.17.7-200608012135.patch
remove-grsecurity-tag.patch
# Nick Piggins Patches
radix-tree-small.patch
mm-non_syncing_lock_page.patch
lockless.patch
lockless-fix.patch
Enjoy!
Patch:
http://distfiles.evolution-mission.org/ … ssion6.bz2
PKGBUILD (Thanks djpharoah!):
http://distfiles.evolution-mission.org/ … /PKGBUILD/
Lockless Patch (Experimental)
http://distfiles.evolution-mission.org/ … less.patch
Broken-Out:
http://distfiles.evolution-mission.org/ … ut.tar.bz2
Broken-Out (Directory):
http://distfiles.evolution-mission.org/ … broken-out
Making it work:
Manual Method:
cd /usr/src
rm linux
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.tar.bz2
tar -xvjf linux-2.6.17.tar.bz2
mv linux-2.6.17 linux-2.6.17-emission6
ln -s linux-2.6.17-emission6 linux
cd linux
wget http://distfiles.evolution-mission.org/sources/2.6.17/emission6/linux-2.6.17-emission6.bz2
wget http://distfiles.evolution-mission.org/sources/2.6.17/emission6/linux-2.6.17-emission6-lockless.patch (Experimental)
bzcat linux-2.6.17-emission6.bz2 | patch -p1
patch -p1 < ./linux-2.6.17-emission6-lockless.patch
rm -rf *.bz2 *.patch
make MENUCONFIG_COLOR=blackbg menuconfig
For all support, requests, patches, please check the Emission-Sources Bugzilla Tracker
and check Emission-Sources Forums for new releases!

E-Mission Sources is based on the Lockless Pagecache by Nick Piggin, GenPatches, and CK.
The main goal of this patchset is to stay updated with genpatches releases,
offer latest features, decrease latencies, decrease overhead, and improve interactivity. Another
goal of E-Mission Sources is stability. We boot test each release and run for an extended amount of time.
/* Notes */
Latest kernel patch, new version of genpatches, new version of lockless, and alot of cleanups and bug fixes.
New feature implemented called: "E-Mission Tunables". It was derived from the idea of the customizable timeslices for Staircase that Tiger683 had released for nitro-sources. What I did was, I took that idea, and moved it to the next level. I do realize that these tunables could be set in sysctl.conf, but I personally like the idea of my own custom starting point. I would appreciate your input on this new feature.
Example of new E-Mission Tunables (All but Custom are preconfigured options):
E-Mission Tunables --->
( ) Server
( ) File Server
( ) Compute Server
( ) Desktop
( ) Low Latency Desktop
( ) Custom
Custom Menu:
E-Mission Kernel Tunables (Custom) --->
(6) Minimum Round Robin Timeslice (in milliseconds) (NEW)
(19) Factor For Default Value (NEW)
(1) Interactive CPU Scheduling (NEW)
(0) Compute CPU Scheduling (NEW)
(80) Percentage CPU For SCHED_ISO Tasks (NEW)
(66) Percentage RAM Filled With Mapped Pages (NEW)
(1) Hard Mapped Limit (NEW)
(1) Tail Large Files (NEW)
(0) Dirty Ratio (NEW)
2.6.17-emission7: The Bullet Dodger
/* Patch List */
001-patch-2.6.17.11
002-genpatches-rollup-2.6.17-6.patch
003-patch-2.6.17-ck1
004-lockless-from-2.6.18-rc4-mm1.patch
005-iosched-rollup-2.6.17.4-2.patch
006-reiser4-for-2.6.17-3.patch
007-2.6.17-kj-resync-from-rc1-kj.patch
008-cpu-support-rollup.patch
009-linux-phc-0.2.6-kernel-vanilla-2.6.17.patch
010-realtime-lsm.patch
011-2.6.17-mm1-rollup-cleanup-patches.patch
012-2.6.17-mm-mutex-conversion.patch
013-sleazy-fpu-feature.patch
014-grsecurity-2.1.9-2.6.17.7-200608012135.patch
015-suspend2-2.2.7.4-2.6.17-r2.patch
016-zlib_inflate-update.patch
017-the-scheduled-removal-of-some-oss-drivers.patch
018-unionfs-1.3.patch
019-shutup-false-positives-2.6.17.patch
020-mark-false-positives-2.6.17-emission7.patch
021-emission-tunables-3-2.6.17-ck1.patch
022-kconfig-add-864hz.patch
023-lxdialog-from-2.6.18-rc4-mm1.patch
024-daconfig-2.3.1-2.6.17.patch
025-name-tag-2.6.17-emission7.patch
Enjoy!
Patch:
http://distfiles.evolution-mission.org/ … ssion7.bz2
PKGBUILD:
Broken-Out:
http://distfiles.evolution-mission.org/ … ut.tar.bz2
Broken-Out (Directory):
http://distfiles.evolution-mission.org/ … broken-out
Making it work:
Manual Method:
cd /usr/src
rm linux
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.tar.bz2
tar -xvjf linux-2.6.17.tar.bz2
mv linux-2.6.17 linux-2.6.17-emission7
ln -s linux-2.6.17-emission7 linux
cd linux
wget http://distfiles.evolution-mission.org/sources/2.6.17/emission7/linux-2.6.17-emission7.bz2
bzcat linux-2.6.17-emission7.bz2 | patch -p1
rm -rf *.bz2 *.patch
make MENUCONFIG_COLOR=blackbg menuconfig
For all support, requests, patches, please check the Emission-Sources Bugzilla Tracker
and check Emission-Sources Forums for new releases!

Similar Messages

Pathological ParallelGC performance w/ big long-lived object (512MB array)

Hoping to improve performance, we recently added a bloom filter -- backed by a single long-lived 512MB array -- to our application. Unfortunately, it's killed performance -- because the app now spends ~16 of every ~19 seconds in garbage collection, from the moment the big array is allocated.
My first theory was that the array was stuck in one of the young generations, never capable of being promoted, and thus being endlessly copied back and forth on every minor young collection. However, some tests indicate the big array winds up in "PS Old" right away... which would seem to be a safe, non-costly place for it to grow old. So I'm perplexed by the GC performance hit.
Here's the tail of a log from a long-running process -- with UseParallelGC on a dual-opteron machine running 32bit OS/VM -- showing the problem:
% tail gc.log
697410.794: [GC [PSYoungGen: 192290K->2372K(195328K)] 1719973K->1535565K(1833728K), 16.4679630 secs]
697432.415: [GC [PSYoungGen: 188356K->1894K(194752K)] 1721549K->1536592K(1833152K), 16.4797510 secs]
697451.419: [GC [PSYoungGen: 188262K->4723K(195200K)] 1722960K->1540085K(1833600K), 16.4797410 secs]
697470.817: [GC [PSYoungGen: 191091K->1825K(195520K)] 1726453K->1541275K(1833920K), 16.4763350 secs]
697490.087: [GC [PSYoungGen: 189025K->8570K(195776K)] 1728475K->1550136K(1834176K), 16.4764320 secs]
697509.644: [GC [PSYoungGen: 195770K->5651K(192576K)] 1737336K->1555061K(1830976K), 16.4785310 secs]
697530.749: [GC [PSYoungGen: 189203K->1971K(194176K)] 1738613K->1556430K(1832576K), 16.4642690 secs]
697551.998: [GC [PSYoungGen: 185523K->1716K(193536K)] 1739982K->1556999K(1831936K), 16.4680660 secs]
697572.424: [GC [PSYoungGen: 185524K->4196K(193984K)] 1740807K->1560197K(1832384K), 16.4727490 secs]
I get similar results from the moment of launch on another machine, and 'jmap -heap' (which isn't working on the long-lived process) indicates the 512MB object is in 'PS Old' right away (this is from a quick launch of a similar app):
jdk1.5.0_04-32bit/bin/jmap -heap 10586Attaching to process ID 10586, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_04-b05
using thread-local object allocation.
Parallel GC with 2 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1887436800 (1800.0MB)
NewSize = 655360 (0.625MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 1441792 (1.375MB)
NewRatio = 8
SurvivorRatio = 8
PermSize = 16777216 (16.0MB)
MaxPermSize = 67108864 (64.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 157286400 (150.0MB)
used = 157286400 (150.0MB)
free = 0 (0.0MB)
100.0% used
From Space:
capacity = 26214400 (25.0MB)
used = 26209080 (24.99492645263672MB)
free = 5320 (0.00507354736328125MB)
99.97970581054688% used
To Space:
capacity = 26214400 (25.0MB)
used = 1556480 (1.484375MB)
free = 24657920 (23.515625MB)
5.9375% used
PS Old Generation
capacity = 1677721600 (1600.0MB)
used = 583893848 (556.8445663452148MB)
free = 1093827752 (1043.1554336547852MB)
34.80278539657593% used
PS Perm Generation
capacity = 16777216 (16.0MB)
used = 10513680 (10.026626586914062MB)
free = 6263536 (5.9733734130859375MB)
62.66641616821289% used
The 'PS Old' generation also looks way oversized here -- 1.6G out of 1.8G! -- and the young/tenured starved, although no non-default constraints have been set on generation sizes, and we had hoped the ballyhooed 'ergonomics' would've adjusted generation sizes sensibly over time.
'-XX:+UseSerialGC' doesn't have the problem, and 'jmap -heap' suggests the big array is in the tenured generation there.
Any ideas why UseParallelGC is behaving pathologically here? Is this, as I suspect, a bug? Any suggestions for getting it to work better through VM options? (Cap the perm size?)
Any way to kick a running ParallelGC VM while it's running to resize it's generations more sensibly?
I may also tweak the bloom filter to use a number of smaller -- and thus more GC-relocatable -- arrays... but I'd expect that to have a slight performance hit from the extra level of indirection/indexing, and it seems I shouldn't have to do this if the VM lets me allocate a giant object in the first place.
Thanks for any tips/insights.
- Gordon @ IA

Yes, in my app, the large array is updated constantly.
However, in the test case below, I'm getting similar behavior without any accesses to the big array at all.
(I'll file this test case via the bug-reporting interface as well.)
Minimal test case which seems to prompt the same behavior:
* Demonstrate problematic ParallelGC behavior with a "big" (512MB)
* object. (bug-id#6298694)
* @author gojomo/archive,org
public class BigLumpGCBug {
int[] bigBitfield;
public static void main(String[] args) {
(new BigLumpGCBug()).instanceMain(args);
private void instanceMain(String[] args) {
bigBitfield = new int[Integer.MAX_VALUE>>>4]; // 512MB worth of ints
while(true) {
byte[] filler = new byte[1024*1024]; // 1MB
Run with java-options "-Xmx700m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps", the GC log is reasonable:
0.000: [GC 0.001: [DefNew: 173K->63K(576K), 0.0036490 secs]0.005: [Tenured: 39K->103K(1408K), 0.0287510 secs] 173K->103K(1984K), 0.0331310 secs]
2.532: [GC 2.532: [DefNew: 0K->0K(576K), 0.0041910 secs]2.536: [Tenured: 524391K->524391K(525700K), 0.0333090 secs] 524391K->524391K(526276K), 0.0401890 secs]
5.684: [GC 5.684: [DefNew: 43890K->0K(49600K), 0.0041230 secs] 568281K->524391K(711296K), 0.0042690 secs]
5.822: [GC 5.822: [DefNew: 43458K->0K(49600K), 0.0036770 secs] 567849K->524391K(711296K), 0.0038330 secs]
5.956: [GC 5.957: [DefNew: 43304K->0K(49600K), 0.0039410 secs] 567695K->524391K(711296K), 0.0137480 secs]
6.112: [GC 6.113: [DefNew: 43202K->0K(49600K), 0.0034930 secs] 567594K->524391K(711296K), 0.0041640 secs]
Run with the ParallelGC, "-Xmx700m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseParallelGC", the long GCs dominate immediately:
0.000: [GC [PSYoungGen: 2272K->120K(3584K)] 526560K->524408K(529344K), 60.8538370 secs]
60.854: [Full GC [PSYoungGen: 120K->0K(3584K)] [PSOldGen: 524288K->524389K(656960K)] 524408K->524389K(660544K) [PSPermGen: 1388K->1388K(8192K)], 0.0279560 secs]
60.891: [GC [PSYoungGen: 2081K->0K(6656K)] 526470K->524389K(663616K), 57.3028060 secs]
118.215: [GC [PSYoungGen: 5163K->0K(6656K)] 529553K->524389K(663616K), 59.5562960 secs]
177.787: [GC
Thanks,
- Gordon @ IA

My pathology book is coming up in my adobe with blank pages? have a new laptop old one broken!

i have purchased a pathology book through adobe and my old hard drive broke, the new computer i have a passport from old one which i back up, when i log into my adobe, the book i purchased, had blank pages, i would like to get my book back please, my exam is in 3 months

Your GMail email is on Google's server, so you should find the same messages there that you saw before. However, any bookmarks, saved passwords, etc. that you did not sync with Weave or another solution that backs up your settings to some place on the web would need to be rescued from your old computer.
Check out this article for more insight: [https://support.mozilla.com/en-US/kb/Recovering+important+data+from+an+old+profile Recovering important data from an old profile]

Pathological Behaviour with Garbage Collection?

I'm experiencing what I believe to be very strange behaviour in the GC...
Running JDK 1.3.1 on Solaris, at a certain point the JVM starts doing a large number of Full GC's - and doesn't stop (one every couple of seconds - each taking up to 1 second of CPU). This can consume up to 60% of the CPU when the application (Resin Servlets) is doing very little.
The current invocation uses -Xmx1024k to set the upper heap size, and this strange behaviour seems to start when the heap grows to about 130k - and doesn't stop even if it shrinks.
Adding a -Xms256k (for example) to the invocation just makes matters worse. In this case the verbose:gc switch shows it doing a Full GC every couple of seconds as soon as the application is started.
Has anyone any experience of similar behaviour - or ideas about how to progress the diagnosis?

Yes - I have tried almost every combination of flags,
both supported and not supported. They make subtle
differences to the behaviour - such as changing the
heap size at which the instability sets in.
In fact increasing NewSize appears to make the problem
worse. The verbose:gc output shows that there is very
little scavanging going on, but an excessive use of
the Full GC (=mark/compact I believe).Can you post some example verbose:gc output? Also, about how often is the Full GC occuring? Have you tried to modify the the Sun RMI system properties to lengthen the time between DGC-induced full GC's (the default is 60 seconds)?
Chuck

Jump in Young GG (ParNew) times after CMS-concurrent-reset

In one of our automated test scenarios following a CMS-concurrent-reset, young GC times dramatically increased. For instance, prior to the CMS reset, average young GC times average 0.15 seconds; after the CMS reset, times would spike to as high as 40 seconds. With our high-load scenario this does not happen. The difference in data load between these two tests is approximately 360:1.
Usually when this occcurs, the young GC times would stay consistently high (between 30 and 40 seconds). However, in one of our most recent test runs, something else happened. Young GC times would spike and then over a period of time gradually decrease back down to tolerable levels (over a period of 3.5+ hours). The log snippet below illustrates this behavior.
The application itself is a custom server built on Java 1.5.0_06 using Java 1.5.0_06erdist1. The hardware is a Dell PowerEdge 6800 running Windows 2003 Server 64-bit w/SP1. The server has 32GB of RAM, no swap file, and 8 logical CPU's (4 hyperthreaded Xeon processors). The system caches large amounts of data in a variety of strong, soft, and weak reference maps is using SleepyCat Software's DBJE for data storage.
We're working on getting memory dumps, but right now, we're thinking it might have something to do with large arrays in the old generation either being reallocated, or references back to the young generation that could be causing this. There is a slight possibility that there is also a large object being allocated in the young generation, although this seems like it is less likely (the average data packet is less than 1K in this test scenario and most object allocations are usually less than 4K in size -- byte arrays usually).
We're also looking at possibly testing with the latest 1.6 beta to see if the problem goes away, or at the very least we can profile the app when it gets into trouble without having to turn off CMS.
Does anybody have any thoughts on what might be causing this, or has anyone else run into something similar before?
The JVM Args used are:
-server
-Xms28G
-Xmx28G
-enableassertions
-XX:-UsePerfData
-XX:+PrintVMOptions
-XX:-TraceClassUnloading
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-verbose:gc
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=20000
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:ParallelGCThreads=7
-XX:NewSize=128M
-XX:MaxNewSize=128M
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:CMSIncrementalDutyCycleMin=0
-XX:CMSIncrementalDutyCycle=10
-XX:CMSMarkStackSize=8M
-XX:CMSMarkStackSizeMax=32M
-XX:+UseLargePages
-XX:+DisableExplicitGC
64255.634: [GC 64255.634: [ParNew: 127304K->0K(131008K), 0.2955450 secs] 14370952K->14273019K(28671936K) icms_dc=0 , 0.2957102 secs]
64270.829: [GC 64270.829: [ParNew: 130944K->0K(131008K), 0.2805740 secs] 14403963K->14286161K(28671936K) icms_dc=0 , 0.2807523 secs]
64287.158: [GC 64287.158: [ParNew: 129676K->0K(131008K), 0.2908535 secs] 14415837K->14289089K(28671936K) icms_dc=0 , 0.2911033 secs]
64297.966: [GC 64297.966: [ParNew: 130602K->0K(131008K), 0.2682085 secs] 14419692K->14311368K(28671936K) icms_dc=0 , 0.2683754 secs]
64308.812: [GC 64308.812: [ParNew: 126613K->0K(131008K), 0.2965320 secs] 14437982K->14336914K(28671936K) icms_dc=0 , 0.2967070 secs]
64319.249: [GC 64319.249: [ParNew: 128966K->0K(131008K), 0.2878087 secs] 14465880K->14364326K(28671936K) icms_dc=3 , 0.2879923 secs]
64329.585: [GC [1 CMS-initial-mark: 14364326K(28540928K)] 14428075K(28671936K), 0.0367576 secs]
64329.622: [CMS-concurrent-mark-start]
64340.074: [GC 64340.074: [ParNew: 130758K->0K(131008K), 0.2824739 secs] 14495085K->14372339K(28671936K) icms_dc=3 , 0.2826438 secs]
64341.411: [CMS-concurrent-mark: 0.873/11.789 secs]
64341.411: [CMS-concurrent-preclean-start]
64341.411: [CMS-concurrent-preclean: 0.000/0.000 secs]
64341.557: [CMS-concurrent-abortable-preclean-start]
64341.557: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs]
64350.684: [GC 64350.684: [ParNew: 130944K->0K(131008K), 0.2899810 secs] 14503283K->14395507K(28671936K) icms_dc=3 , 0.2901651 secs]
64363.365: [GC 64363.365: [ParNew: 130944K->0K(131008K), 0.2975003 secs] 14526451K->14421343K(28671936K) icms_dc=3 , 0.2976791 secs]
64372.076: [GC[YG occupancy: 83987 K (131008 K)]64372.076: [Rescan (parallel) , 0.0674530 secs]64372.144: [weak refs processing, 0.0308673 secs]64372.175: [class unloading[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27]
64372.182: [scrub symbol & string tables, 0.0083922 secs] [1 CMS-remark: 14421343K(28540928K)] 14505331K(28671936K), 0.1281897 secs]
64372.206: [CMS-concurrent-sweep-start]
64376.703: [GC 64376.703: [ParNew: 130944K->0K(131008K), 0.3150279 secs] 14552287K->14443248K(28671936K) icms_dc=3 , 0.3152040 secs]
64393.182: [GC 64393.183: [ParNew: 130944K->0K(131008K), 0.2967053 secs] 14570664K->14441684K(28671936K) icms_dc=3 , 0.2968985 secs]
64395.855: [CMS-concurrent-sweep: 2.118/23.650 secs]
64395.856: [CMS-concurrent-reset-start]
64396.202: [CMS-concurrent-reset: 0.346/0.346 secs]
64404.846: [GC 64404.846: [ParNew: 130944K->0K(131008K), 11.8319372 secs] 452302K->326245K(28671936K) icms_dc=0 , 11.8321090 secs]
64420.865: [GC 64420.865: [ParNew: 130944K->0K(131008K), 11.8750319 secs] 457189K->352203K(28671936K) icms_dc=0 , 11.8752811 secs]
64435.536: [GC 64435.536: [ParNew: 130944K->0K(131008K), 11.8229769 secs] 483147K->374039K(28671936K) icms_dc=0 , 11.8231579 secs]
64448.564: [GC 64448.564: [ParNew: 130944K->0K(131008K), 12.2692927 secs] 504983K->376605K(28671936K) icms_dc=0 , 12.2694811 secs]
64462.276: [GC 64462.276: [ParNew: 130944K->0K(131008K), 12.0860714 secs] 507549K->401835K(28671936K) icms_dc=0 , 12.0862452 secs]
64478.522: [GC 64478.522: [ParNew: 126250K->0K(131008K), 12.3507999 secs] 528085K->428168K(28671936K) icms_dc=0 , 12.3509812 secs]
64492.047: [GC 64492.047: [ParNew: 130944K->0K(131008K), 11.7977262 secs] 559112K->450172K(28671936K) icms_dc=0 , 11.7979055 secs]
64505.333: [GC 64505.333: [ParNew: 130944K->0K(131008K), 11.6679971 secs] 581116K->475511K(28671936K) icms_dc=0 , 11.6681951 secs]
64523.377: [GC 64523.377: [ParNew: 130944K->0K(131008K), 12.1859479 secs] 606455K->496955K(28671936K) icms_dc=0 , 12.1861274 secs]
64537.004: [GC 64537.004: [ParNew: 130944K->0K(131008K), 12.3909395 secs] 627899K->500863K(28671936K) icms_dc=0 , 12.3911184 secs]
64550.836: [GC 64550.836: [ParNew: 130944K->0K(131008K), 13.3435750 secs] 631807K->522967K(28671936K) icms_dc=0 , 13.3437711 secs]
64565.421: [GC 64565.421: [ParNew: 130943K->0K(131008K), 12.1759533 secs] 653910K->551883K(28671936K) icms_dc=0 , 12.1761337 secs]
64578.968: [GC 64578.968: [ParNew: 129670K->0K(131008K), 11.7518116 secs] 681553K->565967K(28671936K) icms_dc=0 , 11.7519997 secs]
64595.156: [GC 64595.156: [ParNew: 130944K->0K(131008K), 12.7430090 secs] 696911K->588549K(28671936K) icms_dc=0 , 12.7431996 secs]
64609.154: [GC 64609.155: [ParNew: 127642K->0K(131008K), 13.4122057 secs] 716191K->613789K(28671936K) icms_dc=0 , 13.4123919 secs]
64623.588: [GC 64623.588: [ParNew: 130934K->0K(131008K), 11.8692832 secs] 744723K->634958K(28671936K) icms_dc=0 , 11.8694631 secs]
64636.690: [GC 64636.690: [ParNew: 130944K->0K(131008K), 12.2170544 secs] 765902K->640533K(28671936K) icms_dc=0 , 12.2172308 secs]
64652.592: [GC 64652.592: [ParNew: 130153K->0K(131008K), 12.5039780 secs] 770687K->670050K(28671936K) icms_dc=0 , 12.5041652 secs]
64665.798: [GC 64665.798: [ParNew: 130944K->0K(131008K), 11.8129872 secs] 800994K->683828K(28671936K) icms_dc=0 , 11.8131695 secs]
64678.710: [GC 64678.710: [ParNew: 130944K->0K(131008K), 12.2518054 secs] 814772K->686723K(28671936K) icms_dc=0 , 12.2519837 secs]
64692.491: [GC 64692.491: [ParNew: 128795K->0K(131008K), 12.4420712 secs] 815519K->711677K(28671936K) icms_dc=0 , 12.4422521 secs]
64706.761: [GC 64706.761: [ParNew: 130944K->0K(131008K), 12.1215915 secs] 842621K->733167K(28671936K) icms_dc=0 , 12.1217708 secs]
64720.486: [GC 64720.486: [ParNew: 130944K->0K(131008K), 12.4783432 secs] 864111K->736715K(28671936K) icms_dc=0 , 12.4785325 secs]
64734.374: [GC 64734.374: [ParNew: 129901K->0K(131008K), 12.6808484 secs] 866617K->763351K(28671936K) icms_dc=0 , 12.6810264 secs]
64748.544: [GC 64748.544: [ParNew: 130944K->0K(131008K), 13.0768090 secs] 894295K->772967K(28671936K) icms_dc=0 , 13.0769936 secs]
64762.836: [GC 64762.836: [ParNew: 130944K->0K(131008K), 12.6763159 secs] 903911K->797964K(28671936K) icms_dc=0 , 12.6764964 secs]
64777.255: [GC 64777.255: [ParNew: 125172K->0K(131008K), 11.3742001 secs] 923137K->823513K(28671936K) icms_dc=0 , 11.3743819 secs]
64789.913: [GC 64789.913: [ParNew: 130944K->0K(131008K), 12.6647870 secs] 954457K->845201K(28671936K) icms_dc=0 , 12.6649700 secs]
64803.944: [GC 64803.944: [ParNew: 127557K->0K(131008K), 12.4356678 secs] 972758K->870316K(28671936K) icms_dc=0 , 12.4358488 secs]
64818.674: [GC 64818.674: [ParNew: 130944K->0K(131008K), 12.1313661 secs] 1001260K->895535K(28671936K) icms_dc=0 , 12.1315657 secs]
64834.764: [GC 64834.764: [ParNew: 129555K->0K(131008K), 11.9518352 secs] 1025090K->925104K(28671936K) icms_dc=0 , 11.9520134 secs]
64848.138: [GC 64848.138: [ParNew: 130944K->0K(131008K), 12.9124929 secs] 1056048K->939587K(28671936K) icms_dc=0 , 12.9127034 secs]
64862.318: [GC 64862.318: [ParNew: 123806K->0K(131008K), 13.0569914 secs] 1063394K->964362K(28671936K) icms_dc=0 , 13.0571688 secs]
64877.530: [GC 64877.530: [ParNew: 130944K->0K(131008K), 12.3056353 secs] 1095306K->986163K(28671936K) icms_dc=0 , 12.3058150 secs]
64891.725: [GC 64891.725: [ParNew: 130944K->0K(131008K), 13.0376824 secs] 1117107K->1007903K(28671936K) icms_dc=0 , 13.0378846 secs]
76591.355: [GC 76591.355: [ParNew: 130944K->0K(131008K), 0.2791594 secs] 15265232K->15138597K(28671936K) icms_dc=0 , 0.2793572 secs]
76600.766: [GC 76600.766: [ParNew: 130944K->0K(131008K), 0.2752963 secs] 15269541K->15160685K(28671936K) icms_dc=0 , 0.2754887 secs]
76611.557: [GC 76611.557: [ParNew: 130944K->0K(131008K), 0.2790154 secs] 15291629K->15165049K(28671936K) icms_dc=0 , 0.2792048 secs]
76622.303: [GC 76622.303: [ParNew: 130944K->0K(131008K), 0.2849006 secs] 15295993K->15187817K(28671936K) icms_dc=0 , 0.2850920 secs]
76632.887: [GC 76632.887: [ParNew: 130944K->0K(131008K), 0.2689896 secs] 15318761K->15213844K(28671936K) icms_dc=0 , 0.2693092 secs]
76643.956: [GC 76643.956: [ParNew: 130879K->0K(131008K), 0.2662928 secs] 15344723K->15236015K(28671936K) icms_dc=0 , 0.2664782 secs]
76656.379: [GC 76656.379: [ParNew: 130944K->0K(131008K), 0.2872218 secs] 15366959K->15242404K(28671936K) icms_dc=0 , 0.2874433 secs]
76664.841: [GC 76664.841: [ParNew: 130868K->0K(131008K), 0.2931068 secs] 15373273K->15269283K(28671936K) icms_dc=0 , 0.2932965 secs]
76677.732: [GC 76677.732: [ParNew: 130920K->0K(131008K), 0.2475306 secs] 15400203K->15276884K(28671936K) icms_dc=0 , 0.2477292 secs]
76697.644: [GC 76697.644: [ParNew: 130944K->0K(131008K), 0.2879494 secs] 15407828K->15280611K(28671936K) icms_dc=0 , 0.2881359 secs]
76707.426: [GC 76707.426: [ParNew: 130944K->0K(131008K), 0.2911420 secs] 15411555K->15285072K(28671936K) icms_dc=0 , 0.2913233 secs]
76717.857: [GC 76717.857: [ParNew: 130944K->0K(131008K), 0.2701989 secs] 15416016K->15309374K(28671936K) icms_dc=0 , 0.2703928 secs]
76728.560: [GC 76728.561: [ParNew: 130944K->0K(131008K), 0.2931724 secs] 15440318K->15333673K(28671936K) icms_dc=0 , 0.2933609 secs]
76739.261: [GC 76739.262: [ParNew: 126315K->0K(131008K), 0.2973160 secs] 15459989K->15363224K(28671936K) icms_dc=0 , 0.2974960 secs]
76749.705: [GC 76749.705: [ParNew: 127396K->0K(131008K), 0.2972422 secs] 15490620K->15382987K(28671936K) icms_dc=0 , 0.2975240 secs]
76767.794: [GC 76767.794: [ParNew: 130944K->0K(131008K), 0.2838999 secs] 15513931K->15391591K(28671936K) icms_dc=0 , 0.2840901 secs]
76773.076: [GC 76773.076: [ParNew: 130944K->0K(131008K), 0.2552678 secs] 15522535K->15393025K(28671936K) icms_dc=0 , 0.2554467 secs]
76790.058: [GC 76790.058: [ParNew: 130860K->0K(131008K), 0.2934145 secs] 15523885K->15394843K(28671936K) icms_dc=0 , 0.2936080 secs]
76802.396: [GC 76802.396: [ParNew: 130604K->0K(131008K), 0.2972303 secs] 15525447K->15397713K(28671936K) icms_dc=0 , 0.2974271 secs]
76813.202: [GC 76813.202: [ParNew: 130169K->0K(131008K), 0.2880799 secs] 15527883K->15422970K(28671936K) icms_dc=0 , 0.2882746 secs]
76823.896: [GC 76823.896: [ParNew: 129924K->0K(131008K), 0.2832927 secs] 15552894K->15452588K(28671936K) icms_dc=0 , 0.2834869 secs]
76834.357: [GC 76834.358: [ParNew: 130767K->0K(131008K), 0.2764792 secs] 15583355K->15474140K(28671936K) icms_dc=0 , 0.2766770 secs]
76855.497: [GC 76855.497: [ParNew: 130897K->0K(131008K), 0.2894657 secs] 15605037K->15488217K(28671936K) icms_dc=0 , 0.2896626 secs]
76866.253: [GC 76866.253: [ParNew: 130944K->0K(131008K), 0.2999675 secs] 15619161K->15490659K(28671936K) icms_dc=0 , 0.3001493 secs]
76879.530: [GC 76879.530: [ParNew: 130944K->0K(131008K), 0.2816982 secs] 15621603K->15493104K(28671936K) icms_dc=0 , 0.2818897 secs]
76888.031: [GC 76888.031: [ParNew: 130944K->0K(131008K), 0.2760192 secs] 15624048K->15498672K(28671936K) icms_dc=0 , 0.2762118 secs]
76898.232: [GC 76898.232: [ParNew: 125483K->0K(131008K), 0.3109439 secs] 15624155K->15528482K(28671936K) icms_dc=0 , 0.3111342 secs]
76914.416: [GC 76914.416: [ParNew: 130944K->0K(131008K), 0.3065572 secs] 15659426K->15542745K(28671936K) icms_dc=0 , 0.3067479 secs]
76924.748: [GC 76924.748: [ParNew: 130883K->0K(131008K), 0.2937243 secs] 15673628K->15544088K(28671936K) icms_dc=0 , 0.2938988 secs]
76942.226: [GC 76942.226: [ParNew: 130944K->0K(131008K), 0.2960886 secs] 15675032K->15547980K(28671936K) icms_dc=0 , 0.2962779 secs]
76951.345: [GC 76951.345: [ParNew: 130944K->0K(131008K), 0.3050291 secs] 15678924K->15573683K(28671936K) icms_dc=0 , 0.3052196 secs]
76961.936: [GC 76961.936: [ParNew: 130944K->0K(131008K), 0.2914707 secs] 15704627K->15595952K(28671936K) icms_dc=0 , 0.2916589 secs]
76972.575: [GC 76972.575: [ParNew: 129330K->0K(131008K), 0.2838618 secs] 15725283K->15621207K(28671936K) icms_dc=0 , 0.2840550 secs]
76986.283: [GC 76986.284: [ParNew: 130939K->0K(131008K), 0.2933898 secs] 15752146K->15643239K(28671936K) icms_dc=0 , 0.2935624 secs]
76997.181: [GC 76997.181: [ParNew: 130944K->0K(131008K), 0.3075952 secs] 15774183K->15645463K(28671936K) icms_dc=0 , 0.3077828 secs]
77007.913: [GC 77007.913: [ParNew: 130944K->0K(131008K), 0.2958084 secs] 15776407K->15647422K(28671936K) icms_dc=0 , 0.2960437 secs]
77018.011: [GC 77018.011: [ParNew: 130910K->0K(131008K), 0.2964236 secs] 15778333K->15649978K(28671936K) icms_dc=0 , 0.2966224 secs]
77034.694: [GC 77034.694: [ParNew: 130944K->0K(131008K), 0.3092856 secs] 15780922K->15652024K(28671936K) icms_dc=0 , 0.3094763 secs]
77047.053: [GC 77047.054: [ParNew: 130944K->0K(131008K), 0.3015745 secs] 15782968K->15657359K(28671936K) icms_dc=0 , 0.3017561 secs]
77057.249: [GC 77057.249: [ParNew: 130944K->0K(131008K), 0.3185589 secs] 15788303K->15680967K(28671936K) icms_dc=0 , 0.3188166 secs]
77070.581: [GC 77070.581: [ParNew: 130944K->0K(131008K), 0.3062147 secs] 15811911K->15706444K(28671936K) icms_dc=0 , 0.3064015 secs]
77078.801: [GC 77078.801: [ParNew: 123632K->0K(131008K), 0.3221894 secs] 15830077K->15732164K(28671936K) icms_dc=0 , 0.3223794 secs]
77092.282: [GC 77092.282: [ParNew: 130944K->0K(131008K), 0.3141997 secs] 15863108K->15754279K(28671936K) icms_dc=0 , 0.3143788 secs]
77104.965: [GC 77104.965: [ParNew: 130936K->0K(131008K), 0.2957323 secs] 15885215K->15756444K(28671936K) icms_dc=0 , 0.2959160 secs]
77120.762: [GC 77120.762: [ParNew: 130944K->0K(131008K), 0.2967338 secs] 15887388K->15759133K(28671936K) icms_dc=0 , 0.2969243 secs]
77125.566: [GC 77125.566: [ParNew: 130944K->0K(131008K), 0.3192459 secs] 15890077K->15761500K(28671936K) icms_dc=0 , 0.3194239 secs]
77134.280: [GC 77134.280: [ParNew: 130930K->0K(131008K), 0.3128382 secs] 15892430K->15763820K(28671936K) icms_dc=0 , 0.3130111 secs]
77148.565: [GC 77148.565: [ParNew: 130944K->0K(131008K), 0.3146629 secs] 15894764K->15766067K(28671936K) icms_dc=0 , 0.3148481 secs]
77166.762: [GC 77166.762: [ParNew: 130944K->0K(131008K), 0.2902859 secs] 15897011K->15769686K(28671936K) icms_dc=0 , 0.2904642 secs]
77173.957: [GC 77173.957: [ParNew: 130944K->0K(131008K), 0.3132502 secs] 15900630K->15794955K(28671936K) icms_dc=0 , 0.3134416 secs]
77184.689: [GC 77184.689: [ParNew: 130944K->0K(131008K), 0.3122028 secs] 15925899K->15817089K(28671936K) icms_dc=0 , 0.3123857 secs]
77195.961: [GC 77195.961: [ParNew: 130944K->0K(131008K), 0.3101975 secs] 15948033K->15823153K(28671936K) icms_dc=0 , 0.3103801 secs]
77212.697: [GC 77212.697: [ParNew: 130944K->0K(131008K), 0.3109774 secs] 15954097K->15845044K(28671936K) icms_dc=0 , 0.3111528 secs]
77217.947: [GC 77217.947: [ParNew: 130933K->0K(131008K), 0.2913624 secs] 15975978K->15846428K(28671936K) icms_dc=0 , 0.2915358 secs]
77237.788: [GC 77237.788: [ParNew: 129457K->0K(131008K), 0.2967028 secs] 15975885K->15849363K(28671936K) icms_dc=0 , 0.2968859 secs]
77248.232: [GC 77248.232: [ParNew: 130441K->0K(131008K), 0.3249377 secs] 15979804K->15872164K(28671936K) icms_dc=0 , 0.3251147 secs]
77259.231: [GC 77259.231: [ParNew: 130939K->0K(131008K), 0.3262398 secs] 16003104K->15893957K(28671936K) icms_dc=0 , 0.3264322 secs]
77269.578: [GC 77269.578: [ParNew: 130944K->0K(131008K), 0.3244924 secs] 16024901K->15897801K(28671936K) icms_dc=0 , 0.3246682 secs]
77280.109: [GC 77280.109: [ParNew: 129785K->0K(131008K), 0.3440615 secs] 16027587K->15923895K(28671936K) icms_dc=0 , 0.3442430 secs]
77290.806: [GC 77290.806: [ParNew: 125725K->0K(131008K), 0.3185027 secs] 16049620K->15950674K(28671936K) icms_dc=0 , 0.3186805 secs]
77302.797: [GC 77302.797: [ParNew: 130859K->0K(131008K), 0.3043288 secs] 16081533K->15958590K(28671936K) icms_dc=0 , 0.3045180 secs]
77312.365: [GC 77312.365: [ParNew: 130944K->0K(131008K), 0.2918505 secs] 16089534K->15960772K(28671936K) icms_dc=0 , 0.2920314 secs]
77322.862: [GC 77322.863: [ParNew: 130944K->0K(131008K), 0.3015737 secs] 16091716K->15963449K(28671936K) icms_dc=0 , 0.3017476 secs]
77343.729: [GC 77343.729: [ParNew: 130944K->0K(131008K), 0.3236395 secs] 16094393K->15967208K(28671936K) icms_dc=0 , 0.3238134 secs]
77354.586: [GC 77354.586: [ParNew: 130944K->0K(131008K), 0.3131893 secs] 16098152K->15970188K(28671936K) icms_dc=0 , 0.3133721 secs]
77364.952: [GC 77364.952: [ParNew: 130944K->0K(131008K), 0.3102413 secs] 16101132K->15992949K(28671936K) icms_dc=0 , 0.3104283 secs]
77377.458: [GC 77377.458: [ParNew: 130935K->0K(131008K), 0.3312504 secs] 16123885K->16015429K(28671936K) icms_dc=0 , 0.3314204 secs]
77386.642: [GC 77386.642: [ParNew: 130944K->0K(131008K), 0.3294257 secs] 16146373K->16018109K(28671936K) icms_dc=0 , 0.3296290 secs]
77397.408: [GC 77397.408: [ParNew: 130944K->0K(131008K), 0.3466004 secs] 16149053K->16041639K(28671936K) icms_dc=0 , 0.3467640 secs]
77407.720: [GC 77407.720: [ParNew: 127181K->0K(131008K), 0.3291395 secs] 16168820K->16067781K(28671936K) icms_dc=0 , 0.3293197 secs]
77419.256: [GC 77419.257: [ParNew: 127292K->0K(131008K), 0.3417004 secs] 16195074K->16101594K(28671936K) icms_dc=3 , 0.3418733 secs]
77426.886: [GC [1 CMS-initial-mark: 16101594K(28540928K)] 16165188K(28671936K), 0.0349381 secs]
77426.921: [CMS-concurrent-mark-start]
77428.941: [GC 77428.941: [ParNew: 130944K->0K(131008K), 0.2950079 secs] 16232538K->16108380K(28671936K) icms_dc=3 , 0.2951910 secs]
77438.826: [CMS-concurrent-mark: 0.913/11.904 secs]
77438.826: [CMS-concurrent-preclean-start]
77438.826: [CMS-concurrent-preclean: 0.000/0.000 secs]
77439.028: [CMS-concurrent-abortable-preclean-start]
77439.028: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs]
77439.480: [GC 77439.480: [ParNew: 126872K->0K(131008K), 0.3251631 secs] 16235253K->16139128K(28671936K) icms_dc=3 , 0.3253437 secs]
77447.411: [GC[YG occupancy: 66352 K (131008 K)]77447.411: [Rescan (parallel) , 0.0551127 secs]77447.466: [weak refs processing, 0.0308647 secs]77447.497: [class unloading[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor28]
77447.504: [scrub symbol & string tables, 0.0084419 secs] [1 CMS-remark: 16139128K(28540928K)] 16205480K(28671936K), 0.1156339 secs]
77447.528: [CMS-concurrent-sweep-start]
77450.331: [GC 77450.331: [ParNew: 127555K->0K(131008K), 0.3276881 secs] 16257886K->16148782K(28671936K) icms_dc=3 , 0.3278723 secs]
77471.167: [GC 77471.168: [ParNew: 130944K->0K(131008K), 0.3443864 secs] 16264980K->16156348K(28671936K) icms_dc=3 , 0.3445624 secs]
77484.330: [GC 77484.331: [ParNew: 130944K->0K(131008K), 0.3138211 secs] 16282337K->16155285K(28671936K) icms_dc=3 , 0.3139971 secs]
77490.323: [GC 77490.323: [ParNew: 130887K->0K(131008K), 0.3463333 secs] 16266084K->16158092K(28671936K) icms_dc=3 , 0.3465076 secs]
77503.075: [GC 77503.075: [ParNew: 130944K->0K(131008K), 0.3230820 secs] 16246239K->16117912K(28671936K) icms_dc=3 , 0.3232641 secs]
77513.508: [GC 77513.508: [ParNew: 130944K->0K(131008K), 0.3446744 secs] 16232772K->16104890K(28671936K) icms_dc=3 , 0.3449231 secs]
77526.076: [GC 77526.076: [ParNew: 130173K->0K(131008K), 0.3488064 secs] 16219569K->16093708K(28671936K) icms_dc=3 , 0.3490094 secs]
77535.048: [GC 77535.048: [ParNew: 130944K->0K(131008K), 0.3444742 secs] 16091984K->15984538K(28671936K) icms_dc=3 , 0.3447258 secs]
77540.284: [CMS-concurrent-sweep: 1.808/92.757 secs]
77540.285: [CMS-concurrent-reset-start]
77540.637: [CMS-concurrent-reset: 0.352/0.352 secs]
77547.483: [GC 77547.483: [ParNew: 130944K->0K(131008K), 21.2268491 secs] 569606K->465738K(28671936K) icms_dc=0 , 21.2270424 secs]
77570.056: [GC 77570.056: [ParNew: 129623K->0K(131008K), 21.8197998 secs] 595362K->492094K(28671936K) icms_dc=0 , 21.8200025 secs]
77592.450: [GC 77592.450: [ParNew: 130901K->0K(131008K), 22.4998156 secs] 622995K->513309K(28671936K) icms_dc=0 , 22.5000306 secs]
77615.471: [GC 77615.471: [ParNew: 130865K->0K(131008K), 24.1625520 secs] 644174K->515142K(28671936K) icms_dc=0 , 24.1627472 secs]

Might it be possible for you to check if the same behaviour reproduces
with the latest (b76?) Mustang JVM?
If so, let us know and we'll look into this more closely.
PS: Could you also pstack the process once it gets into
one of these long scavenges, and see if you find a lot
of "block_start" calls at the top of the stack (during card
scanning). It's almost certainly some performace pathology
related to block offset table accesses during card scanning
in the presence of large contiguous blocks would be my
guess.

JVM Crash: ConcMarkSweepGC related?

We're experiencing the following JVM crash approximately 2-5 times per day on a well traffic application server running Tomcat 5.5 under Java 1.5 (05) on Linux with a 2.4.21-37 kernel. I'm posting here because we kind of have no route to take: I can't seem to work around the crash in any way. There's a potential that UseParallelGC might fix this, but I'm unable to run the app reliably with a ParallelGC, as we have a very large cache and the Full GC's get unbelievably long (2GB heap).
How can we find out what's going on? Is there a way to obtain a debug build so that:
V [libjvm.so+0x4c2c53]
V [libjvm.so+0x1bea7c]
We're also seeing this in the standard output:
An exception occurred on the SessionReaperDaemon[null, 2005-11-01 20:02:14.81] thread while processing (task="com.tangosol.coherence.servlet.SessionHelper$SessionReaperDaemon$IncrementalReapTask@1ba4ac8"):
java.lang.NullPointerException
     at com.tangosol.coherence.servlet.SessionHelper$SessionReaperDaemon$IncrementalReapTask.run(SessionHelper.java:3072)
     at com.tangosol.util.TaskDaemon.run(TaskDaemon.java:69)
     at com.tangosol.coherence.servlet.SessionHelper$SessionReaperDaemon.run(SessionHelper.java:2709)
     at com.tangosol.util.Daemon$1.run(Daemon.java:63)
Prior to the crash, but this is an impossible place for an exception:
3070: if (iter.hasNext())
3071: {
3072: String sId = (String) iter.next();3073: ...
Any help or next steps would be greatly appreciated.
An attached dump:
# An unexpected error has been detected by HotSpot Virtual Machine:
# Internal Error (4E4D4554484F440E4350500521), pid=1705, tid=907377584
# Java VM: Java HotSpot(TM) Server VM (1.5.0_05-b05 mixed mode)
--------------- T H R E A D ---------------
Current thread (0x36bc7068): JavaThread "SessionReaperDaemon[null, 2005-11-01 22:38:23.388]" daemon [_thread_in_Java, id=1739]
Stack: [0x360d7000,0x36158000), sp=0x36156eec, free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x4c2c53]
V [libjvm.so+0x1bea7c]
V [libjvm.so+0x3e2bee]
V [libjvm.so+0x4564cb]
V [libjvm.so+0x3f8956]
V [libjvm.so+0x3f5c64]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0894eba0 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=4871]
0x0864ea88 JavaThread "RMI TCP Connection(341)-192.168.100.62" daemon [_thread_in_native, id=4802]
0x083e6568 JavaThread "http-80-Processor300" daemon [_thread_in_native, id=4708]
0x08d4e058 JavaThread "http-80-Processor299" daemon [_thread_in_native, id=4707]
0x08bc7c20 JavaThread "http-80-Processor298" daemon [_thread_blocked, id=4706]
0x0845fd78 JavaThread "http-80-Processor297" daemon [_thread_in_native, id=4705]
0x08e39330 JavaThread "http-80-Processor296" daemon [_thread_in_native, id=4704]
0x08bceb30 JavaThread "http-80-Processor295" daemon [_thread_blocked, id=4703]
0x08bb9de8 JavaThread "http-80-Processor294" daemon [_thread_in_native, id=4702]
0x08b692d0 JavaThread "http-80-Processor293" daemon [_thread_in_native, id=4701]
0x084fa8e0 JavaThread "http-80-Processor292" daemon [_thread_in_native, id=4700]
0x08446908 JavaThread "http-80-Processor291" daemon [_thread_blocked, id=4699]
0x08e4c4c0 JavaThread "http-80-Processor290" daemon [_thread_in_native, id=4698]
0x08625e60 JavaThread "http-80-Processor289" daemon [_thread_blocked, id=4697]
0x0843d5b8 JavaThread "http-80-Processor288" daemon [_thread_in_native, id=4696]
0x08ca13b8 JavaThread "http-80-Processor287" daemon [_thread_in_native, id=4695]
0x08395e00 JavaThread "http-80-Processor286" daemon [_thread_in_native, id=4694]
0x084c8f28 JavaThread "http-80-Processor285" daemon [_thread_in_native, id=4693]
0x083f6490 JavaThread "http-80-Processor284" daemon [_thread_in_native, id=4692]
0x08e69368 JavaThread "http-80-Processor281" daemon [_thread_blocked, id=4689]
0x08df8218 JavaThread "http-80-Processor280" daemon [_thread_blocked, id=4688]
0x0861ba28 JavaThread "http-80-Processor279" daemon [_thread_blocked, id=4687]
0x083e22d0 JavaThread "http-80-Processor278" daemon [_thread_blocked, id=4686]
0x08e44e80 JavaThread "http-80-Processor277" daemon [_thread_blocked, id=4685]
0x08625478 JavaThread "http-80-Processor276" daemon [_thread_blocked, id=4684]
0x08547890 JavaThread "http-80-Processor275" daemon [_thread_in_native, id=4632]
0x0846c598 JavaThread "http-80-Processor274" daemon [_thread_in_native, id=4631]
0x08beadb8 JavaThread "http-80-Processor270" daemon [_thread_in_native, id=4627]
0x0865b3a0 JavaThread "http-80-Processor269" daemon [_thread_in_native, id=4626]
0x092899e0 JavaThread "http-80-Processor268" daemon [_thread_in_native, id=4625]
0x09277f58 JavaThread "http-80-Processor267" daemon [_thread_blocked, id=4624]
0x086790b0 JavaThread "http-80-Processor265" daemon [_thread_in_native, id=4622]
0x0844d920 JavaThread "http-80-Processor263" daemon [_thread_blocked, id=4620]
0x0846f478 JavaThread "http-80-Processor262" daemon [_thread_in_native, id=4619]
0x08c5f470 JavaThread "http-80-Processor261" daemon [_thread_in_native, id=4618]
0x088cd8d0 JavaThread "http-80-Processor260" daemon [_thread_in_native, id=4617]
0x08d5af30 JavaThread "http-80-Processor259" daemon [_thread_blocked, id=4616]
0x0884bc18 JavaThread "http-80-Processor258" daemon [_thread_blocked, id=4615]
0x08e618a0 JavaThread "http-80-Processor257" daemon [_thread_in_native, id=4614]
0x08be3de0 JavaThread "http-80-Processor256" daemon [_thread_in_native, id=4613]
0x085de680 JavaThread "http-80-Processor255" daemon [_thread_blocked, id=4612]
0x083b8420 JavaThread "http-80-Processor254" daemon [_thread_blocked, id=4611]
0x08e533e0 JavaThread "http-80-Processor253" daemon [_thread_in_native, id=4610]
0x083c5af8 JavaThread "http-80-Processor252" daemon [_thread_in_native, id=4609]
0x084701c0 JavaThread "http-80-Processor251" daemon [_thread_blocked, id=4608]
0x3639cff0 JavaThread "http-80-Processor250" daemon [_thread_in_native, id=4402]
0x2f41cd88 JavaThread "http-80-Processor249" daemon [_thread_in_native, id=4401]
0x36bf7a48 JavaThread "http-80-Processor247" daemon [_thread_blocked, id=4399]
0x35160748 JavaThread "http-80-Processor246" daemon [_thread_in_native, id=4398]
0x36b42ed0 JavaThread "http-80-Processor245" daemon [_thread_blocked, id=4397]
0x359b3ec0 JavaThread "http-80-Processor244" daemon [_thread_in_native, id=4396]
0x3633a1f8 JavaThread "http-80-Processor240" daemon [_thread_blocked, id=4392]
0x35a9ef28 JavaThread "http-80-Processor239" daemon [_thread_in_native, id=4391]
0x35a52840 JavaThread "http-80-Processor238" daemon [_thread_blocked, id=4390]
0x35a8e9e8 JavaThread "http-80-Processor236" daemon [_thread_blocked, id=4388]
0x35a8de38 JavaThread "http-80-Processor235" daemon [_thread_in_native, id=4387]
0x304703b8 JavaThread "http-80-Processor232" daemon [_thread_in_native, id=4384]
0x36b32498 JavaThread "http-80-Processor231" daemon [_thread_blocked, id=4383]
0x2e149a40 JavaThread "http-80-Processor230" daemon [_thread_blocked, id=4382]
0x304731f8 JavaThread "http-80-Processor229" daemon [_thread_blocked, id=4381]
0x2e186cd0 JavaThread "http-80-Processor228" daemon [_thread_in_native, id=4380]
0x35a53c68 JavaThread "http-80-Processor227" daemon [_thread_blocked, id=4379]
0x083a85d0 JavaThread "RMI TCP Connection(315)-192.168.100.62" daemon [_thread_blocked, id=4343]
0x08cdd748 JavaThread "JMX server connection timeout 1017" daemon [_thread_blocked, id=3885]
0x08df9fc0 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=3884]
0x083c7f68 JavaThread "http-80-Processor225" daemon [_thread_blocked, id=3816]
0x083c85b8 JavaThread "http-80-Processor223" daemon [_thread_blocked, id=3814]
0x08df3be8 JavaThread "http-80-Processor222" daemon [_thread_in_native, id=3813]
0x087a49b8 JavaThread "http-80-Processor221" daemon [_thread_in_native, id=3812]
0x084c9db8 JavaThread "http-80-Processor220" daemon [_thread_in_native, id=3811]
0x086203e0 JavaThread "http-80-Processor218" daemon [_thread_in_native, id=3809]
0x088e2090 JavaThread "http-80-Processor216" daemon [_thread_in_native, id=3807]
0x088bd2d0 JavaThread "http-80-Processor214" daemon [_thread_in_native, id=3805]
0x08863490 JavaThread "http-80-Processor211" daemon [_thread_in_native, id=3802]
0x08e9f4b8 JavaThread "http-80-Processor207" daemon [_thread_in_native, id=3798]
0x08c9f748 JavaThread "http-80-Processor206" daemon [_thread_in_native, id=3797]
0x0861fa00 JavaThread "http-80-Processor205" daemon [_thread_in_native, id=3796]
0x08408ec8 JavaThread "http-80-Processor204" daemon [_thread_blocked, id=3795]
0x0869bf28 JavaThread "http-80-Processor203" daemon [_thread_blocked, id=3794]
0x08ba2648 JavaThread "http-80-Processor201" daemon [_thread_blocked, id=3792]
0x08e70980 JavaThread "http-80-Processor200" daemon [_thread_blocked, id=3550]
0x08e707e8 JavaThread "http-80-Processor199" daemon [_thread_blocked, id=3549]
0x088e83b0 JavaThread "http-80-Processor196" daemon [_thread_blocked, id=3546]
0x08634758 JavaThread "http-80-Processor195" daemon [_thread_blocked, id=3545]
0x08198ea8 JavaThread "http-80-Processor194" daemon [_thread_blocked, id=3544]
0x0831d5a8 JavaThread "http-80-Processor191" daemon [_thread_in_native, id=3541]
0x088c6788 JavaThread "http-80-Processor189" daemon [_thread_in_native, id=3539]
0x0835d868 JavaThread "http-80-Processor187" daemon [_thread_blocked, id=3537]
0x08be2788 JavaThread "http-80-Processor186" daemon [_thread_in_native, id=3536]
0x08d5b638 JavaThread "http-80-Processor182" daemon [_thread_blocked, id=3532]
0x088c6488 JavaThread "http-80-Processor180" daemon [_thread_in_native, id=3530]
0x08d83d70 JavaThread "http-80-Processor173" daemon [_thread_blocked, id=3158]
0x088f3ca8 JavaThread "http-80-Processor171" daemon [_thread_blocked, id=3156]
0x08862858 JavaThread "http-80-Processor168" daemon [_thread_blocked, id=3153]
0x08629e38 JavaThread "http-80-Processor162" daemon [_thread_blocked, id=3147]
0x086b9dd0 JavaThread "http-80-Processor159" daemon [_thread_in_native, id=3144]
0x086e1c88 JavaThread "http-80-Processor153" daemon [_thread_in_native, id=3138]
0x082fae10 JavaThread "http-80-Processor151" daemon [_thread_in_native, id=3136]
0x084c7c40 JavaThread "http-80-Processor149" daemon [_thread_blocked, id=2489]
0x087a5568 JavaThread "http-80-Processor147" daemon [_thread_in_native, id=2487]
0x08c5f648 JavaThread "http-80-Processor146" daemon [_thread_blocked, id=2486]
0x08c64090 JavaThread "http-80-Processor135" daemon [_thread_blocked, id=2475]
0x08d60c68 JavaThread "http-80-Processor133" daemon [_thread_in_native, id=2473]
0x08cdc058 JavaThread "http-80-Processor129" daemon [_thread_blocked, id=2469]
0x0854a978 JavaThread "http-80-Processor128" daemon [_thread_blocked, id=2468]
0x08d940c0 JavaThread "http-80-Processor126" daemon [_thread_in_native, id=2466]
0x087c4998 JavaThread "http-80-Processor114" daemon [_thread_blocked, id=2186]
0x0895ec90 JavaThread "http-80-Processor113" daemon [_thread_in_native, id=2185]
0x0895e028 JavaThread "http-80-Processor112" daemon [_thread_blocked, id=2184]
0x08e9bbb8 JavaThread "http-80-Processor82" daemon [_thread_in_native, id=2147]
0x08634cf8 JavaThread "http-80-Processor80" daemon [_thread_in_native, id=2145]
0x085d93c0 JavaThread "http-80-Processor68" daemon [_thread_blocked, id=2129]
0x085d8658 JavaThread "http-80-Processor67" daemon [_thread_blocked, id=2128]
0x085d7310 JavaThread "http-80-Processor66" daemon [_thread_in_native, id=2127]
0x084c57e0 JavaThread "http-80-Processor60" daemon [_thread_blocked, id=2121]
0x084c4a78 JavaThread "http-80-Processor59" daemon [_thread_blocked, id=2120]
0x08588db8 JavaThread "http-80-Processor56" daemon [_thread_blocked, id=2117]
0x086a2130 JavaThread "http-80-Processor54" daemon [_thread_in_native, id=2115]
0x0878cfa8 JavaThread "http-80-Processor50" daemon [_thread_in_native, id=2108]
0x087b9148 JavaThread "http-80-Processor39" daemon [_thread_blocked, id=2097]
0x082f38c8 JavaThread "http-80-Processor29" daemon [_thread_blocked, id=2087]
0x087a6a30 JavaThread "http-80-Processor28" daemon [_thread_in_native, id=2086]
0x36b33540 JavaThread "http-80-Monitor" [_thread_blocked, id=1769]
0x351178e0 JavaThread "http-80-Processor4" daemon [_thread_blocked, id=1747]
0x35a72210 JavaThread "http-80-Processor1" daemon [_thread_blocked, id=1744]
0x363ed3b0 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=1743]
0x35a60238 JavaThread "Thread-4" [_thread_blocked, id=1742]
=>0x36bc7068 JavaThread "SessionReaperDaemon[null, 2005-11-01 22:38:23.388]" daemon [_thread_in_Java, id=1739]
0x363633c8 JavaThread "DistributedCache:DistributedSessions:EventDispatcher" daemon [_thread_blocked, id=1738]
0x363771a0 JavaThread "DistributedCache:DistributedSessions" daemon [_thread_blocked, id=1737]
0x36ba85f0 JavaThread "ReplicatedCache:ReplicatedSessionsMisc" daemon [_thread_blocked, id=1736]
0x36bbb118 JavaThread "TcpRingListener" daemon [_thread_in_native, id=1735]
0x36373478 JavaThread "Cluster" daemon [_thread_blocked, id=1733]
0x36351268 JavaThread "PacketPublisher" daemon [_thread_blocked, id=1732]
0x36350d60 JavaThread "PacketReceiver" daemon [_thread_blocked, id=1731]
0x36ba8858 JavaThread "PacketListenerN" daemon [_thread_in_native, id=1730]
0x36bb11e8 JavaThread "PacketListener1" daemon [_thread_in_native, id=1729]
0x36b9f530 JavaThread "Logger@9508926 3.0/315" daemon [_thread_blocked, id=1728]
0x081b0910 JavaThread "RMI TCP Accept-9000" daemon [_thread_in_native, id=1726]
0x081b5c90 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=1725]
0x081b57c0 JavaThread "Timer-0" daemon [_thread_blocked, id=1724]
0x08157b60 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1722]
0x081566e8 JavaThread "CompilerThread1" daemon [_thread_blocked, id=1721]
0x08155698 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1720]
0x08154578 JavaThread "AdapterThread" daemon [_thread_blocked, id=1719]
0x08153618 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1718]
0x081528e8 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=1717]
0x08148480 JavaThread "Finalizer" daemon [_thread_blocked, id=1716]
0x08148008 JavaThread "Reference Handler" daemon [_thread_blocked, id=1715]
0x0805c598 JavaThread "main" [_thread_in_native, id=1705]
Other Threads:
0x08145bc8 VMThread [id=1714]
0x082237b0 WatcherThread [id=1727]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
par new generation total 523840K, used 289536K [0x3d1c0000, 0x5d1c0000, 0x5d1c0000)
eden space 523392K, 55% used [0x3d1c0000, 0x4ec800f0, 0x5d0e0000)
from space 448K, 0% used [0x5d150000, 0x5d150000, 0x5d1c0000)
to space 448K, 0% used [0x5d0e0000, 0x5d0e0000, 0x5d150000)
concurrent mark-sweep generation total 1318912K, used 521347K [0x5d1c0000, 0xad9c0000, 0xad9c0000)
concurrent-mark-sweep perm gen total 30720K, used 30499K [0xad9c0000, 0xaf7c0000, 0xb19c0000)
Dynamic libraries:
08048000-08056000 r-xp 00000000 08:02 180163 /usr/java/jdk1.5.0_05/bin/java
08056000-08058000 rwxp 0000e000 08:02 180163 /usr/java/jdk1.5.0_05/bin/java
08058000-0aea7000 rwxp 00000000 00:00 0
2af40000-2b067000 rwxp 00001000 00:00 0
2b067000-2b06a000 ---p 00000000 00:00 0
2b06a000-2b0e8000 rwxp 00003000 00:00 0
2b0e8000-2b0eb000 ---p 00081000 00:00 0
2b0eb000-2b169000 rwxp 00084000 00:00 0
2b169000-2b16c000 ---p 00102000 00:00 0
2b16c000-2b1ea000 rwxp 00105000 00:00 0
2b1ea000-2b1ed000 ---p 00081000 00:00 0
2b1ed000-2b26b000 rwxp 00084000 00:00 0
2b26b000-2b26e000 ---p 00081000 00:00 0
2b26e000-2b2ec000 rwxp 00084000 00:00 0
2b2ec000-2b367000 r-xs 00000000 08:02 1784003 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xsdlib.jar
2b367000-2b382000 r-xs 00000000 08:02 1784002 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xmlrpc-1.2-b1.jar
2b382000-2b39d000 r-xs 00000000 08:02 1784001 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xml-apis.jar
2b39d000-2b476000 r-xs 00000000 08:02 1784000 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xercesImpl.jar
2b476000-2b554000 r-xs 00000000 08:02 1783999 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xalan.jar
2b554000-2b557000 ---p 00000000 00:00 0
2b557000-2b5d5000 rwxp 00003000 00:00 0
2b5d5000-2b5d8000 ---p 00081000 00:00 0
2b5d8000-2b656000 rwxp 00084000 00:00 0
2b656000-2b659000 ---p 00081000 00:00 0
2b659000-2b6d7000 rwxp 00084000 00:00 0
2b6d7000-2b6da000 ---p 00102000 00:00 0
2b6da000-2b758000 rwxp 00105000 00:00 0
2b758000-2b75b000 ---p 00183000 00:00 0
2b75b000-2b7d9000 rwxp 00186000 00:00 0
2b7d9000-2b7dc000 rwxp 00000000 00:00 0
2b7dc000-2b85a000 rwxp 00207000 00:00 0
2b85a000-2b85d000 ---p 00285000 00:00 0
2b85d000-2b8db000 rwxp 00288000 00:00 0
2b8db000-2b8de000 ---p 00102000 00:00 0
2b8de000-2b95c000 rwxp 00105000 00:00 0
2b95c000-2b95f000 ---p 00183000 00:00 0
2b95f000-2ba5d000 rwxp 00186000 00:00 0
2ba5d000-2ba60000 ---p 00182000 00:00 0
2ba60000-2bade000 rwxp 00185000 00:00 0
2bade000-2bae1000 ---p 00203000 00:00 0
2bae1000-2bb5f000 rwxp 00206000 00:00 0
2bb5f000-2bb62000 ---p 00284000 00:00 0
2bb62000-2bbe0000 rwxp 00287000 00:00 0
2bbe0000-2bbe3000 ---p 00305000 00:00 0
2bbe3000-2bc61000 rwxp 00308000 00:00 0
2bc61000-2bc64000 ---p 00204000 00:00 0
2bc64000-2bce2000 rwxp 00207000 00:00 0
2bce2000-2bce5000 ---p 00285000 00:00 0
2bce5000-2bd63000 rwxp 00288000 00:00 0
2bd63000-2bd66000 rwxp 00000000 00:00 0
2bd66000-2bde4000 rwxp 00186000 00:00 0
2bde4000-2bde7000 ---p 00204000 00:00 0
2bde7000-2be65000 rwxp 00207000 00:00 0
2be65000-2be68000 rwxp 00000000 00:00 0
2be68000-2bee6000 rwxp 00084000 00:00 0
2bee6000-2bee9000 ---p 00102000 00:00 0
2bee9000-2bf67000 rwxp 00105000 00:00 0
2bf67000-2bf6a000 ---p 00183000 00:00 0
2bf6a000-2bfe8000 rwxp 00186000 00:00 0
2bfe8000-2bfeb000 ---p 00204000 00:00 0
2bfeb000-2c069000 rwxp 00207000 00:00 0
2c069000-2c06c000 ---p 00081000 00:00 0
2c06c000-2c0ea000 rwxp 00084000 00:00 0
2c0ea000-2c0ed000 ---p 00102000 00:00 0
2c0ed000-2c16b000 rwxp 00105000 00:00 0
2c16b000-2c16e000 ---p 00183000 00:00 0
2c16e000-2c1ec000 rwxp 00186000 00:00 0
2c1ec000-2c1ef000 rwxp 00000000 00:00 0
2c1ef000-2c26d000 rwxp 00207000 00:00 0
2c26d000-2c270000 ---p 00285000 00:00 0
2c270000-2c2ee000 rwxp 00288000 00:00 0
2c2ee000-2c2f1000 ---p 00306000 00:00 0
2c2f1000-2c36f000 rwxp 00309000 00:00 0
2c36f000-2c372000 ---p 00387000 00:00 0
2c372000-2c3f0000 rwxp 0038a000 00:00 0
2c3f0000-2c3f3000 ---p 00204000 00:00 0
2c3f3000-2c471000 rwxp 00207000 00:00 0
2c471000-2c474000 ---p 00285000 00:00 0
2c474000-2c4f2000 rwxp 00288000 00:00 0
2c4f2000-2c4f5000 ---p 00102000 00:00 0
2c4f5000-2c573000 rwxp 00105000 00:00 0
2c573000-2c576000 ---p 00183000 00:00 0
2c576000-2c5f4000 rwxp 00186000 00:00 0
2c5f4000-2c5f7000 ---p 00204000 00:00 0
2c5f7000-2c675000 rwxp 00207000 00:00 0
2c675000-2c678000 ---p 00285000 00:00 0
2c678000-2c6f6000 rwxp 00288000 00:00 0
2c6f6000-2c6f9000 ---p 00306000 00:00 0
2c6f9000-2c777000 rwxp 00309000 00:00 0
2c777000-2c77a000 ---p 00387000 00:00 0
2c77a000-2c7f8000 rwxp 0038a000 00:00 0
2c7f8000-2c7fb000 ---p 00081000 00:00 0
2c7fb000-2c879000 rwxp 00084000 00:00 0
2c879000-2c87c000 ---p 00102000 00:00 0
2c87c000-2c8fa000 rwxp 00105000 00:00 0
2c8fa000-2c8fd000 ---p 00183000 00:00 0
2c8fd000-2c97b000 rwxp 00186000 00:00 0
2c97b000-2c97e000 rwxp 00000000 00:00 0
2c97e000-2c9fc000 rwxp 00207000 00:00 0
2c9fc000-2c9ff000 ---p 00285000 00:00 0
2c9ff000-2ca7d000 rwxp 00288000 00:00 0
2ca7d000-2ca80000 ---p 00285000 00:00 0
2ca80000-2cafe000 rwxp 00288000 00:00 0
2cafe000-2cb01000 rwxp 00000000 00:00 0
2cb01000-2cb7f000 rwxp 00309000 00:00 0
2cb7f000-2cb82000 ---p 00387000 00:00 0
2cb82000-2cc58000 rwxp 0038a000 00:00 0
2cc58000-2cd00000 ---p 00029000 00:00 0
2cd7f000-2cd82000 ---p 00000000 00:00 0
2cd82000-2cefe000 rwxp 00003000 00:00 0
2cefe000-2cf00000 ---p 00029000 00:00 0
2cf00000-2d0fe000 rwxp 0002b000 00:00 0
2d0fe000-2d100000 ---p 00038000 00:00 0
2d17b000-2d17e000 ---p 00000000 00:00 0
2d17e000-2d1fc000 rwxp 00003000 00:00 0
2d1fc000-2d1ff000 ---p 00081000 00:00 0
2d1ff000-2d27d000 rwxp 00084000 00:00 0
2d27d000-2d280000 rwxp 00000000 00:00 0
2d280000-2d2fe000 rwxp 00105000 00:00 0
2d2fe000-2d301000 ---p 00183000 00:00 0
2d301000-2d37f000 rwxp 00186000 00:00 0
2d37f000-2d382000 ---p 00204000 00:00 0
2d382000-2d600000 rwxp 00207000 00:00 0
2d67f000-2d682000 ---p 00000000 00:00 0
2d682000-2e1ff000 rwxp 00003000 00:00 0
2e1ff000-2e200000 ---p 00029000 00:00 0
2e200000-2e25d000 rwxp 0002a000 00:00 0
2e25d000-2e300000 ---p 00059000 00:00 0
2e300000-2e700000 rwxp 00000000 00:00 0
2e77f000-2e782000 ---p 00000000 00:00 0
2e782000-30500000 rwxp 00003000 00:00 0
3057f000-30582000 rwxp 00000000 00:00 0
30582000-30700000 rwxp 00003000 00:00 0
3077d000-30780000 ---p 00000000 00:00 0
30780000-307fe000 rwxp 00003000 00:00 0
307fe000-30801000 ---p 00081000 00:00 0
30801000-3087f000 rwxp 00084000 00:00 0
3087f000-30882000 ---p 00102000 00:00 0
30882000-30a00000 rwxp 00105000 00:00 0
30a7f000-30a82000 ---p 00000000 00:00 0
30a82000-30ffc000 rwxp 00003000 00:00 0
30ffc000-31000000 ---p 00029000 00:00 0
31000000-311fc000 rwxp 00095000 00:00 0
311fc000-31200000 ---p 00029000 00:00 0
31200000-313fc000 rwxp 0002d000 00:00 0
313fc000-31400000 ---p 000fd000 00:00 0
31416000-31434000 r-xs 00000000 08:02 1783998 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/wsdl4j.jar
31434000-31442000 r-xs 00000000 08:02 1783997 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/wrapper.jar
31442000-3147f000 r-xs 00000000 08:02 1783995 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/webInstaller.jar
3147f000-31482000 ---p 00000000 00:00 0
31482000-315ff000 rwxp 00003000 00:00 0
315ff000-31600000 ---p 00000000 00:00 0
31625000-3167f000 r-xs 00000000 08:02 1783994 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/velocity-1.4-rc1.jar
3167f000-31682000 ---p 00000000 00:00 0
31682000-31dfb000 rwxp 00003000 00:00 0
31dfb000-31e00000 ---p 00000000 00:00 0
31e14000-31e17000 ---p 00000000 00:00 0
31e17000-31e95000 rwxp 00003000 00:00 0
31e95000-31e98000 ---p 00081000 00:00 0
31e98000-31f16000 rwxp 00084000 00:00 0
31f16000-31f19000 ---p 00102000 00:00 0
31f19000-31f97000 rwxp 00105000 00:00 0
31f97000-31f9a000 ---p 00183000 00:00 0
31f9a000-32018000 rwxp 00186000 00:00 0
32018000-3201b000 ---p 00183000 00:00 0
3201b000-32099000 rwxp 00186000 00:00 0
32099000-3209c000 ---p 00204000 00:00 0
3209c000-3211a000 rwxp 00207000 00:00 0
3211a000-3211d000 ---p 00285000 00:00 0
3211d000-3219b000 rwxp 00288000 00:00 0
3219b000-3219e000 ---p 00102000 00:00 0
3219e000-3221c000 rwxp 00105000 00:00 0
3221c000-3221f000 ---p 00183000 00:00 0
3221f000-3229d000 rwxp 00186000 00:00 0
3229d000-322a0000 ---p 00204000 00:00 0
322a0000-3231e000 rwxp 00207000 00:00 0
3231e000-32321000 ---p 00285000 00:00 0
32321000-3239f000 rwxp 00288000 00:00 0
3239f000-323a2000 ---p 00306000 00:00 0
323a2000-32420000 rwxp 00309000 00:00 0
32420000-32423000 ---p 00387000 00:00 0
32423000-324a1000 rwxp 0038a000 00:00 0
324a1000-324a4000 ---p 00408000 00:00 0
324a4000-32522000 rwxp 0040b000 00:00 0
32522000-32525000 ---p 00285000 00:00 0
32525000-325a3000 rwxp 00288000 00:00 0
325a3000-325a6000 ---p 00102000 00:00 0
325a6000-32624000 rwxp 00105000 00:00 0
32624000-32627000 ---p 00183000 00:00 0
32627000-326a5000 rwxp 00186000 00:00 0
326a5000-326a8000 ---p 00204000 00:00 0
326a8000-32726000 rwxp 00207000 00:00 0
32726000-32729000 ---p 00285000 00:00 0
32729000-327a7000 rwxp 00288000 00:00 0
327a7000-327aa000 ---p 00102000 00:00 0
327aa000-32828000 rwxp 00105000 00:00 0
32828000-3282b000 rwxp 00000000 00:00 0
3282b000-328a9000 rwxp 00186000 00:00 0
328a9000-328ac000 ---p 00204000 00:00 0
328ac000-3292a000 rwxp 00207000 00:00 0
3292a000-3292d000 ---p 00285000 00:00 0
3292d000-329ab000 rwxp 00288000 00:00 0
329ab000-329ae000 rwxp 00000000 00:00 0
329ae000-32a2c000 rwxp 00309000 00:00 0
32a2c000-32a2f000 ---p 00183000 00:00 0
32a2f000-32aad000 rwxp 00186000 00:00 0
32aad000-32ab0000 rwxp 00000000 00:00 0
32ab0000-32b2e000 rwxp 00207000 00:00 0
32b2e000-32b31000 ---p 00000000 00:00 0
32b31000-32baf000 rwxp 00003000 00:00 0
32baf000-32bb2000 ---p 00081000 00:00 0
32bb2000-32c30000 rwxp 00084000 00:00 0
32c30000-32c33000 ---p 00102000 00:00 0
32c33000-32cb1000 rwxp 00105000 00:00 0
32cb1000-32cb4000 rwxp 00000000 00:00 0
32cb4000-32d32000 rwxp 00186000 00:00 0
32d32000-32d35000 rwxp 00000000 00:00 0
32d35000-32db3000 rwxp 00084000 00:00 0
32db3000-32db6000 ---p 00102000 00:00 0
32db6000-32e34000 rwxp 00105000 00:00 0
32e34000-32e37000 rwxp 00000000 00:00 0
32e37000-32eb5000 rwxp 00186000 00:00 0
32eb5000-32eb8000 ---p 00204000 00:00 0
32eb8000-32f36000 rwxp 00207000 00:00 0
32f36000-32f39000 rwxp 00000000 00:00 0
32f39000-32fb7000 rwxp 00288000 00:00 0
32fb7000-32fba000 ---p 00306000 00:00 0
32fba000-33038000 rwxp 00309000 00:00 0
33038000-3303b000 rwxp 00000000 00:00 0
3303b000-330b9000 rwxp 00186000 00:00 0
330b9000-330bc000 ---p 00204000 00:00 0
330bc000-3313a000 rwxp 00207000 00:00 0
3313a000-3313d000 ---p 00285000 00:00 0
3313d000-331bb000 rwxp 00288000 00:00 0
331bb000-331be000 ---p 00102000 00:00 0
331be000-3323c000 rwxp 00105000 00:00 0
3323c000-3323f000 ---p 00183000 00:00 0
3323f000-332bd000 rwxp 00186000 00:00 0
332bd000-332c0000 ---p 00204000 00:00 0
332c0000-3333e000 rwxp 00207000 00:00 0
3333e000-33341000 ---p 00285000 00:00 0
33341000-333bf000 rwxp 00288000 00:00 0
333bf000-333c2000 ---p 00306000 00:00 0
333c2000-33440000 rwxp 00309000 00:00 0
33440000-33443000 ---p 00183000 00:00 0
33443000-334c1000 rwxp 00186000 00:00 0
334e2000-33531000 r-xs 00000000 08:02 1783993 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/torque-3.0.2.jar
33531000-33537000 r-xs 00000000 08:02 1783992 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/textile4j-1.20.jar
33537000-336e9000 r-xs 00000000 08:02 1783991 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/tangosol.jar
336e9000-33710000 r-xs 00000000 08:02 1783990 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/ss_css2.jar
33710000-33716000 r-xs 00000000 08:02 1783989 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/smtp.jar
33716000-3371b000 r-xs 00000000 08:02 1783987 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/saaj.jar
3371b000-33720000 r-xs 00000000 08:02 1783986 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/relaxngDatatype.jar
33720000-33726000 r-xs 00000000 08:02 1783985 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/pop3.jar
33726000-3373e000 r-xs 00000000 08:02 1783984 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/p6spy.jar
3373e000-3375b000 r-xs 00000000 08:02 1783983 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/oscache-2.1.1.jar
3375b000-337ff000 r-xs 00000000 08:02 1783982 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/optional.jar
337ff000-3396a000 r-xs 00000000 08:02 1783981 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/msv.jar
3396a000-33979000 r-xs 00000000 08:02 1783980 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/msutil.jar
33979000-3398a000 r-xs 00000000 08:02 1783979 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/mssqlserver.jar
3398a000-339d1000 r-xs 00000000 08:02 1783978 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/msbase.jar
339d1000-339e8000 r-xs 00000000 08:02 1783977 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/metadata-extractor-2.2.2.jar
339e8000-33a15000 r-xs 00000000 08:02 1783976 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/mailapi.jar
33a15000-33a6c000 r-xs 00000000 08:02 1783975 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/log4j-1.2.8.jar
33a6c000-33a6e000 r-xs 00000000 08:02 1783974 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jdbc2_0-stdext.jar
33a6e000-33ab6000 r-xs 00000000 08:02 1783973 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jcs.jar
33ab6000-33ac2000 r-xs 00000000 08:02 1783972 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jazzy-core.jar
33ac2000-33acb000 r-xs 00000000 08:02 1783971 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jaxrpc.jar
33acb000-33b06000 r-xs 00000000 08:02 1783970 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jaxen-1.1-beta-4.jar
33b06000-33b0e000 r-xs 00000000 08:02 1783969 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jakarta-regexp-1.2.jar
33b0e000-33b1e000 r-xs 00000000 08:02 1783968 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jakarta-oro-2.0.6.jar
33b1e000-33c9f000 r-xs 00000000 08:02 1783967 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jai_core.jar
33c9f000-33cd3000 r-xs 00000000 08:02 1783966 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jai_codec.jar
33cd3000-33d03000 r-xs 00000000 08:02 1783965 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/isorelax.jar
33d03000-33d25000 r-xs 00000000 08:02 1783964 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/imap.jar
33d25000-33d54000 r-xs 00000000 08:02 1783963 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/hessian-3.0.13.jar
33d54000-33d58000 r-xs 00000000 08:02 1783962 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/fast-md5-2.5.jar
33d58000-33db8000 r-xs 00000000 08:02 1783961 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/dom4j-1.5-rc1.jar
33db8000-33df5000 r-xs 00000000 08:02 1783960 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/dnsjava-1.6.6.jar
33df5000-33ef6000 r-xs 00000000 08:02 1783958 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/db-ojb-1.0.1.jar
33ef6000-33f0f000 r-xs 00000000 08:02 1783957 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/crossdb.jar
33f0f000-33f1d000

An aside: When I switch to ParallelGC, afterrunning
for about 5-6 hours, GC times start doubling
(approximately) in time, eventually moving upwardsof
2 minutes per full GC.That sounds to me like you're swapping during full
GCs. I wouldn't assume that 2 GB of physical RAM is
sufficient to run an app with an 1800 MB heap. Have
you done any heap profiling to determine the real
working set of your application? When the GCs get
this pathologically long, is there disk activity on
the machine?Hmm.. So what should we size our heap to on a server with 2GB? The server is essentially only running the Java process. What would you consider "safe"?
While heap profiling with jconsole, the application only utilizes 1.1GB of the heap at peaks. We want to expand that eventually as we size our caches larger.
I don't believe there's disk activity with long GCs, but I can't confirm that 100% right now. The server is reading files from another machine, so the disks are more or less untouched while the app is running. There's heavy caching. The only reason disks would be hititng if your guess regarding the swapping turns out correct. After you suggest a safer way to calc a max heap size for our machine, I'll try again with ParallelGC and a smaller heap.
Some points:
- I tried increasing perm generation size. It'snot
even close to filled (according to gc log).(As you mentioned, "old generation" instead of "perm
generation".) What about when the full collection
times are taking 2 minutes in duration? Is there any
disk activity indicating swapping?Again, not positive on the disk situation -- should get an answer above to figure this out.
- I have not tried the -client VM, will this break
performance on a production application?The client VM does not generate as high quality
machine code as the server VM but on the x86
architecture many benchmark scores for the two
compilers are comparable. I would recommend you try
that option if only to narrow down the scope of the
crash you're seeing.
- I saw a bug related to a ConcMarkSweepGC crashon
4-cpu machines opened on Oct 17th. I've testedthis
on 4-cpu (2 proc with hyperthreading) and 2-cpu
(hyperthreading off) -- both exhibit this issue.What bug ID is this?http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6337544
- Are there any known issues with this collectorand
Linux 2.4.21-37.ELsmp? Should we be using
LD_ASSUME_KERNEL=2.2.5 or anything? For java to
recognize this variable, do we just need itspecified
anywhere in the environment at JVM start? How dowe
test if it worked?No issues as far as I know, but I am not a Linux
expert. I think in order to toggle which thread
library is used you just need that variable exported
in the environment when the JVM starts up. You can
probably use pmap to see which thread library got
loaded to verify the one you expect is present in the
loaded objects.Trying various combinations of LD_ASSUME_KERNEL to switch threading versions has no real effect on stability there. We're seeing crashes repeatedly still.
I'm going to attempt to deploy some machines with a new kernel as well and see if we can reproduce on those. I'll try ParallelGC again as soon as I get your recommendations on the configuration.

Applet hangs web server when upgrading from 1.6 to 1.7

I've got a client-side printing application which works fine on 1.6 and fails on 1.7 where it essentially hang the server on the connect option to the server and loading the jar file. Specifics:
PC Client running windows 7 x64
Browser versions IE9, IE10, Firefox 17.0.1 (PC-client) all fail
Client PC has latest Java 1.7.0 10-b 18 installed
Web server is Apache2.4.3 x64 running on Windows 2008 Server R2 through the network
Jar file is signed and the certificate is loaded
Apache server is sitting on another server through a network connection. Backrev java to 1.6 and it passes. I've specified the -Djava.net.prefIPv4Stack=true option in the applet call but it doesn't fix the issue.
The java console loads but then hangs on the connecting portion:
Java Plug-in 10.10.2.18
Using JRE version 1.7.0_10-b18 Java HotSpot(TM) Client VM
User home directory = C:\Users\user
c: clear console window
f: finalize objects on finalization queue
g: garbage collect
h: display this help message
l: dump classloader list
m: print memory usage
o: trigger logging
q: hide console
r: reload policy configuration
s: dump system and deployment properties
t: dump thread list
v: dump thread stack
x: clear classloader cache
0-5: set trace level to <n>
cache: Initialize resource manager: com.sun.deploy.cache.ResourceProviderImpl@1ae81ab
security: property package.access value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.
security: property package.access new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws
security: property package.access value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws
security: property package.access new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy
security: property package.access value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy
security: property package.access new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy,com.sun.jnlp
security: property package.definition value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.
security: property package.definition new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws
security: property package.definition value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws
security: property package.definition new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy
security: property package.definition value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy
security: property package.definition new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy,com.sun.jnlp
security: property package.access value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy,com.sun.jnlp
security: property package.access new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy,com.sun.jnlp,org.mozilla.jss
security: property package.definition value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy,com.sun.jnlp
security: property package.definition new value sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,com.sun.org.apache.xerces.internal.utils.,com.sun.org.apache.xalan.internal.utils.,com.sun.org.glassfish.external.,com.sun.org.glassfish.gmbal.,com.sun.javaws,com.sun.deploy,com.sun.jnlp,org.mozilla.jss
security: --- parseCommandLine converted : -Djava.net.preferIPv4Stack=true
into:
[-Djava.net.preferIPv4Stack=true]
basic: Added progress listener: sun.plugin.util.ProgressMonitorAdapter@8bbbf4
basic: Plugin2ClassLoader.addURL parent called for http://10.100.1.61/patholog/java/patholog/dist/PathologLibS.jar
security: Blacklist revocation check is enabled
security: Trusted libraries list check is enabled
network: Cache entry found [url: http://10.100.1.61/patholog/java/patholog/dist/PathologLibS.jar, version: null] prevalidated=true/0
cache: Adding MemoryCache entry: http://10.100.1.61/patholog/java/patholog/dist/PathologLibS.jar
cache: Resource http://10.100.1.61/patholog/java/patholog/dist/PathologLibS.jar has expired.
network: Connecting http://10.100.1.61/patholog/java/patholog/dist/PathologLibS.jar with proxy=DIRECT
network: Connecting http://10.100.1.61:80/ with proxy=DIRECT
network: Connecting http://10.100.1.61/patholog/java/patholog/dist/PathologLibS.jar with cookie "Patholog=vl141e491gffhpc07aupt0rnq5"
What is most irritating is that a local instance for debugging (i.e. apache2.4.3 x64) running locally and running the application on the same machine) passes perfectly so it appears that it involves a network connection rather than a localhost operation.
I noticed on a good connection the server returns with ResponseCode 304. I do not see that line in the log on the failing applet load. Apache logs set at debug show no errors even though the operation hangs the server.
Anyone got any ideas on how to debug and resolve this?
-Thanks
-G

More information:
Installed jre.1.6.37 (x32 version) along with 1.7.10 (x32 version) and the applet loads. Installed 1.7.10 (x32 version) only on the windows 7 x64 client fails. I've done both the x-java-applet method and the JNLP method and both hang. The JNLP j2se version is set to "1.6+" so in theory 1.7 should jre should work fine without a 1.6.37 loaded.
I've got wireshark dumps that I'm analyzing and noticed that there two OCSP responses (successful) but nothing after that, specifically where it hangs. Any ideas on how to debug this further?

When I bootup the icon for Elements shows up in orange. It changes to green only when I've gone to Finder and clicked on Elelments under the Devices list. This is a recent development. The icon used to be green upon bootup. Does anyone know why ?

When I bootup the icon for Elements shows up in orange. It changes to green only when I've gone to Finder and clicked on Elelments under the Devices list. This is a recent development. The icon used to be green upon bootup. Does anyone know why ?

I'm sorry to hear that.
I'm not affiliated w/ the developer, just a happy user that gave up fighting the apple podcast app a while ago. I used to have a bunch of smart playlists in itunes for my podcasts, and come home every day and pathologically synced my phone as soon as I walked in the door, and again before I walked out the door in the morning.
Since my wife was doing this too, we were fighting over who's turn it was to sync their phone.
Since I've switched to Downcast, I no longer worry about syncing my phone to itunes at all. I can go weeks between syncs.
Setup a "playlist" in downcast (ex., "Commute") and add podcasts to that playlist. Add another playlist ("walk" or "workout") and add different podcasts to that one.
Set podcast priorities on a per-feed basis (ex., high priority for some daily news feeds, medium priority for some favorite podcasts, lower priority for other stuff). Downcast will play the things in the priority you specify, and within that priority, it will play in date order (oldest to newest).
Allegedly, it will also sync your play status to other devices, although that is not a feature I currently use and can't vouch for. It uses apple's iCloud APIs, so to some extent may be limited by what Apple's APIs can do.

SpryMenuBar & IE

Hello,
I searched through but couldn't find an answer to this, so I've made a new thread. I wasn't sure whether to post this here or the main Dreamweaver forums, so hopefully here is correct.
I am using the SpryMenuBar to make a dropdown menu for a website I'm working on, and I'm experience the nasty IE bug where I don't get any background on the menubar, AND the display is horizontal for the dropdown menu instead of vertical as it should be. I'm using SpryMenuBar version 0.13 - Spry Pre-Release 1.6.1. If there is a patch or newer version of this where the bug is fixed, please point me in that direction. I couldn't find one, however. In any case, I'm pasting my code below, and the test site I'm using is http://www.umich.edu/~rdawane/precise/about_us.html. The correct view can be found using firefox, and the incorrect one in IE. Mainly just look at the menubar part of the code; that is where I am having my difficulties.
Thanks in advance!
html file:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width" />
<title>Precise Diagnostics, Inc.</title>
<script src="SpryMenuBar.js" type="text/javascript"></script>
<script src="DrawMain.js" type="text/javascript"></script>
<link href="stylesheet_desktop.css" rel="stylesheet" type="text/css" />

<link href="stylesheet_iphone.css" rel="stylesheet" type="text/css" media="only screen and (max-width:320px)" />

</head>
<body>
<br />
<div align="center">
<ul id="MenuBar" class="MenuBarHorizontal">
    <li class="MenuBarMain">
      <div align="center"><a href="about_us.html">About Us</a></div>
    </li>
    <li class="MenuBarMain"><a href="#" class="MenuBarItemSubmenu">Services</a>
      <ul>
        <li class="MenuBarMain"><a href="#" class="MenuBarItemSubmenu">Cytopathology   </a>
          <ul>
            <li class="MenuBarMain"><a href="#">Gynecological Cytology Services</a></li>
            <li class="MenuBarMain"><a href="#">Non-Gynecological Cytology Services</a></li>
          </ul>
        </li>
        <li class="MenuBarMain"><a href="#">Dermatopathology</a></li>
        <li class="MenuBarMain"><a href="#">Gynecologic pathology</a></li>
        <li class="MenuBarMain"><a href="#">H & E Stains</a></li>
        <li class="MenuBarMain"><a href="#">Immunohistochemical Stains</a></li>
        <li class="MenuBarMain"><a href="#">Oral Pathology</a></li>
        <li class="MenuBarMain"><a href="#">Podiatric Pathology</a></li>
        <li class="MenuBarMain"><a href="#">Special Stains</a></li>
        <li class="MenuBarMain"><a href="#">Gastrointestinal Pathology</a></li>
        <li class="MenuBarMain"><a href="#">Urological Pathology</a></li>
      </ul>
    </li>
    <li class="MenuBarMain"><a href="#">Our Pathologists</a>    </li>
    <li class="MenuBarMain"><a href="#">Patient Information</a></li>
    <li class="MenuBarMain"><a href="#">Careers</a></li>
    <li class="MenuBarMain"><a href="#">Contact Us</a></li>
</ul>
</div>
<script type="text/javascript">
var MenuBarMain = new Spry.Widget.MenuBar("MenuBar",{imgDown:"SpryAssets/SpryMenuBarDownHover.gif", imgRight:"SpryAssets/SpryMenuBarRightHover.gif"});
//alert("CW: " + getWidth() + "\nCH: " + getHeight() + "\nSW: " + screen.width + "\nSH: " + screen.height)
if (getDevice() == deviceIphone)
    document.write("<div class='center_apron'>");
    document.write("<div class='center_display'>");
    document.write("<img class='logo_position' src='logo-black-white-text-below-no-bg.gif' />");
else
    document.write("<div class='center_apron'>");
    document.write("<div class='center_display'>");
    document.write("<img class='logo_position' src='logo-black-white-text-below-no-bg.gif' />");
//    document.write("<div class='center_apron' style='width:" + (getWidth()*0.76) + "px; top:" + (getHeight()*0.083) + "px;'>");
//    document.write("<div class='center_display' style='width:" + (getWidth()*0.70) + "px;'>");
//    document.write("<img class='logo_position' src='logo-black-white-text-below-no-bg.gif' style='left:" + (-(getHeight()*0.02762)) + "'>");
document.write("<div> </div>");
t1draw("Laboratory Overview");
t3draw("Precise Diagnostics Laboratory is an independent CLIA certified full-service anatomical pathology laboratory dedicated to provide diagnostic excellence. Our facility utilizes state-of-the-art instrumentation and medical technology. Our experienced, board-certified   pathologists strive to meet the highest standards of pathology service.");
drawbreak();
t3draw("At Precise Diagnostics, Inc. a highly trained team of laboratory professionals provide fast, accurate, laboratory services in our quest for excellence.");
drawbreak();
t1draw("Certification & Accreditation");
t3draw("Precise Diagnostics, Inc. is a fully accredited, CLIA certified laboratory with license in New York.");
drawbreak();
t1draw("Hours of Operation");
t2draw("Laboratory");
t4draw("Monday through Saturday: 10 AM to 8 PM; Sunday: 10 AM to 5 PM");
drawbreak();
t2draw("Patient Billing Queries");
t4draw("Monday through Friday: 9 AM to 5 PM");
</script>



</div>
</div>
</body>
</html>
The .js file:
// SpryMenuBar.js - version 0.13 - Spry Pre-Release 1.6.1
// Copyright (c) 2006. Adobe Systems Incorporated.
// All rights reserved.
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
//   * Redistributions of source code must retain the above copyright notice,
//     this list of conditions and the following disclaimer.
//   * Redistributions in binary form must reproduce the above copyright notice,
//     this list of conditions and the following disclaimer in the documentation
//     and/or other materials provided with the distribution.
//   * Neither the name of Adobe Systems Incorporated nor the names of its
//     contributors may be used to endorse or promote products derived from this
//     software without specific prior written permission.
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
// POSSIBILITY OF SUCH DAMAGE.
SpryMenuBar.js
This file handles the JavaScript for Spry Menu Bar. You should have no need
to edit this file. Some highlights of the MenuBar object is that timers are
used to keep submenus from showing up until the user has hovered over the parent
menu item for some time, as well as a timer for when they leave a submenu to keep
showing that submenu until the timer fires.
(function() { // BeginSpryComponent
if (typeof Spry == "undefined") window.Spry = {}; if (!Spry.Widget) Spry.Widget = {};
Spry.BrowserSniff = function()
    var b = navigator.appName.toString();
    var up = navigator.platform.toString();
    var ua = navigator.userAgent.toString();
    this.mozilla = this.ie = this.opera = this.safari = false;
    var re_opera = /Opera.([0-9\.]*)/i;
    var re_msie = /MSIE.([0-9\.]*)/i;
    var re_gecko = /gecko/i;
    var re_safari = /(applewebkit|safari)\/([\d\.]*)/i;
    var r = false;
    if ( (r = ua.match(re_opera))) {
        this.opera = true;
        this.version = parseFloat(r[1]);
    } else if ( (r = ua.match(re_msie))) {
        this.ie = true;
        this.version = parseFloat(r[1]);
    } else if ( (r = ua.match(re_safari))) {
        this.safari = true;
        this.version = parseFloat(r[2]);
    } else if (ua.match(re_gecko)) {
        var re_gecko_version = /rv:\s*([0-9\.]+)/i;
        r = ua.match(re_gecko_version);
        this.mozilla = true;
        this.version = parseFloat(r[1]);
    this.windows = this.mac = this.linux = false;
    this.Platform = ua.match(/windows/i) ? "windows" :
                    (ua.match(/linux/i) ? "linux" :
                    (ua.match(/mac/i) ? "mac" :
                    ua.match(/unix/i)? "unix" : "unknown"));
    this[this.Platform] = true;
    this.v = this.version;
    if (this.safari && this.mac && this.mozilla) {
        this.mozilla = false;
Spry.is = new Spry.BrowserSniff();
// Constructor for Menu Bar
// element should be an ID of an unordered list (<ul> tag)
// preloadImage1 and preloadImage2 are images for the rollover state of a menu
Spry.Widget.MenuBar = function(element, opts)
    this.init(element, opts);
Spry.Widget.MenuBar.prototype.init = function(element, opts)
    this.element = this.getElement(element);
    // represents the current (sub)menu we are operating on
    this.currMenu = null;
    this.showDelay = 250;
    this.hideDelay = 600;
    if(typeof document.getElementById == 'undefined' || (navigator.vendor == 'Apple Computer, Inc.' && typeof window.XMLHttpRequest == 'undefined') || (Spry.is.ie && typeof document.uniqueID == 'undefined'))
        // bail on older unsupported browsers
        return;
    // Fix IE6 CSS images flicker
    if (Spry.is.ie && Spry.is.version < 7){
        try {
            document.execCommand("BackgroundImageCache", false, true);
        } catch(err) {}
    this.upKeyCode = Spry.Widget.MenuBar.KEY_UP;
    this.downKeyCode = Spry.Widget.MenuBar.KEY_DOWN;
    this.leftKeyCode = Spry.Widget.MenuBar.KEY_LEFT;
    this.rightKeyCode = Spry.Widget.MenuBar.KEY_RIGHT;
    this.escKeyCode = Spry.Widget.MenuBar.KEY_ESC;
    this.hoverClass = 'MenuBarItemHover';
    this.subHoverClass = 'MenuBarItemSubmenuHover';
    this.subVisibleClass ='MenuBarSubmenuVisible';
    this.hasSubClass = 'MenuBarItemSubmenu';
    this.activeClass = 'MenuBarActive';
    this.isieClass = 'MenuBarItemIE';
    this.verticalClass = 'MenuBarVertical';
    this.horizontalClass = 'MenuBarHorizontal';
    this.enableKeyboardNavigation = true;
    this.hasFocus = false;
    // load hover images now
    if(opts)
        for(var k in opts)
            if (typeof this[k] == 'undefined')
                var rollover = new Image;
                rollover.src = opts[k];
        Spry.Widget.MenuBar.setOptions(this, opts);
    // safari doesn't support tabindex
    if (Spry.is.safari)
        this.enableKeyboardNavigation = false;
    if(this.element)
        this.currMenu = this.element;
        var items = this.element.getElementsByTagName('li');
        for(var i=0; i<items.length; i++)
            if (i > 0 && this.enableKeyboardNavigation)
                items[i].getElementsByTagName('a')[0].tabIndex='-1';
            this.initialize(items[i], element);
            if(Spry.is.ie)
                this.addClassName(items[i], this.isieClass);
                items[i].style.position = "static";
        if (this.enableKeyboardNavigation)
            var self = this;
            this.addEventListener(document, 'keydown', function(e){self.keyDown(e); }, false);
        if(Spry.is.ie)
            if(this.hasClassName(this.element, this.verticalClass))
                this.element.style.position = "relative";
            var linkitems = this.element.getElementsByTagName('a');
            for(var i=0; i<linkitems.length; i++)
                linkitems[i].style.position = "relative";
Spry.Widget.MenuBar.KEY_ESC = 27;
Spry.Widget.MenuBar.KEY_UP = 38;
Spry.Widget.MenuBar.KEY_DOWN = 40;
Spry.Widget.MenuBar.KEY_LEFT = 37;
Spry.Widget.MenuBar.KEY_RIGHT = 39;
Spry.Widget.MenuBar.prototype.getElement = function(ele)
    if (ele && typeof ele == "string")
        return document.getElementById(ele);
    return ele;
Spry.Widget.MenuBar.prototype.hasClassName = function(ele, className)
    if (!ele || !className || !ele.className || ele.className.search(new RegExp("\\b" + className + "\\b")) == -1)
        return false;
    return true;
Spry.Widget.MenuBar.prototype.addClassName = function(ele, className)
    if (!ele || !className || this.hasClassName(ele, className))
        return;
    ele.className += (ele.className ? " " : "") + className;
Spry.Widget.MenuBar.prototype.removeClassName = function(ele, className)
    if (!ele || !className || !this.hasClassName(ele, className))
        return;
    ele.className = ele.className.replace(new RegExp("\\s*\\b" + className + "\\b", "g"), "");
// addEventListener for Menu Bar
// attach an event to a tag without creating obtrusive HTML code
Spry.Widget.MenuBar.prototype.addEventListener = function(element, eventType, handler, capture)
    try
        if (element.addEventListener)
            element.addEventListener(eventType, handler, capture);
        else if (element.attachEvent)
            element.attachEvent('on' + eventType, handler);
    catch (e) {}
// createIframeLayer for Menu Bar
// creates an IFRAME underneath a menu so that it will show above form controls and ActiveX
Spry.Widget.MenuBar.prototype.createIframeLayer = function(menu)
    var layer = document.createElement('iframe');
    layer.tabIndex = '-1';
    layer.src = 'javascript:""';
    layer.frameBorder = '0';
    layer.scrolling = 'no';
    menu.parentNode.appendChild(layer);
    layer.style.left = menu.offsetLeft + 'px';
    layer.style.top = menu.offsetTop + 'px';
    layer.style.width = menu.offsetWidth + 'px';
    layer.style.height = menu.offsetHeight + 'px';
// removeIframeLayer for Menu Bar
// removes an IFRAME underneath a menu to reveal any form controls and ActiveX
Spry.Widget.MenuBar.prototype.removeIframeLayer = function(menu)
    var layers = ((menu == this.element) ? menu : menu.parentNode).getElementsByTagName('iframe');
    while(layers.length > 0)
        layers[0].parentNode.removeChild(layers[0]);
// clearMenus for Menu Bar
// root is the top level unordered list (<ul> tag)
Spry.Widget.MenuBar.prototype.clearMenus = function(root)
    var menus = root.getElementsByTagName('ul');
    for(var i=0; i<menus.length; i++)
        this.hideSubmenu(menus[i]);
    this.removeClassName(this.element, this.activeClass);
// bubbledTextEvent for Menu Bar
// identify bubbled up text events in Safari so we can ignore them
Spry.Widget.MenuBar.prototype.bubbledTextEvent = function()
    return Spry.is.safari && (event.target == event.relatedTarget.parentNode || (event.eventPhase == 3 && event.target.parentNode == event.relatedTarget));
// showSubmenu for Menu Bar
// set the proper CSS class on this menu to show it
Spry.Widget.MenuBar.prototype.showSubmenu = function(menu)
    if(this.currMenu)
        this.clearMenus(this.currMenu);
        this.currMenu = null;
    if(menu)
        this.addClassName(menu, this.subVisibleClass);
        if(typeof document.all != 'undefined' && !Spry.is.opera && navigator.vendor != 'KDE')
            if(!this.hasClassName(this.element, this.horizontalClass) || menu.parentNode.parentNode != this.element)
                menu.style.top = menu.parentNode.offsetTop + 'px';
        if(Spry.is.ie && Spry.is.version < 7)
            this.createIframeLayer(menu);
    this.addClassName(this.element, this.activeClass);
// hideSubmenu for Menu Bar
// remove the proper CSS class on this menu to hide it
Spry.Widget.MenuBar.prototype.hideSubmenu = function(menu)
    if(menu)
        this.removeClassName(menu, this.subVisibleClass);
        if(typeof document.all != 'undefined' && !Spry.is.opera && navigator.vendor != 'KDE')
            menu.style.top = '';
            menu.style.left = '';
        if(Spry.is.ie && Spry.is.version < 7)
            this.removeIframeLayer(menu);
// initialize for Menu Bar
// create event listeners for the Menu Bar widget so we can properly
// show and hide submenus
Spry.Widget.MenuBar.prototype.initialize = function(listitem, element)
    var opentime, closetime;
    var link = listitem.getElementsByTagName('a')[0];
    var submenus = listitem.getElementsByTagName('ul');
    var menu = (submenus.length > 0 ? submenus[0] : null);
    if(menu)
        this.addClassName(link, this.hasSubClass);
    if(!Spry.is.ie)
        // define a simple function that comes standard in IE to determine
        // if a node is within another node
        listitem.contains = function(testNode)
            // this refers to the list item
            if(testNode == null)
                return false;
            if(testNode == this)
                return true;
            else
                return this.contains(testNode.parentNode);
    // need to save this for scope further down
    var self = this;
    this.addEventListener(listitem, 'mouseover', function(e){self.mouseOver(listitem, e);}, false);
    this.addEventListener(listitem, 'mouseout', function(e){if (self.enableKeyboardNavigation) self.clearSelection(); self.mouseOut(listitem, e);}, false);
    if (this.enableKeyboardNavigation)
        this.addEventListener(link, 'blur', function(e){self.onBlur(listitem);}, false);
        this.addEventListener(link, 'focus', function(e){self.keyFocus(listitem, e);}, false);
Spry.Widget.MenuBar.prototype.keyFocus = function (listitem, e)
    this.lastOpen = listitem.getElementsByTagName('a')[0];
    this.addClassName(this.lastOpen, listitem.getElementsByTagName('ul').length > 0 ? this.subHoverClass : this.hoverClass);
    this.hasFocus = true;
Spry.Widget.MenuBar.prototype.onBlur = function (listitem)
    this.clearSelection(listitem);
Spry.Widget.MenuBar.prototype.clearSelection = function(el){
    //search any intersection with the current open element
    if (!this.lastOpen)
        return;
    if (el)
        el = el.getElementsByTagName('a')[0];
        // check children
        var item = this.lastOpen;
        while (item != this.element)
            var tmp = el;
            while (tmp != this.element)
                if (tmp == item)
                    return;
                try{
                    tmp = tmp.parentNode;
                }catch(err){break;}
            item = item.parentNode;
    var item = this.lastOpen;
    while (item != this.element)
        this.hideSubmenu(item.parentNode);
        var link = item.getElementsByTagName('a')[0];
        this.removeClassName(link, this.hoverClass);
        this.removeClassName(link, this.subHoverClass);
        item = item.parentNode;
    this.lastOpen = false;
Spry.Widget.MenuBar.prototype.keyDown = function (e)
    if (!this.hasFocus)
        return;
    if (!this.lastOpen)
        this.hasFocus = false;
        return;
    var e = e|| event;
    var listitem = this.lastOpen.parentNode;
    var link = this.lastOpen;
    var submenus = listitem.getElementsByTagName('ul');
    var menu = (submenus.length > 0 ? submenus[0] : null);
    var hasSubMenu = (menu) ? true : false;
    var opts = [listitem, menu, null, this.getSibling(listitem, 'previousSibling'), this.getSibling(listitem, 'nextSibling')];
    if (!opts[3])
        opts[2] = (listitem.parentNode.parentNode.nodeName.toLowerCase() == 'li')?listitem.parentNode.parentNode:null;
    var found = 0;
    switch (e.keyCode){
        case this.upKeyCode:
            found = this.getElementForKey(opts, 'y', 1);
            break;
        case this.downKeyCode:
            found = this.getElementForKey(opts, 'y', -1);
            break;
        case this.leftKeyCode:
            found = this.getElementForKey(opts, 'x', 1);
            break;
        case this.rightKeyCode:
            found = this.getElementForKey(opts, 'x', -1);
            break;
        case this.escKeyCode:
        case 9:
            this.clearSelection();
            this.hasFocus = false;
        default: return;
    switch (found)
        case 0: return;
        case 1:
            //subopts
            this.mouseOver(listitem, e);
            break;
        case 2:
            //parent
            this.mouseOut(opts[2], e);
            break;
        case 3:
        case 4:
            // left - right
            this.removeClassName(link, hasSubMenu ? this.subHoverClass : this.hoverClass);
            break;
    var link = opts[found].getElementsByTagName('a')[0];
    if (opts[found].nodeName.toLowerCase() == 'ul')
        opts[found] = opts[found].getElementsByTagName('li')[0];
    this.addClassName(link, opts[found].getElementsByTagName('ul').length > 0 ? this.subHoverClass : this.hoverClass);
    this.lastOpen = link;
    opts[found].getElementsByTagName('a')[0].focus();
        //stop further event handling by the browser
    return Spry.Widget.MenuBar.stopPropagation(e);
Spry.Widget.MenuBar.prototype.mouseOver = function (listitem, e)
    var link = listitem.getElementsByTagName('a')[0];
    var submenus = listitem.getElementsByTagName('ul');
    var menu = (submenus.length > 0 ? submenus[0] : null);
    var hasSubMenu = (menu) ? true : false;
    if (this.enableKeyboardNavigation)
        this.clearSelection(listitem);
    if(this.bubbledTextEvent())
        // ignore bubbled text events
        return;
    if (listitem.closetime)
        clearTimeout(listitem.closetime);
    if(this.currMenu == listitem)
        this.currMenu = null;
    // move the focus too
    if (this.hasFocus)
        link.focus();
    // show menu highlighting
    this.addClassName(link, hasSubMenu ? this.subHoverClass : this.hoverClass);
    this.lastOpen = link;
    if(menu && !this.hasClassName(menu, this.subHoverClass))
        var self = this;
        listitem.opentime = window.setTimeout(function(){self.showSubmenu(menu);}, this.showDelay);
Spry.Widget.MenuBar.prototype.mouseOut = function (listitem, e)
    var link = listitem.getElementsByTagName('a')[0];
    var submenus = listitem.getElementsByTagName('ul');
    var menu = (submenus.length > 0 ? submenus[0] : null);
    var hasSubMenu = (menu) ? true : false;
    if(this.bubbledTextEvent())
        // ignore bubbled text events
        return;
    var related = (typeof e.relatedTarget != 'undefined' ? e.relatedTarget : e.toElement);
    if(!listitem.contains(related))
        if (listitem.opentime)
            clearTimeout(listitem.opentime);
        this.currMenu = listitem;
        // remove menu highlighting
        this.removeClassName(link, hasSubMenu ? this.subHoverClass : this.hoverClass);
        if(menu)
            var self = this;
            listitem.closetime = window.setTimeout(function(){self.hideSubmenu(menu);}, this.hideDelay);
        if (this.hasFocus)
            link.blur();
Spry.Widget.MenuBar.prototype.getSibling = function(element, sibling)
    var child = element[sibling];
    while (child && child.nodeName.toLowerCase() !='li')
        child = child[sibling];
    return child;
Spry.Widget.MenuBar.prototype.getElementForKey = function(els, prop, dir)
    var found = 0;
    var rect = Spry.Widget.MenuBar.getPosition;
    var ref = rect(els[found]);
    var hideSubmenu = false;
    //make the subelement visible to compute the position
    if (els[1] && !this.hasClassName(els[1], this.MenuBarSubmenuVisible))
        els[1].style.visibility = 'hidden';
        this.showSubmenu(els[1]);
        hideSubmenu = true;
    var isVert = this.hasClassName(this.element, this.verticalClass);
    var hasParent = els[0].parentNode.parentNode.nodeName.toLowerCase() == 'li' ? true : false;
    for (var i = 1; i < els.length; i++){
        //when navigating on the y axis in vertical menus, ignore children and parents
        if(prop=='y' && isVert && (i==1 || i==2))
            continue;
        //when navigationg on the x axis in the FIRST LEVEL of horizontal menus, ignore children and parents
        if(prop=='x' && !isVert && !hasParent && (i==1 || i==2))
            continue;
        if (els[i])
            var tmp = rect(els[i]);
            if ( (dir * tmp[prop]) < (dir * ref[prop]))
                ref = tmp;
                found = i;
    // hide back the submenu
    if (els[1] && hideSubmenu){
        this.hideSubmenu(els[1]);
        els[1].style.visibility = '';
    return found;
Spry.Widget.MenuBar.camelize = function(str)
    if (str.indexOf('-') == -1){
        return str;
    var oStringList = str.split('-');
    var isFirstEntry = true;
    var camelizedString = '';
    for(var i=0; i < oStringList.length; i++)
        if(oStringList[i].length>0)
            if(isFirstEntry)
                camelizedString = oStringList[i];
                isFirstEntry = false;
            else
                var s = oStringList[i];
                camelizedString += s.charAt(0).toUpperCase() + s.substring(1);
    return camelizedString;
Spry.Widget.MenuBar.getStyleProp = function(element, prop)
    var value;
    try
        if (element.style)
            value = element.style[Spry.Widget.MenuBar.camelize(prop)];
        if (!value)
            if (document.defaultView && document.defaultView.getComputedStyle)
                var css = document.defaultView.getComputedStyle(element, null);
                value = css ? css.getPropertyValue(prop) : null;
            else if (element.currentStyle)
                    value = element.currentStyle[Spry.Widget.MenuBar.camelize(prop)];
    catch (e) {}
    return value == 'auto' ? null : value;
Spry.Widget.MenuBar.getIntProp = function(element, prop)
    var a = parseInt(Spry.Widget.MenuBar.getStyleProp(element, prop),10);
    if (isNaN(a))
        return 0;
    return a;
Spry.Widget.MenuBar.getPosition = function(el, doc)
    doc = doc || document;
    if (typeof(el) == 'string') {
        el = doc.getElementById(el);
    if (!el) {
        return false;
    if (el.parentNode === null || Spry.Widget.MenuBar.getStyleProp(el, 'display') == 'none') {
        //element must be visible to have a box
        return false;
    var ret = {x:0, y:0};
    var parent = null;
    var box;
    if (el.getBoundingClientRect) { // IE
        box = el.getBoundingClientRect();
        var scrollTop = doc.documentElement.scrollTop || doc.body.scrollTop;
        var scrollLeft = doc.documentElement.scrollLeft || doc.body.scrollLeft;
        ret.x = box.left + scrollLeft;
        ret.y = box.top + scrollTop;
    } else if (doc.getBoxObjectFor) { // gecko
        box = doc.getBoxObjectFor(el);
        ret.x = box.x;
        ret.y = box.y;
    } else { // safari/opera
        ret.x = el.offsetLeft;
        ret.y = el.offsetTop;
        parent = el.offsetParent;
        if (parent != el) {
            while (parent) {
                ret.x += parent.offsetLeft;
                ret.y += parent.offsetTop;
                parent = parent.offsetParent;
        // opera & (safari absolute) incorrectly account for body offsetTop
        if (Spry.is.opera || Spry.is.safari && Spry.Widget.MenuBar.getStyleProp(el, 'position') == 'absolute')
            ret.y -= doc.body.offsetTop;
    if (el.parentNode)
            parent = el.parentNode;
    else
        parent = null;
    if (parent.nodeName){
        var cas = parent.nodeName.toUpperCase();
        while (parent && cas != 'BODY' && cas != 'HTML') {
            cas = parent.nodeName.toUpperCase();
            ret.x -= parent.scrollLeft;
            ret.y -= parent.scrollTop;
            if (parent.parentNode)
                parent = parent.parentNode;
            else
                parent = null;
    return ret;
Spry.Widget.MenuBar.stopPropagation = function(ev)
    if (ev.stopPropagation)
        ev.stopPropagation();
    else
        ev.cancelBubble = true;
    if (ev.preventDefault)
        ev.preventDefault();
    else
        ev.returnValue = false;
Spry.Widget.MenuBar.setOptions = function(obj, optionsObj, ignoreUndefinedProps)
    if (!optionsObj)
        return;
    for (var optionName in optionsObj)
        if (ignoreUndefinedProps && optionsObj[optionName] == undefined)
            continue;
        obj[optionName] = optionsObj[optionName];
})(); // EndSpryComponent
Finally, the .css file:
@charset "UTF-8";
/* SpryMenuBarHorizontal.css - version 0.6 - Spry Pre-Release 1.6.1 */
/* Copyright (c) 2006. Adobe Systems Incorporated. All rights reserved. */
LAYOUT INFORMATION: describes box model, positioning, z-order
.t1
    /*font-family: Calibri;*/
    font-weight: bold;
    font-size: 1.3em;
    display: block;
    padding-left: 0px;
.t2
    /*font-family: Calibri;*/
    font-weight: bold;
    font-size: 1em;
    display: block;
    padding-left: 20px;
.t3
    /*font-family: Calibri;*/
    font-weight: normal;
    font-size: 1em;
    display: block;
    padding-left: 20px;
.t4
    /*font-family: Calibri;*/
    font-weight: normal;
    font-size: 0.9em;
    display: block;
    padding-left: 40px;
.t5
    /*font-family: Calibri;*/
    font-weight: normal;
    font-size: 0.9em;
    display: inline;
    padding-left: 40px;
/* Main body text (list format)*/
li
    list-style-type: none;
    margin-left: 0px;
    padding-left: 0px;
    margin-bottom: 0px;
/* Menubar text (list format)*/
li.MenuBarMain
    font-family: Arial;
    font-weight: bold;
    font-size: 24px;
/* The outermost container of the Menu Bar, an auto width box with no margin or padding */
ul.MenuBarHorizontal
    margin: 0 auto; /* If using auto to center, width must be set to total menubar width */
    padding: 0;
    list-style-type: none;
    font-size: 100%;
    cursor: default;
    width: 48em; /* Change to the total width of your own menu bar */
/* Set the active Menu Bar with this class, currently setting z-index to accomodate IE rendering bug: http://therealcrisp.xs4all.nl/meuk/IE-zindexbug.html */
ul.MenuBarActive
    z-index: 1000;
/* Menu item containers, position children relative to this container and are a fixed width */
ul.MenuBarHorizontal li
    margin: 0;
    padding: 0;
    list-style-type: none;
    font-size: 100%;
    position: relative;
    text-align: center;
    cursor: pointer;
    width: 8em;
    float: left;
    /*text-wrap: none;*/
    /*white-space: inherit;*/
    /*top: 0px;*/
/* Submenus should appear below their parent (top: 0) with a higher z-index, but they are initially off the left side of the screen (-1000em) */
ul.MenuBarHorizontal ul
    margin: 0;
    padding: 0;
    list-style-type: none;
    font-size: 100%;
    z-index: 1020;
    cursor: default;
    width: auto;
/*    width: 8.2em;*/
    position: absolute;
    left: -1000em;
    background-color: #3093f5;
/* Submenu that is showing with class designation MenuBarSubmenuVisible, we set left to auto so it comes onto the screen below its parent menu item */
ul.MenuBarHorizontal ul.MenuBarSubmenuVisible
    left: auto;
/* Menu item containers are same fixed width as parent */
ul.MenuBarHorizontal ul li
    width: auto;
/*    width: 8.2em;*/
    text-align: left;
    background-color: #3093f5;
/* Submenus should appear slightly overlapping to the right (95%) and up (-5%) */
ul.MenuBarHorizontal ul ul
    position: absolute;
    margin: -5% 0 0 95%;
/* Submenu that is showing with class designation MenuBarSubmenuVisible, we set left to 0 so it comes onto the screen */
ul.MenuBarHorizontal ul.MenuBarSubmenuVisible ul.MenuBarSubmenuVisible
    left: auto;
    top: 0;
DESIGN INFORMATION: describes color scheme, borders, fonts
/* Submenu containers have borders on all sides */
ul.MenuBarHorizontal ul
    border: 1px solid #CCC;
/* Menu items are a light gray block with padding and no text decoration */
ul.MenuBarHorizontal a
    display: block;
    cursor: pointer;
    /*background-color: #FFF; */
    padding: 0.5em 0.75em;
    text-decoration: none;
    background-repeat: no-repeat;
    font-size: 0.83em;
    /*font-size: 19px;*/
    color: #000;
/* Menu items that have mouse over or focus have a blue background and white text */
ul.MenuBarHorizontal a:hover, ul.MenuBarHorizontal a:focus
    /* [disabled]background-color: #33C; */
    color: #FFF;
/* Menu items that are open with submenus are set to MenuBarItemHover with a blue background and white text */
ul.MenuBarHorizontal a.MenuBarItemHover, ul.MenuBarHorizontal a.MenuBarItemSubmenuHover, ul.MenuBarHorizontal a.MenuBarSubmenuVisible
    /*background-color: #33C;*/
    color: #FFF;
SUBMENU INDICATION: styles if there is a submenu under a given menu item
/* Menu items that have a submenu have the class designation MenuBarItemSubmenu and are set to use a background image positioned on the far right (95%) and centered vertically (50%) */
ul.MenuBarHorizontal a.MenuBarItemSubmenu
/*    background-image: url(SpryMenuBarDown.gif);*/
    background-repeat: no-repeat;
    background-position: 95% 50%;
/* Menu items that have a submenu have the class designation MenuBarItemSubmenu and are set to use a background image positioned on the far left (95%) and centered vertically (50%) */
ul.MenuBarHorizontal ul a.MenuBarItemSubmenu
    background-image: url(rightarrow.gif);
    background-repeat: no-repeat;
    background-position: 95% 50%;
/* Menu items that are open with submenus have the class designation MenuBarItemSubmenuHover and are set to use a "hover" background image positioned on the far left (95%) and centered vertically (50%) */
ul.MenuBarHorizontal a.MenuBarItemSubmenuHover
/*    background-image: url(SpryMenuBarDown.gif);*/
    background-repeat: no-repeat;
    background-position: 95% 50%;
/* Menu items that are open with submenus have the class designation MenuBarItemSubmenuHover and are set to use a "hover" background image positioned on the far left (95%) and centered vertically (50%) */
ul.MenuBarHorizontal ul a.MenuBarItemSubmenuHover
    background-image: url(rightarrowhover.gif);
    background-repeat: no-repeat;
    background-position: 95% 50%;
BROWSER HACKS: the hacks below should not be changed unless you are an expert
/* HACK FOR IE: to make sure the sub menus show above form controls, we underlay each submenu with an iframe */
ul.MenuBarHorizontal iframe
    position: absolute;
    z-index: 1010;
    filter:alpha(opacity:0.1);
/* HACK FOR IE: to stabilize appearance of menu items; the slash in float is to keep IE 5.0 from parsing */
@media screen, projection
    ul.MenuBarHorizontal li.MenuBarItemIE
        display: inline;
        f/loat: left;
        background: #FFF;
Body
    font-family: Calibri;
    color: black;
    background-image: url(background-plain-original.jpg);
    background-repeat: repeat-x;
.logo_position
    position: absolute;
    top: 20px;
    left: -20px;
    /*left: -20px;*/
.center_display
    top: 0px;
    /*width: 1000px;*/
    width: 700px;
    height: auto;
    position: relative;
    margin-left: auto;
    margin-right: auto;
    background-color: white;
    border-top-left-radius: 25px 18px;
    border-top-right-radius: 25px 18px;
    border-bottom-left-radius: 25px 18px;
    border-bottom-right-radius: 25px 18px;
    border-bottom: none;
    padding-top: 160px;
.center_apron
    /*top: 60px;*/
    /*width: 1080px;*/
    top: 100px;
    width: 780px;
    height: auto;
    position: relative;
    margin-left: auto;
    margin-right: auto;
    padding-bottom: 40px;
    background-color: white;
    border-top-left-radius: 25px 18px;
    border-top-right-radius: 25px 18px;
    border-bottom-left-radius: 25px 18px;
    border-bottom-right-radius: 25px 18px;
    border-width: .1em;
    border-style: solid;
    border-color: black;

Add or modify
ul.MenuBarHorizontal ul li { */ this will stop the submenu from showing horizontal */
    display: block;
    float: none !important;
    width: auto;
    white-space: nowrap;
ul.MenuBarHorizontal li.MenuBarItemIE { /* this will influence thr bg colour */
    background: transparent;
Gramps

Cache and/or Connection problems under load

I have a Kodo web app that's been running just fine in
production for many months now. However, recently the web
traffic has shot up by a huge amount, literally overnight.
But unfortunately, it's caused the app to fail very ungracefully
under the strain.
It's been a crazy few days, and I haven't been able to do
very much analysis because of higher priorities. But from
what I have been able to gleen, it now looks like Kodo is
the most likely culprit. From what I've read in other messages
here, it appears others may have been experiencing similar
problems.
My environment: Redhat Linux 8, Postgres 7.3.4 with the
included JDBC3 driver, Apache 1.3.x, Tomcat 4.1.x and the
webapp connector. Similar behavior was seen with Apache 2.x,
Tomcat 4.1.x and the JK2 connector (that was on the new machine
I setup to handle the new traffic, which, of course, died the
night before).
As I mentioned, this app has been running reliably for
months with no problems. But when placed under heavy load,
it appears to get into some sort of pathological state where
it slows down dramatically (asymptotically?) to the point where
it's effectively locked up. In one case, where the app was
left running for several hours in this state, requests were
taking 90 minutes to complete (normal is 1-5 seconds).
From what I can deduce, there seem to be four things
going on, three of which have been mentioned in recent threads
here:
1) Excessive memory consumption. When the app is
operating normally, I see fairly flat memory usage for
the JVM process. Under load, the JVM steadily expands
until it hits its heap limit. I've gotten OutOfMemory
exceptions with a heap size of 350MB, which should be plenty.
2) Level 2 cache locking issues. I've seen dozens of
threads waiting on a lock in the DataCache code. Not sure
if there's a deadlock happening here or just that the
threads are waiting on a lock that's being held for a long time.
3) Database Connection leaks or contention. I see threads
spinning in the DataSource code trying to get a connection.
I also see dozens of connections from the Postgres side which
seem to be sitting idle, but in the middle of a transaction.
When things get bad, I also see exceptions being thrown because
of timeouts waiting for a connection to become available. It's
a web app, PMs should not be tied up for more than a few seconds.
4) CPU usage pegged or nearly so for the JVM. I suspect
this is related to #3. Something very bad is going on here.
If I stop all inbound requests to the JVM when it's in this
bad state, it will continue to burn CPU at 90%+ for a very
long time. I think it will eventually finish what it's doing,
but I haven't had the luxury of waiting for it. It's definitely
not a linear slowdown proportional to the load.
Attached are my kodo.properties file and some thread stack
traces along with some comments. Any advice would be greatly
appreciated. This is not a complicated app nor am I doing
anything unusual. It doesn't seem logical that Kodo could
breakdown so dramatically under load, so I'm hoping it's some
sort of interaction thing that I can work around.
Thanks.
Ron Hitchens {mailto:[email protected]} RonSoft Technologies
(510) 494-9597 (Home Office) http://www.ronsoft.com
(707) 924-3878 (fax) Bit Twiddling At Its Finest
"Born with a broken heart" -Kenny Wayne Shepard

Please read prior posts regarding level 2 cache. It is unusable under stress
as far I am concerned. Basically entire cache gets locked on any database
read. Makes it very unscalable
Are you using 2.5.3? It will request a connection from a pool every time it
resolves reference to a PC even if it is cached in PM and therefore Kodo
does not need to read any. As result if you iterate over 100 objects in your
query and for each object resolve reference to another object (always the
same) kodo will request 100 database connections from the pool (and note
they issue rollback on every time they return a connection to the pool so
getting connection might be fairly expensive)
In conjunction with level 2 cache contention this causes application to go
into a stupor.
Try to go back to 2.5.2 (or may be 2.5.4 they promised in the near future
with a workaround) or use "persistent-manager" connection retention if you
discard PM after each HTTP invocation - it will take care of connection
pooling issue. As far as L2 cache I was unable to find any work around so
far - see if you might be better of without cache. You might if your object
graph is not very complex
"Ron Hitchens" <[email protected]> wrote in message
news:[email protected]...
>
I have a Kodo web app that's been running just fine in
production for many months now. However, recently the web
traffic has shot up by a huge amount, literally overnight.
But unfortunately, it's caused the app to fail very ungracefully
under the strain.
It's been a crazy few days, and I haven't been able to do
very much analysis because of higher priorities. But from
what I have been able to gleen, it now looks like Kodo is
the most likely culprit. From what I've read in other messages
here, it appears others may have been experiencing similar
problems.
My environment: Redhat Linux 8, Postgres 7.3.4 with the
included JDBC3 driver, Apache 1.3.x, Tomcat 4.1.x and the
webapp connector. Similar behavior was seen with Apache 2.x,
Tomcat 4.1.x and the JK2 connector (that was on the new machine
I setup to handle the new traffic, which, of course, died the
night before).
As I mentioned, this app has been running reliably for
months with no problems. But when placed under heavy load,
it appears to get into some sort of pathological state where
it slows down dramatically (asymptotically?) to the point where
it's effectively locked up. In one case, where the app was
left running for several hours in this state, requests were
taking 90 minutes to complete (normal is 1-5 seconds).
From what I can deduce, there seem to be four things
going on, three of which have been mentioned in recent threads
here:
1) Excessive memory consumption. When the app is
operating normally, I see fairly flat memory usage for
the JVM process. Under load, the JVM steadily expands
until it hits its heap limit. I've gotten OutOfMemory
exceptions with a heap size of 350MB, which should be plenty.
2) Level 2 cache locking issues. I've seen dozens of
threads waiting on a lock in the DataCache code. Not sure
if there's a deadlock happening here or just that the
threads are waiting on a lock that's being held for a long time.
3) Database Connection leaks or contention. I see threads
spinning in the DataSource code trying to get a connection.
I also see dozens of connections from the Postgres side which
seem to be sitting idle, but in the middle of a transaction.
When things get bad, I also see exceptions being thrown because
of timeouts waiting for a connection to become available. It's
a web app, PMs should not be tied up for more than a few seconds.
4) CPU usage pegged or nearly so for the JVM. I suspect
this is related to #3. Something very bad is going on here.
If I stop all inbound requests to the JVM when it's in this
bad state, it will continue to burn CPU at 90%+ for a very
long time. I think it will eventually finish what it's doing,
but I haven't had the luxury of waiting for it. It's definitely
not a linear slowdown proportional to the load.
Attached are my kodo.properties file and some thread stack
traces along with some comments. Any advice would be greatly
appreciated. This is not a complicated app nor am I doing
anything unusual. It doesn't seem logical that Kodo could
breakdown so dramatically under load, so I'm hoping it's some
sort of interaction thing that I can work around.
Thanks.
Ron Hitchens {mailto:[email protected]} RonSoft Technologies
(510) 494-9597 (Home Office) http://www.ronsoft.com
(707) 924-3878 (fax) Bit Twiddling At Its Finest
"Born with a broken heart" -Kenny Wayne Shepard
With cahce enabled, 2.5.3
Here the app had recently slowed down and then effectively locked up.
There where many outstanding web requests that were not receiving output.
At this point most threads seemed to be waiting at the same location.
There were a large number of active database connections and most of
them had open transactions (according to pg_stat_activity). The app
was not responding to any web requests.
It would seem that db transactions had been started, then the thread
got stuck for a long time on a synchronization lock in the cache lookup.
Below are two randomly chosen thread stack dumps.
Thread-72[1] where
[1] java.lang.Object.wait (native method)
[2] java.lang.Object.wait (Object.java:429)
[3]oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquir
e (WriterPreferenceReadWriteLock.java:169)
[4]com.solarmetric.kodo.runtime.datacache.AbstractCacheImpl.acquireReadLock
(AbstractCacheImpl.java:384)
[5]com.solarmetric.kodo.runtime.datacache.TimedDataCache.acquireReadLock
(TimedDataCache.java:256)
[6] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:595)
[7] com.solarmetric.kodo.runtime.StateManagerImpl.loadFields(StateManagerImpl.java:2,330)
[8] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:897)
[9] com.europeasap.data.City.jdoGetname (null)
[10] com.europeasap.data.City.getName (City.java:39)
[11] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:922)
[12] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
[13] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
[14] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
[15] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[16] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [17]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[18] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[19] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[20] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[21] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[22] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[23]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[24] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[25] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[26] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[27]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[28] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[29]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[30] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[31] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[32] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[33] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[35] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[36]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[37] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[38]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[39] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[40] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[41] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[42]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[43] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[44] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[45] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[46] org.apache.catalina.connector.warp.WarpConnection.run (null)
[47] java.lang.Thread.run (Thread.java:534)
Thread-64[1] where
[1] java.lang.Object.wait (native method)
[2] java.lang.Object.wait (Object.java:429)
[3]oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquir
e (WriterPreferenceReadWriteLock.java:169)
[4]com.solarmetric.kodo.runtime.datacache.AbstractCacheImpl.acquireReadLock
(AbstractCacheImpl.java:384)
[5]com.solarmetric.kodo.runtime.datacache.TimedDataCache.acquireReadLock
(TimedDataCache.java:256)
[6] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:595)
[7] com.solarmetric.kodo.runtime.StateManagerImpl.loadField(StateManagerImpl.java:2,248)
[8] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:899)
[9] com.europeasap.data.HotelPrices.jdoGetseasonalPrices (null)
[10] com.europeasap.data.HotelPrices.normalizeIndex(HotelPrices.java:113)
[11] com.europeasap.data.HotelPrices.getCost (HotelPrices.java:45)
[12] com.europeasap.logic.CostHelper.findLowestHotel(CostHelper.java:181)
[13] com.europeasap.logic.CostHelper.computeBasePackageCost(CostHelper.java:297)
[14] com.europeasap.logic.CostHelper.computeFinalPackageCost(CostHelper.java:246)
[15] com.europeasap.form.CustomerBookingForm.updateDisplayCosts(CustomerBookingForm.java:1,440)
[16] com.europeasap.form.CustomerBookingForm.updateCustomizeDisplayInfo(CustomerBookingForm.java:1,407)
[17] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:68)
[18] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[19] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [20]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[21] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[22] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[23] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[24] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[25] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[26]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[27] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[28] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[29] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[30]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[31] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[32]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[33] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[34] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[35] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[36] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[37]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[38] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[39]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[40] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[42] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[43] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[44] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[45]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[46] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[47] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[48] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[49] org.apache.catalina.connector.warp.WarpConnection.run (null)
[50] java.lang.Thread.run (Thread.java:534)
while running slow, 2.5.3
At this point, the app had been running several hours normally, then
apparently slowed down and locked up while I was away. When looking
at the app threads and database activity, everything appeared idle.
No transactions seemed to be open in the db. But the app was not
behaving normally. Web requests that did not make use of JDO worked
fine (but slow). But requests that hit the db either blocked or were
very slow to respond.
Looking back at the log, there had been a large number of requests
that threw exceptions because they could not get a connection within
five seconds.
Most threads were idle, waiting on read, but some were in the state
shown by the following two stack dumps. Unlike the cache threads above,
they did not seem to be waiting for a lock to be granted, they seemed
to be spinning in the connection management code. Apparently trying
to get a connection. I suspended and resumed the same thread repeatedly
and it always seemd to be doing the same thing. Single stepping was
very difficult because the debugger was slow to respond, apparently
because of other threads also busy spinning.
Postgres indicated that there where lots of connections open and
that they were all idle, so there should not have been a shortage
of connections in the pool. There are two mysteries here: 1) why
can't this thread get a connection? and 2) Why is it busy spinning?
Thread-56[1] where
[1]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
reStatement (PreparedStatementCache.java:184)
[2]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
reStatement (PreparedStatementCache.java:169)
[3] com.solarmetric.datasource.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:199)
[4]com.solarmetric.kodo.impl.jdbc.schema.dict.AbstractDictionary.isClosed
(AbstractDictionary.java:1,912)
[5]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnectionFromFact
ory (SQLExecutionManagerImpl.java:186)
[6] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnection(SQLExecutionManagerImpl.java:147)
[7]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.newSQLExecutionManag
er (JDBCStoreManager.java:828)
[8]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getSQLExecutionManag
er (JDBCStoreManager.java:714)
[9]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getDatastoreConnecti
on (JDBCStoreManager.java:287)
[10]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.getDatastoreCon
nection (DataCacheStoreManager.java:465)
[11] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:591)
[12] com.solarmetric.kodo.runtime.StateManagerImpl.loadFields(StateManagerImpl.java:2,330)
[13] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:897)
[14] com.europeasap.data.City.jdoGetname (null)
[15] com.europeasap.data.City.getName (City.java:39)
[16] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:922)
[17] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
[18] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
[19] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
[20] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[21] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [22]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[23] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[24] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[25] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[26] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[27] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[28]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[29] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[30] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[31] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[32]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[33] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[35] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[36] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[37] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[38] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[39]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[40] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[42] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[43]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[44] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[45] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[46] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[47]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[48] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[49] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[50] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[51] org.apache.catalina.connector.warp.WarpConnection.run (null)
[52] java.lang.Thread.run (Thread.java:534)
Thread-56[1] where
[1]com.solarmetric.datasource.DataSourceImpl$AbstractPool.findConnection
(DataSourceImpl.java:826)
[2] com.solarmetric.datasource.DataSourceImpl$AbstractPool.getConnection(DataSourceImpl.java:605)
[3] com.solarmetric.datasource.DataSourceImpl.getConnection(DataSourceImpl.java:363)
[4] com.solarmetric.datasource.DataSourceImpl.getConnection(DataSourceImpl.java:356)
[5]com.solarmetric.kodo.impl.jdbc.runtime.DataSourceConnector.getConnection
(DataSourceConnector.java:63)
[6]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnectionFromFact
ory (SQLExecutionManagerImpl.java:185)
[7] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnection(SQLExecutionManagerImpl.java:147)
[8]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.newSQLExecutionManag
er (JDBCStoreManager.java:828)
[9]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getSQLExecutionManag
er (JDBCStoreManager.java:714)
[10]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getDatastoreConnecti
on (JDBCStoreManager.java:287)
[11]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.getDatastoreCon
nection (DataCacheStoreManager.java:465)
[12]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.initialize
(DataCacheStoreManager.java:519)
[13] com.solarmetric.kodo.runtime.StateManagerImpl.loadInitialState(StateManagerImpl.java:215)
[14]com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectByIdFilter
(PersistenceManagerImpl.java:1,278)
[15] com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:1,179)
[16]com.solarmetric.kodo.runtime.datacache.query.CacheAwareQuery$CachedResultLis
t.get (CacheAwareQuery.java:432)
[17] java.util.AbstractList$Itr.next (AbstractList.java:421)
[18] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:919)
[19] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
[20] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
[21] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
[22] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[23] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [24]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[25] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[26] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[27] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[28] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[29] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[30]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[31] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[32] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[33] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[35] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[36]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[37] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[38] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[39] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[40] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[42] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[43]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[44] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[45]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[46] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[47] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[48] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[49]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[50] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[51] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[52] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[53] org.apache.catalina.connector.warp.WarpConnection.run (null)
[54] java.lang.Thread.run (Thread.java:534)
With cache disabled 2.4.3
This run was an accident. I inadvertently ran the app with the older
2.4.3 version of Kodo, with the cache disabled. This one got into trouble
almost immediately. It also seemed to lockup with lots of opentransactions
in the db. It's also interesting that these two threads also seem to be
hanging around the same method as in 2.5.3.
Thread-63[1] where 0x9f9
[1]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
reStatement (PreparedStatementCache.java:184)
[2] com.solarmetric.datasource.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:377)
[3]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.prepareStatementInter
nal (SQLExecutionManagerImpl.java:807)
[4]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executePreparedQueryI
nternal (SQLExecutionManagerImpl.java:761)
[5]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQueryInternal
(SQLExecutionManagerImpl.java:691)
[6] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQuery(SQLExecutionManagerImpl.java:372)
[7] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQuery(SQLExecutionManagerImpl.java:356)
[8] com.solarmetric.kodo.impl.jdbc.ormapping.ClassMapping.loadByPK(ClassMapping.java:950)
[9] com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.initialize(JDBCStoreManager.java:263)
[10] com.solarmetric.kodo.runtime.StateManagerImpl.loadInitialState(StateManagerImpl.java:174)
[11]com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectByIdFilter
(PersistenceManagerImpl.java:1,023)
[12] com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:942)
[13] com.solarmetric.kodo.impl.jdbc.ormapping.OneToOneMapping.load(OneToOneMapping.java:147)
[14] com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.load(JDBCStoreManager.java:375)
[15] com.solarmetric.kodo.runtime.StateManagerImpl.loadField(StateManagerImpl.java:2,035)
[16] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:720)
[17] com.europeasap.data.CityMarkup.jdoGetcity (null)
[18] com.europeasap.data.CityMarkup.getCity (CityMarkup.java:30)
[19] com.europeasap.logic.CostHelper.getCityMarkup (CostHelper.java:81)
[20] com.europeasap.logic.CostHelper.computeBasePackageCost(CostHelper.java:289)
[21] com.europeasap.logic.CostHelper.computeFinalPackageCost(CostHelper.java:246)
[22] com.europeasap.form.CustomerBookingForm.updateDisplayCosts(CustomerBookingForm.java:1,440)
[23] com.europeasap.form.CustomerBookingForm.updateCustomizeDisplayInfo(CustomerBookingForm.java:1,407)
[24] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:68)
[25] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[26] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [27]
org.apache.struts.action.ActionServlet.doPost (ActionServlet.java:510)
[28] javax.servlet.http.HttpServlet.service (HttpServlet.java:760)
[29] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[30] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[31] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[32] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[33]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[34] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[35] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[36] org.apache.catal

Polite use of these forums

To Apple Community Forum Moderators:
I have owned Apples for almost twenty years and periodically have found myself in need of advice on an Apple hardware issue, network compatibility, or Apple Application issue. The Apple website “recommends” that we go to these Forums for information it seems like a good idea. It has the potential to expand technical help to those of us that have a problem with using a Mac or an Apple application. However, in every case that I have accessed the Forum I have either viewed attacks by some of what I'll call the “loyal users” against others that note problems with Apple, or have been the object of nasty responses myself:
- Early on I engaged the Forum about the difficulty of getting Apples to work in a Windows Network Environment. When I got on I saw nasty comments about similar questions from other users chastising the user for being on a Windows network. Well there is little one can do about that. The institution's leadership decides on what kind of network will be used, not an individual user. To dismiss this problem is unrealistic at best and arrogant at worst.
- After Microsoft moved to Office 2007 and left Mac users in the lurch I purchased iWork based upon what users in the Forum were lauding about about the Suite and what Apple said about it. I found it limited (from what I could gain from being a new user) and sought out answers to questions I had. The equation editor (the industry standard) at the time could not be used in iWork. So I pointed this out on the Forum. I got flamed by this guy that said that this was not appropriate place to point out limitations. I pointed out that Apple Developers would be nuts not to monitor these forums so it would be appropriate to point out functional limitations for future versions of iwork. All I got from the moderator was that I shouldn't have posted it. Others in this thread were flamed because they wanted more functionality from Numbers. They were told that anyone who is serious about analysis would use another database program (Excel maybe? - they didn't identify what program that might be.)
- Just recently I sought out the Forum because I have had this laptop for two years and am having to replace my battery the second year in a row. I tried to get Apple Help on this after my first replacement and did (what I thought was) everything correct. Someone else was having a similar problem. I related my understanding of how to evaluate battery performance (cycle count), and pointed out that Apple has had a problem with defective batteries (most notably mine) and that users need to watch these metrics closely. Well apparently according to one reply post I mis-stated how to evaluate cycle count (though I think I am still correct on that – our IT people basically told me what to look for.) The reply was that I had not really read the manual on batteries. Well I did, though I could have misunderstood what the battery document was trying to say. Instead of accusing me of lying they could have elaborated on why my interpretation was mistaken.
Now in all these cases and more there are social forum communication flaws that Apple needs to be aware of. This flaw is a social pathology that is characteristic of the medium. First attacking a frustrated user is not a constructive exercise. There is a difference between a negative comment about Apple or its applications and negative comments directed toward users. A negative comment at an Apple Inc. product should not be viewed as some kind attack on all users or a violation of a misguided religious doctrine. It should be viewed as a window into a potential improvement. Apple should be “man enough” to take those graciously. This gives Apple a window into problems in either its Help Manuals or its design. Contrary to trying to stifle comments like this Apple should encourage this. (In fact after I complained about the inability of iWork to use Mathtype - Decision Sciences - an upgrade of iWork works with Mathtype.) In a similar manner Politicians in our state regularly lambast the University I work with. I don't have a problem with their attack on the Institution. It goes with the territory. I do have a problem if they focus on a particular faculty member. Apple Inc. can handle criticisms.
On the other hand the “loyal user's” attacks or nasty comments on users that are critical of Apple or frustrated users are another issue. We purchase the hardware or software and expect it to work based upon what we interpret from Apple advertising claims. It is not working the way it is advertised. We may not be accessing the computer correctly, accessing the Help Manuals correctly, using the Apps correctly, or there may be a problem with the product itself. That is why we go on these Forums. To get chastised or insulted by a “loyal user” does not solve our problem and it does not make us want to tell other people how great Macs are. (My experience with iWork pushed me to OpenOffice - which I am writing this letter in rather than Word or iWork.)
More importantly though it makes us very hesitant about engaging in this social forum about a problem we have with an App or hardware. Ultimately this could mean that many technical/business users such as myself will migrate away from Apple because they can't get the solution they need and the mere size of Apples competitors (e.g. Microsoft) means that user support is much more available. We can forego the nastiness by changing products.
Now this is not say that these issues I raise are unique to Apple Community Forums. I was just on an OpenOffice community forum and saw similar behavior so this really is a social pathology that is part of this form of communication. But moderators need to be aware of this pathology if the company wants users to rely on this medium for good advice. Apple hopefully views these forums as ways to see where they could improve products or add functionality to preserve or enhance market share as well as providing broad technical support. But moderators need to more forcefully impose politeness and gracious responses to users that may not sit well with the “loyal user” part of the community. Thanks.

Roger Coupal wrote:
However, in every case that I have accessed the Forum I have either viewed attacks by some of what I'll call the “loyal users” against others that note problems with Apple, or have been the object of nasty responses myself:
"Every" time? I waste a lot of time here and I very rarely see anything that could be construed as "nasty". I read through all of your postings and I can't find any response that could be considered "nasty" or an "attack". In one case you posted some incorrect information and a couple of people called you on it. That's life and it happens to the best of us. No need to get upset about it.
Early on I engaged the Forum about the difficulty of getting Apples to work in a Windows Network Environment. When I got on I saw nasty comments about similar questions from other users chastising the user for being on a Windows network. Well there is little one can do about that. The institution's leadership decides on what kind of network will be used, not an individual user. To dismiss this problem is unrealistic at best and arrogant at worst.
Maybe you should increase your participation. You've only posted 17 times in 3 years. If you were at Level 2 (pretty easy to reach) you would have the ability to automatically alert the moderators about any postings that violate the Apple Terms of Use. Until then, it would be better to notify the moderators via this forum about specific posting that you think are improper. Otherwise, it is just hearsay. Perhaps those postings were from known trolls or people who are otherwise "looking for a fight". People often only start posting here when they have problems and, unfortunately, they sometimes wait until they are already very angry and upset. When they start threads with an argumentative tone, it usually doesn't end well.
I can't really say much about Windows networking. All of MacOS X uses SAMBA, so if you can access it via xterm, you can access via the Finder. Maybe start a new thread with more details and keep trying. Windows networking is very complex and SAMBA is just an open-source reverse-engineering effort.
- After Microsoft moved to Office 2007 and left Mac users in the lurch I purchased iWork based upon what users in the Forum were lauding about about the Suite and what Apple said about it. I found it limited (from what I could gain from being a new user) and sought out answers to questions I had. The equation editor (the industry standard) at the time could not be used in iWork. So I pointed this out on the Forum. I got flamed by this guy that said that this was not appropriate place to point out limitations.
"Flamed"? Again, I read those messages. That was hardly being "flamed". Plus, it had nothing to do with you pointing out limitations.
I pointed out that Apple Developers would be nuts not to monitor these forums so it would be appropriate to point out functional limitations for future versions of iwork.
Sorry, but that is not correct. There is way too much disinformation here for developers at Apple to get meaningful information out of it. If people find bugs or limitations they are usually encouraged to contact Apple via the appropriate feedback channels to improve the products. That actually happened in your case.
No one outside Apple really knows if Apple does or does not read these forums. The official word is that they don't and I accept that. This is a user-to-user tech support forum. It is not a place to complain, report bugs, or suggest enhancements. There are better places for that. This discussion forum is meant to help end-users, not Apple. Apple is a big, multinational corporation with lots of money. They have people whose job it is to track down complains, fix bugs, and enhance products.
All I got from the moderator was that I shouldn't have posted it.
There was no moderator posting in that thread. You can identify Apple employee by the Apple logo under their name.
Others in this thread were flamed because they wanted more functionality from Numbers. They were told that anyone who is serious about analysis would use another database program (Excel maybe? - they didn't identify what program that might be.)
You are the one who mentioned Excel. Someone correctly pointed out that if you are already a user of advanced Excel features you might not be happy with a 1.0 application like Numbers. Excel is 25 years old. No one expects Numbers to match it for functionality. But it might be easier for casual users than Excel. Again, there was no flaming going on.
The reply was that I had not really read the manual on batteries. Well I did, though I could have misunderstood what the battery document was trying to say. Instead of accusing me of lying they could have elaborated on why my interpretation was mistaken.
They didn't accuse you of lying and they did elaborate on why your interpretation was mistaken. You provided misinformation and suggested people take their batteries in to the Apple store for a problem that doesn't exist.
Now in all these cases and more there are social forum communication flaws that Apple needs to be aware of. This flaw is a social pathology that is characteristic of the medium.
I think Apple is well aware of that. That is why they have a good system of moderation that they have been working on for several years. This forum is much better than anything you will find anywhere else. I'm sorry, but you have no idea what a true "flame" really is
First attacking a frustrated user is not a constructive exercise. There is a difference between a negative comment about Apple or its applications and negative comments directed toward users. A negative comment at an Apple Inc. product should not be viewed as some kind attack on all users or a violation of a misguided religious doctrine.
Sometimes people just want to vent. It is pretty clear when someone starts a thread where they don't really want a solution but just want to complain. I always try to give them a solution, while at the same time tearing down all their logical fallacies. Sometimes they are just frustrated and are happy to get the problem resolved. Sometimes they take the bait and keep fighting. Either I've solved someone's problem or I have provided a solution to some jerk who then won't accept it just because it came from me. Either way I'm pleased.
It should be viewed as a window into a potential improvement. Apple should be “man enough” to take those graciously. This gives Apple a window into problems in either its Help Manuals or its design.
Again, you are assuming Apple reads these forums. They don't. These forums are for your benefit, not Apple's.
To get chastised or insulted by a “loyal user” does not solve our problem and it does not make us want to tell other people how great Macs are.
I gave up evangelizing a long time ago. At this point, everyone knows about Macs. If people are too boneheaded to use Macs, I'm content to let them suffer with Windows or Linux.
(My experience with iWork pushed me to OpenOffice - which I am writing this letter in rather than Word or iWork.)
Really? Safari has really nice text editing features with a built-in spell checker. Also, what's wrong with Word? You are obviously a power user, why not use Office? Microsoft got their start with Office, which started out as a Mac program and is still one of the best Mac programs you can buy.
I was just on an OpenOffice community forum and saw similar behavior so this really is a social pathology that is part of this form of communication.
So, in other words, you just invalidated your whole thesis?
But moderators need to more forcefully impose politeness and gracious responses to users that may not sit well with the “loyal user” part of the community.
Moderators really only get involved when there are violations of the terms of use. Being polite is part of that, but graciousness isn't. I think if you continue to post and start answering questions you will soon be able to easily report postings you think are "impolite". Hopefully, when you get to that level, you will be able to better identify a true "flame".

A way to sync tasks into one calendar on N8, appoi...

I am currently syncing appointments into one calendar and tasks into another, on my Nokia N8. Why do I do this? Skip about halfway down this post, for my rationale.
I'd like to know if there's an easier way to get this done.
I am using a Nokia N8 and syncing with Outlook 2007.
I have both Ovi Suite and Nokia PC Suite (7.1.30.9, which is one of the last versions that doesn't prohibit a connection with the N8). And, I've synced succesfully with both of them.
Nokia PC Sync is set to synchronize tasks and files/folders, but not appointments or contacts. I set it to sync an empty Calendar folder within Outlook. But, it syncs my normal Tasks folder. That way, Tasks go into a calendar on the N8 called "PC Sync."
Then, Ovi Suite synchronizes appointments and contacts, but not tasks or files/folders. I set it to sync an empty Tasks folder within Outlook, but it syncs my normal Calendar folder. Ovi Suite places the appointments into the default N8 calendar called "Personal"
Is there a way, preferrably using Ovi Sync, where I can place tasks into one calendar, on the N8, and Appointments into a different calendar?
Why do I do this? Nokia didn't provide a way to exclude tasks from any of the calendar views (but provided a task-only view - what were they thinking!?). This means the daily and weekly views are filled with tasks, all set ahead of the appointments. I cannot imagine why anyone would want that to constitute their default view of the calendar. Particularly since there is a view available, of only tasks.
By placing tasks in a separate calendar from appointments, I can pick and choose whether I want to see tasks with appointments, in the daily and weekly views.
Now, if only Symbian would be able to sync the categories associated with my tasks and appointments, in Outlook. . . But, I realize (even after years of hoping Nokia would introduce the concept of categories to its software) that you have to walk, before you can run.
I hope you have found this post to be entertaining. But, back to my original question: Is there any way to sync - using only one software solution on the PC, tasks into one of the Nokia N8's calendars and appointments into another?
Thanks for any ideas.

Here is the text of my feedback letter regarding Tasks...
Come on Apple! You have to be kidding me! I got caught up in the iPhone 3G euphoria and blindly ordered my 16GB Black about a week ago. (still waiting for it BTW) I had not purchased an iPhone Edge specifically because of its lack of Exchange connectivity. I have been "suffering" with an AT&T Tilt (HPC 8925, TyTn II) for the past year while I waited for Apple to "catch up" as it were.
Well, imagine my surprise a few minutes ago when I read that "Exchange ActiveSync Support" on the iPhone is not as capable as it is on a Windows Mobile device or even a Blackberry for that matter!
Look, any junior Exchange admin knows how important Public Folders are in the enterprise. There is absolutely no excuse for the lack of Public Folder support on the iPhone. This is a serious issue that will lose Apple and AT&T important high use, high revenue enterprise customers.
Then there is the issue of tasks. It has been my experience that many Exchange users utilize the Tasks feature extensively. I am dumbfounded that task support is missing from the iPhone.
Finally comes Notes. While it might not be used as much as Public Folders and Tasks, from the outside looking in it seems that integrating Exchange Notes would be a fairly simple endeavor.
PLEASE tell me and the rest of your customers (and potential customers) that support for Exchange Tasks, Notes, and Public Folders is in the works. We need something integrated just like mail, calendars, and contacts, not some patch-on workaround.
I and the rest of your users look forward to your response.
Sincerely,
eddiela

An Apology to SDN and a Request for Advice

After thinking about Mark Finnern's suggestion yesterday that I stop blogging so frequently, I realized that I had made an honest mistake about the way in which I was using the SDN blog area.
Let me try to explain how and why this mistake occurred.
First, it goes without saying that what SDN does well, it does extremely well. I have used the Expert Forums on several critical occasions in the past few months, and they function "above and beyong". I also keep a list of "good_sdn_links" when I see great guides and explanations in blog posts, so I don't have to bother searching for them later, and here again, SDN is doing its job extremely well.
But the fact that SDN does certain things very very well doesn't mean it is doing everything it should be doing.
What do I mean by this?
Well, many people here at SDN would agree with me that there are many things terribly wrong with the IT community today and inside the SAP community as well. Furthermoe, it's my opinion that SDN is doing nothing to try and change these things, when it in fact can and should be trying to change them.
So I decided to do my part as a "change agent" by posting a number of "perspective" posts which tried to communicate what I thought was wrong with IT and SAP today.
But here's where the mistake came in.
First, I figured from my read count that I must be doing something right, otherwise, why would my blog posts have @52,000 reads since 10/1/2006, or about 440 reads per calendar day?
Second, for all the criticism I received in personal emails about my blogs, I also received validation at SDN. Let me give you an example of this.
Two well-respected and long-time members of the SDN community indicated to me in personal emails that they had a lot of problems with my two long series of blogs on the WDA component WDR_TEST_UI_ELEMENTS and how it could be very easily converted to do many other useful things.
But at the same time as I was receiving this criticsm , I responded to Thomas Szuecs in one of his threads and therefore took an opportunity to ask him if he knew the SAP developer of WDR_TEST_UI_ELEMETNS, and if so, to convey my gratitude to the developer of WDR_TEST_UI_ELEMENTS for writing such a wonderful piece of code. He responded that he did know this developer and that this developer liked the series very much,
Third, I have received blog responses from several SDNers indicating that my "perspective posts" were registering with them. The one of these that I treasure most is from the SDNer who posted the following: "I like the way you put your finger where it hurts - keep it up."
So, you can see from the above three points how the mistake occurred - I figured that if I had 440 reads per day over four months, and if my technical posts were OK with key SAP developers, and if at least some readers were responding with indications that they understood what I was trying to do in my "perspective" posts, then I must be doing something right.
But if you've been following some recent conversations in this Forum, you'll realize that I was doing something quite wrong.
What I was doing wrong with my "perspective" posts was trying to use the blog space at SDN to effect change in the SAP community and therefore, in the larger IT community.
And if you think about it, this was wrong for a very obvious reason: SDN has this "Suggestions" Forum that I'm posting in right now.
So if I was really serious about trying to get SDN to become a change agent to help improve the SAP community and the larger IT community, then I should have been posting a particular kind of post to the Suggestion Forum, instead of "perspective" blogs.
In particular, I should have been making Suggestion Forum posts which:
a) clearly identify a problem in the SAP community that SDN can actually help to change;
b) offering a concrete suggestion as to how SDN can help to change this problem.
So - having finally realized my mistake, I want to start posting posts to this Suggestion Forum which do (a) and (b).
But before I do, I want to ask two questions up front: one about frequency and one about content.
With respect to frequency, how often should I make one of these posts to the Suggestion Forum - weekly, every other day - you folks tell me.
With respect to content, are there any topics that any one thinks inappropriate for a suggestion post? In particular, would anyone object to Suggestion Forum posts on the following topics:
a) in the SAP community (outside BI/A), matters relating to real business computation are being ignored in favor of an emphasis on tool technologies that have nothing to do with real business computation, but are important to SAP's marketing strategies; what can SDN to change this?
b) in the SAP community, not enough support is being provided to the many external SAP developers who are stuck on systems at or above 4.6c and below 04s; what can SDN do to change this?
c) in the SAP community, there is not enough awareness of deficiencies in the relational database model, and how carefully it must be used; what can SDN do to change this?
d) in the SAP community, there is not enough awareness of how various "organizational pathologies" at customer sites constrain the ways in which SAP can be used to benefit customers; what can SDN do to change this?
e) in the SAP community, there is far too little focus on SAP metadata, particularly at a time when SAP is trying to move into the Enterprise SOA paradigm.
To sum up:
1) If you think Suggestion Posts in any of these five areas are inappropriate, please take a moment to let me know.
2) If you think that Suggestion Posts should also be limited to a certain frequency, please also take a moment to let me know.
3) If you've gotten this far in reading this post, thanks very much for your patience and for whatever advice you have the time and energy to provide.
djh

Hi Eddy -
Thanks very much for the advice; it will probably work out that way anyway because right now I'm really interested in doing a good enough initial "trial-run" that Valery/Rich will be willing to get involved in a mini-collaboration that I'm right now working to start up with an opening post in "Test/Playground".
Anyway, since you're taking some time off, I beg you to take a look at the first two paragraphs of this link:
<a href="http://www.kcoyle.net/granfalloons.html">ReCatsCradle</a>.
and also then read Kurt Vonnegut's short novel "Cat's Cradle" if you haven't already.
It is amazing how Vonnegut's notions of a "granfaloon" and a "karass" are related to various events that transpire here at SDN.
Take care of yourself, buddy.
Best regards
djh

Does anyone know of a Podcasts app substitute?

Does anyone know of an approriate app that I can use in place of Apple's Podcast app that WILL NOT require access Wifi or Celllar Data?
When podcasts used to be part of the Music app, I LOVED listening to podcasts. I actually became information junky as the sacrifice of all the music on my apple devices. If I’m at home or away, I loved that I could seamlessly sync my podcats between my laptop and my apple devices. If a podcast was deleted from one, it was removed from the rest post sync. Also, I never used to sync all podcasts, only the ones that I had yet to listen. If one of my podcast subscriptions had nothing to listen to, then that podcast’s album cover did not appear in my podcast list. Why would they appear if there is nothing to listen to? I could plop my iPhone into my car’s doc station and continue to comfortably listen to podcasts for drives of any length without EVER running into any issues.
NOW, even if there is nothing to listen to, I still see the podcast’s album cover, which I HATE. This fact makes it difficult to listen to podcasts while driving, especially in the evening. I have to stare closely at my phone to see which album cover is accompanied by a number that implies that there are podcasts that have yet to be listened to.
Also, I despise seeing the message, “Cellular Data is Turned Off for ‘Podcasts’ ”. Why do need to be constantly reminded of what I already know? I turned it off on purpose! Imagine how annoying this becomes when I am in the middle of driving on the freeway and this message pop’s up, which effectively stops what I am currently listening to until I click a button to get rid of the message. Safety first? Hmmm...I guess not!
Podcasts in the cloud? If I already have the podcasts downloaded in iTunes, all that I want for this Podcast app is to respect my previously downloaded apps an only display what I wanted sync’d from iTunes. I don’t like seeing podcasts “in the cloud”. This usually means that I am either playing something that I already deleted, unchecked or set as “played” with iTunes or I am streaming something that is already downloaded to iTunes. I DO NOT want to waste my Wifi service streaming something that is already downloaded onto (or deleted from) iTunes.
NEXT...Podcast Settings: Why am I able to go to settings to change some settings, such as “Sync Subscriptions”, “Refresh Podcasts”, “Auto-Downloads”, “Episodes to Keep”, and “Use Cellular Data”, but I don’t have the ability to change “Sort Order”, and “Play Order”? Most of my podcasts require that I listen to the oldest podcasts first, such as news articles or serial podcasts. This, having go to each podcast individually to make those changes...what a WASTE of time! Even worse is when there is an update to the app and I have to go into the each podcast again to change things to how I want them to be...yet again!
Sorry, this is long, but I am VERY passtionate about this. I LOVE listening to podcasts, but despise this Apple's Podcast app! ARGH! The only reason why I started using the Podcasts app is because IOS 7 removed this functionality from the Music app of my iPad Mini & iPhone. If the battery of my iPod Touch 4 wasn’t crap I would start using that again because it still has iOS 6.1.3, meaning the Music app still plays my beloved Podcasts. Sign...

I'm sorry to hear that.
I'm not affiliated w/ the developer, just a happy user that gave up fighting the apple podcast app a while ago. I used to have a bunch of smart playlists in itunes for my podcasts, and come home every day and pathologically synced my phone as soon as I walked in the door, and again before I walked out the door in the morning.
Since my wife was doing this too, we were fighting over who's turn it was to sync their phone.
Since I've switched to Downcast, I no longer worry about syncing my phone to itunes at all. I can go weeks between syncs.
Setup a "playlist" in downcast (ex., "Commute") and add podcasts to that playlist. Add another playlist ("walk" or "workout") and add different podcasts to that one.
Set podcast priorities on a per-feed basis (ex., high priority for some daily news feeds, medium priority for some favorite podcasts, lower priority for other stuff). Downcast will play the things in the priority you specify, and within that priority, it will play in date order (oldest to newest).
Allegedly, it will also sync your play status to other devices, although that is not a feature I currently use and can't vouch for. It uses apple's iCloud APIs, so to some extent may be limited by what Apple's APIs can do.

Nano frozen on selection screen.

Howdy,
My iPod nano is stuck on the main selection screen. When I attempt to connect it to my computer iTunes will not recognize it, but the iPod will charge and shows the "Connected" screen. I have reset it numerous times using the Menu/Select method, and attempted to put it into Disk Mode to no avail. I do not care about losing the music - I have that all backed up. If anyone could be of assistance I would greatly appreciate it. Any info is helpful.
Thanks!

Hello Keith,
It would be helpful to post your solution, just in case someone else runs into your same situation.
Problem Fixed. I am Stupid.
NOT!!! Excitement has a way of causing euphoria, where the mind sometimes overlook the obvious.
Gary

2.6.18-emission1: Pathological Euphoria

Similar Messages

Maybe you are looking for