How to diagnose rapid memory growth in Solaris 10 java server application
Hi,
We have a production java based server that experienced large memory growth recently and we are looking for help in diagnosing what might be the cause. The memory footprint of this java server application is limited to 120M using the jvm -Xmx switch. The server process size held consistently at around 180M until an event happened that caused the process size to rapidly increase to aprox 3G. We used the java jstat command to check the application heap usage within the jvm was within the specified limits.
This application can be performs java Runtime exec's to run simple shell scripts from a variety of java threads in the server app.
We have attempted to search for a cause by analysing the output of the pmap, pfiles, and pstack commands but have been unsuccessfull at this. We have also attempted to run the libumem.so library with the application in the hopes of using mdb to find out where the heap usage is. Unfortunately when we followed the instruction on using limumem the application stopped working due an ELF error.
The following are what we used for libumem:
LD_PRELOAD=/usr/lib/libumem.so
UMEM_DEBUG=default
UMEM_LOGGING=transaction
export LD_PRELOAD UMEM_DEBUG UMEM_LOGGING
<run the application>
The error we got is:
ld.so.1: nohup: fatal: /usr/lib/libumem.so: wrong ELF class: ELFCLASS32
We are looking for advice on how to proceed in finding the cause for this growth. In addition to the diag info below we have captured the core file for the application from gcore. The following is the output from pfiles, pmap, pstack, and the showrev command.
Any help/suggestions appreciated.
Steve
java version information:
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
pmap:
24584: /export/home/sonusComm/jre/bin/java -Dname=agent -Xmx120M -DAGENT_HOME
Address Kbytes RSS Anon Locked Mode Mapped File
00010000 64 8 - - r-x-- java
0002E000 16 8 8 - rwx-- java
00032000 3896 2648 1400 - rwx-- [ heap ]
00400000 3579904 717400 591704 - rwx-- [ heap ]
E50F6000 32 32 24 - rw--R [ anon ]
E50FE000 8 8 8 - rw--R [ stack tid=189562 ]
E52F6000 32 32 16 - rw--R [ anon ]
E52FE000 8 8 8 - rw--R [ stack tid=180717 ]
E5CF6000 32 32 16 - rw--R [ anon ]
E5CFE000 8 8 8 - rw--R [ stack tid=170864 ]
E6DF6000 40 40 40 - rw--R [ stack tid=220763 ]
E6EF6000 40 40 40 - rw--R [ anon ]
E6FF6000 8 8 - - rw--R [ anon ]
E6FF8000 32 32 32 - rw--R [ stack tid=220225 ]
E71F6000 40 40 40 - rw--R [ anon ]
E72F6000 40 40 40 - rw--R [ anon ]
E73F8000 24 24 - - rw--R [ anon ]
E73FE000 8 8 8 - rw--R [ stack tid=193659 ]
E74F6000 40 40 40 - rw--R [ stack tid=220762 ]
E75F6000 40 40 40 - rw--R [ stack tid=215672 ]
E76F6000 8 8 - - rw--R [ anon ]
E76F8000 32 32 32 - rw--R [ stack tid=211403 ]
E77F8000 8 8 - - rw--R [ anon ]
E77FA000 24 24 24 - rw--R [ stack tid=206975 ]
E78F6000 40 40 40 - rw--R [ anon ]
E79F8000 32 32 32 - rw--R [ stack tid=220760 ]
E7AF6000 40 40 40 - rw--R [ anon ]
E7CF6000 40 40 40 - rw--R [ stack tid=220764 ]
E7EF6000 32 32 8 - rw--R [ anon ]
E7EFE000 8 8 8 - rw--R [ stack tid=185268 ]
E7FF6000 16 16 16 - rw--R [ anon ]
E7FFC000 16 16 16 - rw--R [ stack tid=140567 ]
E80F6000 8 8 8 - rw--R [ anon ]
E80FE000 8 8 8 - rw--R [ stack tid=145776 ]
E81FC000 16 16 16 - rw--R [ stack tid=124964 ]
E82F6000 40 40 40 - rw--R [ anon ]
E83F8000 32 32 32 - rw--R [ stack tid=1512 ]
E84F6000 8 8 8 - rw--R [ anon ]
E84FA000 8 8 8 - rw--R [ anon ]
E84FE000 8 8 8 - rw--R [ stack tid=119761 ]
E85F6000 32 32 8 - rw--R [ anon ]
E85FE000 8 8 8 - rw--R [ stack tid=202234 ]
E86F6000 16 16 8 - rw--R [ anon ]
E86FA000 24 24 24 - rw--R [ stack tid=197719 ]
E87F8000 32 32 32 - rw--R [ stack tid=1518 ]
E88F8000 8 8 8 - rw--R [ anon ]
E88FC000 16 16 16 - rw--R [ stack tid=173 ]
E89FA000 24 24 24 - rw--R [ stack tid=161359 ]
E8AF8000 32 32 32 - rw--R [ stack tid=34505 ]
E8BFE000 8 8 8 - rw--R [ stack tid=135384 ]
E8CF8000 8 8 8 - rw--R [ anon ]
E8CFC000 16 16 16 - rw--R [ stack tid=172 ]
E8DF6000 8 8 8 - rw--R [ anon ]
E8DFC000 16 16 16 - rw--R [ stack tid=150966 ]
E8EFA000 8 8 8 - rw--R [ anon ]
E8EFE000 8 8 8 - rw--R [ stack tid=130168 ]
E8FFA000 8 8 8 - rw--R [ anon ]
E8FFE000 8 8 8 - rw--R [ stack tid=166557 ]
E90FE000 8 8 8 - rw--R [ stack tid=44270 ]
E91F8000 32 32 32 - rw--R [ stack tid=368 ]
E92FA000 8 8 - - rw--R [ anon ]
E92FE000 8 8 8 - rw--R [ stack tid=156163 ]
E93F8000 32 32 32 - rw--R [ stack tid=82 ]
E94F8000 8 8 8 - rw--R [ anon ]
E94FC000 16 16 16 - rw--R [ stack tid=72 ]
E95F8000 16 16 8 - rw--R [ anon ]
E95FC000 16 16 16 - rw--R [ stack tid=175828 ]
E96FE000 8 8 8 - rw--R [ stack tid=67 ]
E97F8000 32 32 32 - rw--R [ stack tid=62 ]
E98FE000 8 8 8 - rw--R [ stack tid=65 ]
E99FE000 8 8 8 - rw--R [ stack tid=51517 ]
E9AF8000 32 32 32 - rw--R [ stack tid=59 ]
E9BF8000 32 32 32 - rw--R [ stack tid=56 ]
E9CF8000 8 8 8 - rw--R [ anon ]
E9CFC000 16 16 16 - rw--R [ stack tid=57 ]
E9DF8000 16 16 8 - rw--R [ anon ]
E9DFC000 16 16 16 - rw--R [ stack tid=58 ]
E9EF8000 32 32 32 - rw--R [ stack tid=52 ]
E9FF8000 32 32 32 - rw--R [ stack tid=51 ]
EA0F8000 32 32 32 - rw--R [ stack tid=50 ]
EA1F8000 32 32 32 - rw--R [ stack tid=49 ]
EA2F8000 32 32 32 - rw--R [ stack tid=48 ]
EA3F8000 32 32 32 - rw--R [ stack tid=46 ]
EA4F8000 32 32 32 - rw--R [ stack tid=45 ]
EA5F8000 8 8 8 - rw--R [ anon ]
EA5FC000 8 8 - - rw--R [ anon ]
EA5FE000 8 8 8 - rw--R [ stack tid=44 ]
EA6FE000 8 8 8 - rw--R [ stack tid=43 ]
EA7FE000 8 8 8 - rw--R [ stack tid=42 ]
EA8F6000 40 40 40 - rw--R [ stack tid=41 ]
EA9F8000 32 32 32 - rw--R [ stack tid=38 ]
EAAF8000 32 32 32 - rw--R [ stack tid=40 ]
EABFE000 8 8 8 - rw--R [ stack tid=39 ]
EACFE000 8 8 8 - rw--R [ stack tid=31 ]
EADFE000 8 8 8 - rw--R [ stack tid=30 ]
EAEF8000 32 32 32 - rw--R [ stack tid=33 ]
EAFF8000 32 32 32 - rw--R [ stack tid=28 ]
EB0F8000 32 32 32 - rw--R [ stack tid=27 ]
EB1F8000 8 8 8 - rw--R [ anon ]
EB1FE000 8 8 8 - rw--R [ stack tid=26 ]
EB2F8000 8 8 - - rw--R [ anon ]
EB2FA000 24 24 24 - rw--R [ stack tid=25 ]
EB3F8000 8 8 8 - rw--R [ anon ]
EB3FE000 8 8 8 - rw--R [ stack tid=24 ]
EB4F8000 8 8 8 - rw--R [ anon ]
EB4FC000 16 16 16 - rw--R [ stack tid=21 ]
EB580000 816 - - - r--s- dev:32,24 ino:716444
EB680000 760 - - - r--s- dev:32,24 ino:716443
EB780000 1160 - - - r--s- dev:32,24 ino:716441
EB900000 816 - - - r--s- dev:32,24 ino:728313
EBA00000 760 - - - r--s- dev:32,24 ino:728312
EBB00000 1160 - - - r--s- dev:32,24 ino:728310
EBC80000 1120 - - - r--s- dev:32,24 ino:728308
EBE7E000 8 8 8 - rw--R [ stack tid=20 ]
EBF78000 32 32 32 - rw--R [ stack tid=19 ]
EC07E000 8 8 8 - rw--R [ stack tid=18 ]
EC17E000 8 8 8 - rw--R [ stack tid=17 ]
EC27E000 8 8 8 - rw--R [ stack tid=16 ]
EC378000 32 32 32 - rw--R [ stack tid=15 ]
EC400000 6880 - - - r--s- dev:32,24 ino:716451
ECB78000 32 32 32 - rw--R [ stack tid=14 ]
ECC00000 808 - - - r--s- dev:32,24 ino:716464
ECD00000 592 - - - r--s- dev:32,24 ino:716465
ECE78000 16 16 8 - rw--R [ anon ]
ECE7C000 16 16 16 - rw--R [ stack tid=83 ]
ECF00000 784 - - - r--s- dev:32,24 ino:716758
ED07E000 8 8 8 - rw--R [ stack tid=12 ]
ED27C000 16 16 16 - rw--R [ stack tid=10 ]
ED37E000 8 8 8 - rw--R [ stack tid=9 ]
ED3D0000 128 64 64 - rwx-- [ anon ]
ED400000 16384 15424 15232 - rwx-- [ anon ]
F1400000 1408 1408 1408 - rwx-- [ anon ]
F1560000 3264 3264 3136 - rwx-- [ anon ]
F1890000 5568 5568 5504 - rwx-- [ anon ]
F1E00000 7424 768 704 - rwx-- [ anon ]
F6400000 4096 4096 3968 - rwx-- [ anon ]
F6800000 3072 3072 3072 - rwx-- [ anon ]
F6B00000 1408 1408 1408 - rwx-- [ anon ]
F6C60000 448 448 448 - rwx-- [ anon ]
F6CD0000 1344 1344 1344 - rwx-- [ anon ]
F6E20000 512 512 512 - rwx-- [ anon ]
F6EA0000 832 832 832 - rwx-- [ anon ]
F6F70000 128 128 128 - rwx-- [ anon ]
F6F90000 1344 1344 1344 - rwx-- [ anon ]
F70E0000 448 448 448 - rwx-- [ anon ]
F7150000 384 384 384 - rwx-- [ anon ]
F71B0000 128 128 128 - rwx-- [ anon ]
F71D0000 192 192 192 - rwx-- [ anon ]
F7200000 128 128 128 - rwx-- [ anon ]
F7220000 128 128 128 - rwx-- [ anon ]
F7240000 128 128 128 - rwx-- [ anon ]
F7260000 192 192 192 - rwx-- [ anon ]
F7290000 128 128 128 - rwx-- [ anon ]
F72B0000 64 64 64 - rwx-- [ anon ]
F72C0000 384 384 384 - rwx-- [ anon ]
F7320000 448 448 448 - rwx-- [ anon ]
F7390000 896 896 896 - rwx-- [ anon ]
F7470000 768 768 768 - rwx-- [ anon ]
F7530000 704 704 704 - rwx-- [ anon ]
F75E0000 576 576 576 - rwx-- [ anon ]
F7670000 576 576 576 - rwx-- [ anon ]
F7700000 448 448 448 - rwx-- [ anon ]
F7770000 384 384 384 - rwx-- [ anon ]
F77D0000 128 128 128 - rwx-- [ anon ]
F77F0000 192 192 192 - rwx-- [ anon ]
F7820000 320 320 320 - rwx-- [ anon ]
F7870000 128 128 128 - rwx-- [ anon ]
F7890000 128 128 128 - rwx-- [ anon ]
F78B0000 448 448 448 - rwx-- [ anon ]
F7920000 896 896 896 - rwx-- [ anon ]
F7A00000 768 768 768 - rwx-- [ anon ]
F7AC0000 704 704 704 - rwx-- [ anon ]
F7B70000 832 832 640 - rwx-- [ anon ]
F7C40000 704 704 384 - rwx-- [ anon ]
F7CF0000 448 448 384 - rwx-- [ anon ]
F7D60000 704 704 384 - rwx-- [ anon ]
F7E10000 384 384 192 - rwx-- [ anon ]
F7E70000 384 384 192 - rwx-- [ anon ]
F7ED0000 320 320 256 - rwx-- [ anon ]
F7F20000 256 256 192 - rwx-- [ anon ]
F7F60000 256 256 192 - rwx-- [ anon ]
F7FA0000 128 128 128 - rwx-- [ anon ]
F7FC0000 192 192 128 - rwx-- [ anon ]
F7FF0000 192 192 128 - rwx-- [ anon ]
F8020000 256 256 128 - rwx-- [ anon ]
F8060000 128 128 128 - rwx-- [ anon ]
F8080000 128 128 64 - rwx-- [ anon ]
F80A0000 192 192 192 - rwx-- [ anon ]
F80D0000 384 384 192 - rwx-- [ anon ]
F8130000 64 64 64 - rwx-- [ anon ]
F8140000 64 64 - - rwx-- [ anon ]
F8150000 128 128 128 - rwx-- [ anon ]
F8170000 128 128 128 - rwx-- [ anon ]
F8190000 128 128 128 - rwx-- [ anon ]
F81B0000 128 128 64 - rwx-- [ anon ]
F81D0000 1600 1600 768 - rwx-- [ anon ]
F8360000 1152 1152 384 - rwx-- [ anon ]
F8480000 960 472 - - rwx-- [ anon ]
F8570000 832 80 - - rwx-- [ anon ]
F8640000 128 8 - - rwx-- [ anon ]
F8660000 832 16 - - rwx-- [ anon ]
F8730000 576 - - - rwx-- [ anon ]
F87C0000 448 - - - rwx-- [ anon ]
F8830000 256 - - - rwx-- [ anon ]
F8870000 320 - - - rwx-- [ anon ]
F88C0000 256 - - - rwx-- [ anon ]
F8900000 256 - - - rwx-- [ anon ]
F8940000 256 - - - rwx-- [ anon ]
F8980000 192 - - - rwx-- [ anon ]
F89B0000 384 - - - rwx-- [ anon ]
F8A10000 320 - - - rwx-- [ anon ]
F8A60000 192 - - - rwx-- [ anon ]
F8A90000 256 - - - rwx-- [ anon ]
F8AD0000 256 - - - rwx-- [ anon ]
F8B10000 192 128 64 - rwx-- [ anon ]
F8B40000 128 128 128 - rwx-- [ anon ]
F8B60000 64 64 64 - rwx-- [ anon ]
F8B70000 192 192 128 - rwx-- [ anon ]
F8BA0000 128 128 64 - rwx-- [ anon ]
F8BC0000 128 - - - rwx-- [ anon ]
F8C10000 64 - - - rwx-- [ anon ]
F8C30000 8 8 - - r-x-- librmi.so
F8C40000 8 8 8 - rwx-- librmi.so
F8C60000 64 - - - r--s- dev:32,24 ino:728318
F8C80000 376 144 - - r-x-- pkcs11_softtoken.so.1
F8CEE000 16 - - - rwx-- pkcs11_softtoken.so.1
F8D7E000 8 8 8 - rw--R [ stack tid=8 ]
F8D90000 16 - - - r--s- dev:32,24 ino:728317
F8DA0000 232 - - - r--s- dev:32,24 ino:728316
F8DE0000 88 - - - r--s- dev:32,24 ino:716445
F8E90000 16 - - - r--s- dev:32,24 ino:728315
F8EA0000 136 - - - r--s- dev:32,24 ino:728309
F8ED0000 136 - - - r--s- dev:32,24 ino:728307
F8F78000 32 32 32 - rw--R [ stack tid=6 ]
F8F90000 88 - - - r--s- dev:32,24 ino:728314
F8FB0000 8 8 8 - rwx-- [ anon ]
F8FC0000 88 88 - - r-x-- libpkcs11.so.1
F8FE6000 24 16 8 - rwx-- libpkcs11.so.1
F8FEC000 8 - - - rwx-- libpkcs11.so.1
F9000000 4096 4096 4096 - rwx-- [ anon ]
F9400000 4096 4096 4096 - rwx-- [ anon ]
FB010000 24 24 - - r-x-- libcryptoutil.so.1
FB026000 8 8 - - rwx-- libcryptoutil.so.1
FB030000 56 8 - - r-x-- libj2pkcs11.so
FB04C000 8 - - - rwx-- libj2pkcs11.so
FB060000 352 - - - r--s- dev:32,24 ino:716547
FB0C0000 184 - - - r--s- dev:32,24 ino:716545
FB178000 8 8 8 - rw--R [ anon ]
FB17E000 8 8 8 - rw--R [ stack tid=5 ]
FB190000 8 - - - r---- [ anon ]
FB1A0000 160 - - - r--s- dev:32,24 ino:716544
FB1D0000 72 64 - - r-x-- libnet.so
FB1F0000 8 8 8 - rwx-- libnet.so
FB27E000 8 8 8 - rw--R [ stack tid=4 ]
FB290000 24 8 - - r-x-- libmanagement.so
FB2A4000 8 8 - - rwx-- libmanagement.so
FB2B0000 256 - - - r--s- dev:32,24 ino:716466
FB37E000 8 8 8 - rw--R [ stack tid=3 ]
FB390000 32 - - - r--s- dev:32,24 ino:716442
FB3A0000 120 - - - r--s- dev:32,24 ino:716467
FB3D0000 176 - - - r--s- dev:32,24 ino:716455
FB400000 8520 - - - r--s- dev:32,24 ino:717379
FBC60000 32 - - - r--s- dev:32,24 ino:728311
FBC70000 40 - - - r--s- dev:32,24 ino:716546
FBC80000 304 - - - r--s- dev:32,24 ino:716456
FBCD0000 176 - - - r--s- dev:32,24 ino:716759
FBD7E000 8 8 8 - rw--R [ stack tid=2 ]
FBD90000 32 - - - r--s- dev:32,24 ino:716463
FBDA0000 8 - - - r--s- dev:32,24 ino:716757
FBDB0000 160 - - - r--s- dev:32,24 ino:716756
FBDE0000 64 64 64 - rw--- [ anon ]
FBE00000 64 40 40 - rw--- [ anon ]
FBE20000 32 32 32 - rwx-- [ anon ]
FBE50000 8 8 8 - rwx-- [ anon ]
FBE52000 8 8 8 - rwx-- [ anon ]
FBE54000 8 8 8 - rwx-- [ anon ]
FBE56000 16 16 16 - rwx-- [ anon ]
FBE80000 24 24 24 - rwx-- [ anon ]
FBE86000 8 8 - - rwx-- [ anon ]
FBE88000 8 8 8 - rwx-- [ anon ]
FBE8A000 8 8 8 - rwx-- [ anon ]
FBE8C000 8 8 8 - rwx-- [ anon ]
FBE8E000 8 8 8 - rwx-- [ anon ]
FBE90000 8 8 8 - rwx-- [ anon ]
FBE92000 8 8 8 - rwx-- [ anon ]
FBE94000 8 8 8 - rwx-- [ anon ]
FBE96000 8 8 8 - rwx-- [ anon ]
FBE98000 8 8 8 - rwx-- [ anon ]
FBE9A000 8 8 8 - rwx-- [ anon ]
FBE9C000 8 8 - - rwx-- [ anon ]
FBF10000 32 32 32 - rwx-- [ anon ]
FBF30000 8 8 8 - rwx-- [ anon ]
FBF32000 8 8 8 - rwx-- [ anon ]
FBF34000 8 8 8 - rwx-- [ anon ]
FBF36000 16 16 16 - rwx-- [ anon ]
FBF58000 8 8 8 - rwx-- [ anon ]
FBF5A000 8 8 8 - rwx-- [ anon ]
FBF5C000 8 8 8 - rwx-- [ anon ]
FBF5E000 8 8 8 - rwx-- [ anon ]
FBF60000 8 8 8 - rwx-- [ anon ]
FBF62000 8 8 8 - rwx-- [ anon ]
FBF64000 8 8 8 - rwx-- [ anon ]
FBF66000 8 8 8 - rwx-- [ anon ]
FBF68000 8 8 - - rwx-- [ anon ]
FBF6A000 8 8 8 - rwx-- [ anon ]
FBF6C000 8 - - - rwx-- [ anon ]
FBF80000 480 - - - r--s- dev:32,24 ino:716808
FC000000 35016 - - - r--s- dev:32,24 ino:716764
FE240000 64 64 64 - rwx-- [ anon ]
FE260000 88 - - - r--s- dev:32,24 ino:716807
FE280000 1184 - - - r--s- dev:32,24 ino:716460
FE3B0000 8 - - - r--s- dev:32,24 ino:716468
FE3C0000 200 - - - r--s- dev:32,24 ino:716461
FE400000 1744 - - - r--s- dev:32,24 ino:716459
FE5C0000 64 - - - r--s- dev:32,24 ino:716458
FE5E0000 8 8 8 - rwx-- [ anon ]
FE5F0000 64 24 - - r-x-- libzip.so
FE600000 8 8 8 - rwx-- libzip.so
FE610000 144 96 - - r-x-- libjava.so
FE644000 8 8 8 - rwx-- libjava.so
FE650000 56 16 - - r-x-- libverify.so
FE66E000 8 8 8 - rwx-- libverify.so
FE680000 40 - - - r--s- dev:32,24 ino:716397
FE692000 8 8 - - rwxs- [ anon ]
FE6A0000 32 16 - - rw-s- dev:295,2 ino:5858787
FE6B0000 32 8 - - r-x-- libhpi.so
FE6C8000 8 8 8 - rwx-- libhpi.so
FE6CA000 8 - - - rwx-- libhpi.so
FE6E0000 16 16 - - r-x-- libmp.so.2
FE6F4000 8 8 - - rwx-- libmp.so.2
FE700000 680 160 - - r-x-- libm.so.2
FE7B8000 32 32 - - rwx-- libm.so.2
FE7D0000 8 8 8 - rwx-- [ anon ]
FE7E0000 8 8 - - r-x-- libmd5_psr.so.1
FE7F2000 8 8 - - rwx-- libmd5_psr.so.1
FE800000 7984 4696 - - r-x-- libjvm.so
FEFDC000 408 392 320 - rwx-- libjvm.so
FF042000 56 48 48 - rwx-- libjvm.so
FF060000 8 8 8 - rwx-- [ anon ]
FF070000 8 8 - - r-x-- libmd5.so.1
FF082000 8 8 - - rwx-- libmd5.so.1
FF090000 32 24 - - r-x-- libuutil.so.1
FF0A8000 8 8 - - rwx-- libuutil.so.1
FF0B0000 8 8 - - r-x-- libdoor.so.1
FF0C2000 8 8 - - rwx-- libdoor.so.1
FF0D0000 96 88 - - r-x-- libscf.so.1
FF0F8000 8 8 - - rwx-- libscf.so.1
FF100000 584 584 - - r-x-- libnsl.so.1
FF1A2000 40 40 8 - rwx-- libnsl.so.1
FF1AC000 24 - - - rwx-- libnsl.so.1
FF1C0000 8 8 8 - rwx-- [ anon ]
FF1D0000 16 16 - - r-x-- libm.so.1
FF1E2000 8 8 - - rwx-- libm.so.1
FF1F0000 48 48 - - r-x-- libCrun.so.1
FF20A000 8 8 8 - rwx-- libCrun.so.1
FF20C000 16 - - - rwx-- libCrun.so.1
FF220000 8 8 - - r---- [ anon ]
FF230000 48 48 - - r-x-- libsocket.so.1
FF24C000 8 8 8 - rwx-- libsocket.so.1
FF260000 8 8 - - r-x-- libsched.so.1
FF270000 8 8 - - r-x-- libc_psr.so.1
FF280000 864 864 - - r-x-- libc.so.1
FF368000 32 32 24 - rwx-- libc.so.1
FF370000 8 8 8 - rwx-- libc.so.1
FF380000 8 8 8 - rwx-- [ anon ]
FF390000 24 24 24 - rwx-- [ anon ]
FF3A0000 8 8 - - r-x-- libdl.so.1
FF3B0000 184 184 - - r-x-- ld.so.1
FF3EE000 8 8 8 - rwx-- ld.so.1
FF3F0000 8 8 8 - rwx-- ld.so.1
FF3F8000 16 16 - - r-x-- libthread.so.1
FFB80000 24 - - - ----- [ anon ]
FFBF4000 48 8 8 - rw--- [ stack ]
pfiles:
24584: /export/home/sonusComm/jre/bin/java -Dname=agent -Xmx120M -DAGENT_HOME
Current rlimit: 65536 file descriptors
0: S_IFCHR mode:0666 dev:291,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDONLY|O_LARGEFILE
/devices/pseudo/mm@0:null
1: S_IFREG mode:0644 dev:32,24 ino:716359 uid:50004 gid:6003 size:36526979
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE
/export/home/sonusComm/logs/console.log
2: S_IFREG mode:0644 dev:32,24 ino:716359 uid:50004 gid:6003 size:36526979
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE
/export/home/sonusComm/logs/console.log
3: S_IFCHR mode:0666 dev:291,0 ino:6815772 uid:0 gid:3 rdev:13,12
O_RDWR FD_CLOEXEC
/devices/pseudo/mm@0:zero
4: S_IFDOOR mode:0444 dev:300,0 ino:58 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[104]
/var/run/name_service_door
5: S_IFCHR mode:0644 dev:291,0 ino:99614724 uid:0 gid:3 rdev:190,0
O_RDONLY|O_LARGEFILE
/devices/pseudo/random@0:random
6: S_IFCHR mode:0644 dev:291,0 ino:99614726 uid:0 gid:3 rdev:190,1
O_RDONLY|O_LARGEFILE
/devices/pseudo/random@0:urandom
7: S_IFREG mode:0644 dev:32,24 ino:716361 uid:50004 gid:6003 size:323022
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE
/export/home/sonusComm/server/communicator/log/server.log
8: S_IFIFO mode:0000 dev:299,0 ino:47868548 uid:50004 gid:6003 size:0
O_RDWR
9: S_IFSOCK mode:0666 dev:297,0 ino:4151 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 6098
10: S_IFSOCK mode:0666 dev:297,0 ino:4155 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 42405
12: S_IFSOCK mode:0666 dev:297,0 ino:19198 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 6099
14: S_IFIFO mode:0000 dev:299,0 ino:47865419 uid:50004 gid:6003 size:0
O_RDWR
15: S_IFCHR mode:0644 dev:291,0 ino:99614724 uid:0 gid:3 rdev:190,0
O_RDONLY|O_LARGEFILE
/devices/pseudo/random@0:random
16: S_IFSOCK mode:0666 dev:297,0 ino:48720 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 127.0.0.1 port: 2371
17: S_IFIFO mode:0000 dev:299,0 ino:47865419 uid:50004 gid:6003 size:0
O_RDWR
18: S_IFSOCK mode:0666 dev:297,0 ino:37454 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 9991
19: S_IFREG mode:0644 dev:32,24 ino:717406 uid:50004 gid:6003 size:23436
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/export/home/sonusComm/logs/application.monitor.audit
20: S_IFSOCK mode:0666 dev:297,0 ino:3896 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
SO_BROADCAST,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.0.224.0)
sockname: AF_INET 0.0.0.0 port: 161
21: S_IFSOCK mode:0666 dev:297,0 ino:3897 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 7788
22: S_IFSOCK mode:0666 dev:297,0 ino:22116 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
SO_BROADCAST,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.0.224.0)
sockname: AF_INET 0.0.0.0 port: 9993
23: S_IFSOCK mode:0666 dev:297,0 ino:64893 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
SO_BROADCAST,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.0.224.0)
sockname: AF_INET 0.0.0.0 port: 5200
24: S_IFSOCK mode:0666 dev:297,0 ino:26901 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 2020
26: S_IFREG mode:0644 dev:32,24 ino:716364 uid:50004 gid:6003 size:658924
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/export/home/sonusComm/logs/EventMgr.audit
27: S_IFSOCK mode:0666 dev:297,0 ino:4149 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 127.0.0.1 port: 7788
peername: AF_INET 127.0.0.1 port: 42407
29: S_IFREG mode:0644 dev:32,24 ino:716365 uid:50004 gid:6003 size:49140
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/export/home/sonusComm/logs/EventMgrLog.audit
30: S_IFSOCK mode:0666 dev:297,0 ino:4150 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 19991
31: S_IFSOCK mode:0666 dev:297,0 ino:3714 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
I was finally able to reproduce this problem where I observed that within a 5 minute time span the jvm process grew from around 180M (VS) / 100M (RSS) to 2G (VS) / 1G (RSS). Here are some samplings at the end of my dtrace run showing the largest allocations that were made. We suspect this might be a JVM bug (since jstat reports our heap is well below the 120M limit we specified). Do you have any suggestions on how we might resolve this issue ?
Thanks,
Steve
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x31c
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
2352344
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0x90
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x5d0
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
2807672
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0xd4
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cKNode_ArrayEgrow6MI_v_+0x7c
libjvm.so`__1cMPhaseChaitinbApost_allocate_copy_removal6M_v_+0x544
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1488
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
2817016
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0xd4
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cKNode_ArrayEgrow6MI_v_+0x7c
libjvm.so`__1cMPhaseChaitinbApost_allocate_copy_removal6M_v_+0x518
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1488
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3013552
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2a8
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x100c
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3341112
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x5d0
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3402976
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0x90
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x8b8
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3952576
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2a8
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x720
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
4782376
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x8b8
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
4881328
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cHMatcherFxform6MpnENode_i_2_+0xac
libjvm.so`__1cHMatcherFmatch6M_v_+0x644
libjvm.so`__1cHCompileICode_Gen6M_v_+0xd4
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5090648
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x200
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5252816
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0x90
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1180
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5820336
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5822560
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2c0
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x720
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
6747736
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1180
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
7129056
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x47c
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
16349616
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0x244
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cENodeIout_grow6MI_v_+0x150
libjvm.so`__1cENodeFclone6kM_p0_+0x1f0
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0x8c8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
134217736
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0x244
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cKNode_ArrayEgrow6MI_v_+0x7c
libjvm.so`__1cMPhaseIterGVNWadd_users_to_worklist06MpnENode__v_+0xa4
libjvm.so`__1cMPhaseIterGVNVadd_users_to_worklist6MpnENode__v_+0x8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x3c
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
134217736
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeIout_grow6MI_v_+0xac
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0xacc
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
165122996
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeFclone6kM_p0_+0xb0
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0x8c8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
165122996
libumem.so.1`malloc
libjava.so`JNU_GetStringPlatformChars+0xd8
libnet.so`Java_java_net_Inet4AddressImpl_lookupAllHostAddr+0x54
0xf8c0c280
0xf8c0c224
0xf8c05d3c
0xf8c05d3c
0xf8c05874
0xf8c05874
0xf8c05764
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
-4294966636
Similar Messages
-
Hi,
We have a production java based server that experienced large memory growth recently and we are looking for help in diagnosing what might be the cause. The memory footprint of this java server application is limited to 120M using the jvm -Xmx switch. The server process size held consistently at around 180M until an event happened that caused the process size to increase rapidly to aprox 3G within a 4 minute period. We used the java jstat command to check the application heap usage within the jvm and it was within the specified limits.
We have collected a variety of diagnostic information (pmap, pstack, pfiles, etc - available on request) as well as output from dtrace using a script that is used to count the number of memory allocations. The dtrace script and tail end of this message.
Any help in interpreting the dtrace output in the hopes of pinpointing a cause for the large memory allocations would be greatly appreciated.
Steve
java version information:
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
dtrace script:
#!/usr/sbin/dtrace -d -F
pid$target::malloc:entry
@totalbytes[ ustack() ] = sum( arg0 );
pid$target::valloc:entry
@totalbytes[ ustack() ] = sum( arg0 );
pldd information:
8348: /export/home/sonusComm/jre/bin/java -Dname=agent -Xmx120M -DAGENT_HOME
/lib/libumem.so.1
/lib/libthread.so.1
/lib/libdl.so.1
/lib/libc.so.1
/platform/sun4u-us3/lib/libc_psr.so.1
/export/home/sonusComm/jre/lib/sparc/server/libjvm.so
/lib/libsocket.so.1
/usr/lib/libsched.so.1
/usr/lib/libCrun.so.1
/lib/libm.so.1
/lib/libnsl.so.1
/lib/libm.so.2
/lib/libscf.so.1
/lib/libdoor.so.1
/lib/libuutil.so.1
/lib/libmd5.so.1
/platform/sun4u/lib/libmd5_psr.so.1
/lib/libmp.so.2
/export/home/sonusComm/jre/lib/sparc/native_threads/libhpi.so
/export/home/sonusComm/jre/lib/sparc/libverify.so
/export/home/sonusComm/jre/lib/sparc/libjava.so
/export/home/sonusComm/jre/lib/sparc/libzip.so
/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
/export/home/sonusComm/jre/lib/sparc/libnet.so
/export/home/sonusComm/jre/lib/sparc/libmanagement.so
/export/home/sonusComm/jre/lib/sparc/libj2pkcs11.so
/usr/lib/libpkcs11.so.1
/usr/lib/libcryptoutil.so.1
/usr/lib/security/pkcs11_softtoken.so.1
/export/home/sonusComm/jre/lib/sparc/librmi.so
dtrace output snippet (the numeric value at the end of each stack is the number of bytes allocated)
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2a8
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x720
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
9269948
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cHMatcherFxform6MpnENode_i_2_+0xac
libjvm.so`__1cHMatcherFmatch6M_v_+0x644
libjvm.so`__1cHCompileICode_Gen6M_v_+0xd4
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10008720
libumem.so.1`malloc
libc.so.1`_real_gettext_u+0x98
libc.so.1`dgettext+0x98
libhpi.so`0xff0623d0
libjava.so`JNU_ThrowIOExceptionWithLastError+0x28
libjava.so`Java_java_lang_UNIXProcess_forkAndExec+0x714
0xf8f4f250
0xf90dd264
0xf90e6cdc
0xf9101be0
0xf8ed8078
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10313472
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1180
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10371104
libumem.so.1`malloc
libnet.so`Java_java_net_SocketInputStream_socketRead0+0xcc
0xf8c668ec
0xf8f01088
0xf8c76344
0xf8c058b8
0xf8c05764
0xf8c595b4
0xf8e7e594
0xf8c96238
0xf8c05c2c
0xf8c05d3c
0xf8c05d3c
0xf8c05d3c
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
10469376
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cICHeapObj2n6FI_pv_+0x1c
libjvm.so`__1cKJavaThreadKinitialize6M_v_+0xc0
libjvm.so`__1cKJavaThread2t5B6MpFp0pnGThread__vI_v_+0x60
libjvm.so`JVM_StartThread+0x1c0
0xf8c0c280
0xf8c0c224
0xf8f40c7c
0xf9065edc
0xf913a744
0xf9132638
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10552560
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cICHeapObj2n6FI_pv_+0x1c
libjvm.so`__1cFMutex2t5B6Mipkci_v_+0x20
libjvm.so`__1cHMonitor2t5B6Mipkci_v_+0x10
libjvm.so`__1cGThread2t5B6M_v_+0x118
libjvm.so`__1cKJavaThread2t5B6MpFp0pnGThread__vI_v_+0x34
libjvm.so`JVM_StartThread+0x1c0
0xf8c0c280
0xf8c0c224
0xf8f40c7c
0xf9065edc
0xf913a744
0xf9132638
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
11607816
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
12653032
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cJOopMapSetGall_do6FpknFframe_pnICodeBlob_pknLRegisterMap_pnKOopClosure_pFppnHoopDesc_9E_v9B9B_v_+0x5a0
libjvm.so`__1cJOopMapSetHoops_do6FpknFframe_pnICodeBlob_pknLRegisterMap_pnKOopClosure__v_+0x44
libjvm.so`__1cFframeRoops_code_blob_do6MpnKOopClosure_pknLRegisterMap__v_+0x28
libjvm.so`__1cKJavaThreadHoops_do6MpnKOopClosure__v_+0x19c
libjvm.so`__1cPThreadRootsTaskFdo_it6MpnNGCTaskManager_I_v_+0x50
libjvm.so`__1cMGCTaskThreadDrun6M_v_+0x1e0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
18146824
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x47c
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
20415552
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`jni_GetByteArrayElements+0x144
libjava.so`Java_java_lang_UNIXProcess_forkAndExec+0x58
0xf8f4f250
0xf90dd264
0xf90e6cdc
0xf9101be0
0xf8ed8078
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
38030987
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cICHeapObj2n6FI_pv_+0x1c
libjvm.so`JVM_StartThread+0x1a0
0xf8c0c280
0xf8c0c224
0xf8f40c7c
0xf9065edc
0xf913a744
0xf9132638
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
59094336
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cHThreadsZcreate_thread_roots_tasks6FpnLGCTaskQdDueue__v_+0x4c
libjvm.so`__1cKPSScavengeQinvoke_no_policy6Fpi_i_+0x7d0
libjvm.so`__1cKPSScavengeGinvoke6Fpi_v_+0x5c
libjvm.so`__1cUParallelScavengeHeapTfailed_mem_allocate6MpiIii_pnIHeapWord__+0xbc
libjvm.so`__1cbDVM_ParallelGCFailedAllocationEdoit6M_v_+0xa4
libjvm.so`__1cMVM_OperationIevaluate6M_v_+0x80
libjvm.so`__1cIVMThreadDrun6M_v_+0x6e0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
60336552
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeIout_grow6MI_v_+0xac
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0xacc
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
100790212
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeFclone6kM_p0_+0xb0
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0x8c8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
101445332
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0x244
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cMPhaseIterGVNJtransform6MpnENode__2_+0x98
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0xa3c
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
268435464
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cINodeHashEgrow6M_v_+0x74
libjvm.so`__1cINodeHashQhash_find_insert6MpnENode__2_+0x1ac
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x2f8
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
536936472
libumem.so.1`malloc
libjava.so`JNU_GetStringPlatformChars+0xd8
libjava.so`Java_java_io_UnixFileSystem_getBooleanAttributes0+0x80
0xf8c0c280
0xf8c0c224
0xf8c058b8
0xf8c36cf8
0xf8f83678
0xf8fd515c
0xf8c96238
0xf8c05764
0xf8c05764
0xf8c05764
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
-8588966992This isn't a heap space issue with the java application though, which is what these tools you mentioned would be used to diagnose. The heap space usage of our java app is remaining constant and below the specified limit of 120M but the process continues to grow as we speak.
I can only conclude that this is most likely a jvm memory leak. I should have probably made this clearer in the subject heading.
Anyone know how to file a bug against the solaris jvm ? -
How do I get rid of the alert "(Java Scriprt Application)Type error: Text is undefined> I get it when I get a new Internet page using Firefow
I do not want to half to install things into my PC that i do not trust! I just want this gone and like i said Firefox should of given us the choice to add this and not forced us, because if there is no actual way to remove it then il be forced to remove Firefox till they either remove it or allow us to remove it!
-
How can I measure memory allocation of a Java Application?
We built a web application that enables images to be loaded into a database through a servlet running on Tomcat. We have inconsistent behaivor when more than x number of people are loading images simultaneously. No doubt a memory allocation problem, but how can I measure it?
For example, I'd like to be able to create a graph showing the resources used against the number of people making uploads. I am using Linux and Tomcat/Apache.
I am hoping for something like Memory counters that are available for measurine Windows applications on MS Windows - comes with the Resource Kit. Does this type of tool exist for the Java/Linux market?It is difficult to gauge memory usage from a Java application as the JVM handles garbage collection and even if objects are not referenced they might still be sitting in memory until the next garbage collect.
Anyhow, a way to query available memory is:
Runtime.getRuntime().freeMemory()
You can run this at the begining of a process and at the end and examine the difference. You should call System.gc() before each freeMemory() call in order to try to force garbage collection.
Another method would be to turn to the underlying OS. On linux you could use 'top' or 'ps'. With ps you could do something like:
ps -o rss,vsz,%mem <pid>
Where <pid> is the parent Tomcat process' id. The resulting three columns are:
- Amount of real memory used.
- Amount of virtual memory used.
- Percentage of real memory used.
'man ps' wil give you more options on how to use ps.
There are also fancy graphical stuff like 'gtop' as well.
Hope this helps. -
How can I accomplish this function that control a server application on jav
Now I write a server application,and want write a control shell to operate it,like start,stop,restart and so on.
just like the apache server shell and Tomcat 5.0 server shell,but mine shell just add the system configure panel.
I need your help,Thank u very much!just make your server to listen to some specific
command either in the same port or different port.I don't know what u say,can u give me an example?or very specific introduce.
how to listen to some specific command in the same port or different port. -
How to check virtual Memory Info on Solaris 10
Hello,
Can anybody please tell me command used to calculate or check virtual memory configured on my solaris 10 T2000 servers.
From Solaris Management Console I did get this information but I was wondering which command it used to extract this information.
Thanks in AdvanceHere is vmstat 2 2 s output on my machine
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 18874528 3257488 19 47 32 3 3 0 1 2 3 0 0 964 16476 1115 1 1 98
0 0 0 21063104 5781064 36 135 28 0 0 0 0 0 0 0 0 1509 5049 1946 1 1 99
It says system has 21063104KB of Swap but my solaris management console shows
Physical Memory: 8184 MB
Virtual Memory : 21992 MB
Virtual Memory used : 7%(1537/21992)20455 MB Available
I want to know how Solaris Management Console calculated my Virtual Memory.
Thanks -
How can I modify memory limitation to a Java thread?
While I using
java -Xms64M xxxx (class name)
it seems that 64M is the biggest memory capacity Jvm allows.
How if i wish to use more than 64M ?
Ps: I am using Windows XP
Thanks-Xms is the starting size. Try -Xmx to set the maximum size.
-
How to change heap memory size for general java applications (not applets)
Hi. I made a java class which manipulates images and I sometimes get an out of memory error when the files are large. In these cases I can run it successfully from the command line using:
java -Xms32m -Xmx128m myappbut as I run this class from a firefox extension, I can't use this technique.
Could some one tell me how to set this globally? I've found a lot of info on setting it for applets in the control panel but that's it. I also tried editing my deployment.properties file by adding:
deployment.javapi.jre.1.6.0_11-b03.args=-Xmx128mbut none of these options seem to work.
Thanks for any help.Also you can use use [JAVA_TOOL_OPTIONS|http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html#tooloptions], with more documentation here. JAVA_TOOLS_OPTIONS is available since 1.5. There is no direct documentation on [_JAVA_OPTIONS|http://java.sun.com/j2se/1.5.0/docs/guide/2d/flags.html] but was made available since at least 1.3.
The recent existing forum thread [Java Programming - How to change the default memory limits for java.|http://forums.sun.com/thread.jspa?threadID=5368323&messageID=10615245&start=36] has a long discussion on the issue.
You specified you are not using applets, but if you do, I would also suggest you use the next Generation Plug-in (since 1.6.0_10) that allows you specify the max memory in the applet without having to instruct the user how to make the change. See [JAVA_ARGUMENTS|https://jdk6.dev.java.net/plugin2/#JAVA_ARGUMENTS] for applets. Java Webstart uses resources. -
How can I run a servlet with Sun Java System Application Server PE 8?
I've created a package with a TestServlet.class inside, used the deploytool to create a WAR and deployed this using the autodeploy folder.
The filestructure has been generated and I find the TestServlet.class in
[installdir]\domains\domain1\applications\j2ee-modules\testProject\WEB-INF\classes\[packagefolderstructure]\TestServlet.class.
The context root is working fine, but I have no clue how I can run the servlet directly via the URL.
I've tried many things like
http://localhost:8080/testProject/TestServlet
http://localhost:8080/testProject/servlet/TestServlet
http://localhost:8080/testProject/servlet/[packagenamewithpoints]TestServlet
http://localhost:8080/testProject/servlet/[packagenamewithslashes]TestServlet
etc etc
Can somebody tell me please how to write URLs to deployed servlets? Or send me an example url and xml descriptor files?
Thanks a lot in advance!in web.xml use servlet mapping
<servlet-mapping>
<servlet-name>TestServlet</servlet-name>
<url-pattern>/doit</url-pattern>
</servlet-mapping>then use http://localhost:8080/testProject/doit -
How to make jar files at runtime in a server application
I am using Jung API and i need to load the class files in this JAR file for my application at runtime.
I am using an applet which uses this API which is loaded with the help of a Servlet.
The code is getting compiled but the JAR files and associated classes are not available at the runtime
How to make themk avalable.
I have tried various options like placing jar files in
WEB-INF/lib
$Catlina_Home/commons/lib
But it is not working can anyone help me it is of high urgency to me
How to make that classes in the JAR file available to my applet at runtimeSuperJavaHello wrote:
How to make that classes in the JAR file available to my applet at runtimeAhh! The runtime is an appler! The applet is on the Client, and the Client does not have access to the Server's classpath. You need to un-pack the JAR, then re-pack the classes into the JAR used to pack your Applet, the one that you are using as the code source in the applet tag. -
How to consume SOAP webservice in to my java/j2ee application
Hi,
I am a java developer. how to i consume SAP soap webservice in to my java application.
scenario : 1
We are migrating 2 specific java modules in to SAP System.But master details are stored in to my java application database.Those details are require
to SAP application. So that they provide me WSDL file, How to i consume soap webservice in to my java application and also, How to i provide master data details to SAP system through SOAP webservice.
Any body provide me a detail instruction how to integrate SAP Application in to my java application using SOAP webservice. or
In this scenario i will go for any one open source ESB to consume the soap webservice in to my java application, and provide master data to
SAP application, can any one guide to me.Hi,
What IDE are you using?, for example with Eclipse i used WSDL2JAVA plugin in this way i converted the WSDL in java classes easily. Depending your development environment you can use different tools that makes easier the integration using the WSDL.
Regards. -
How to load a text file in a Java swings application
hi,
I am trying to retrieve a Java code(or say any text from an existing file) and display it in a swing application.
I am not able to do so.
Please help me.public String readFile(File f){
try {
BufferedReader read = new BufferedReader(new FileReader(f));
String line = read.readLine();
String total = "";
while(line != null){
// System.out.println(line);
total += line;
line = read.readLine();
read.close();
return total;
} catch (Exception e){
e.printStackTrace();
return "";
}that will read a file into a string, i hope you can figure out the rest -
How do I set a breakpoint in a Java Server page (.jsp)?
I have set up an application for remote debugging and am able to connect to it from JDeveloper. When the code execution comes to breakpoints that I have set in java (.java) files they work and the application stops so I can debug etc.
This also applies to breakpoints set in java code that is called from Java Server pages.
However if I set breakpoints directly in a java server page whether it is in html code or in java code (such as scriplets) the execution does not stop and the breakpoints are ignored.
Why? Is there some extra thing I have to do? I have tried in Project Properties/Run/Debug/Profile/Edit/Launch Settings to choose server as Virtual Machine for instance but that does not change anything.
Comment. I do not know if this is of interest but I am going to add some java server pages to an application and they are defined in a directory like this:
. . . src\org\yawlfoundation\yawl\resourcing\jsf\jsp
but when the tomcat deploys the .war file they end up here:
. . . apache-tomcat-6.0.18\webapps\resourceService
As I can find no package declaration in a .jsp file this should not matter but I mention it anyway.
Edited by: user10477024 on Nov 25, 2008 5:32 AMHi user10477024,
thanks for explaining your background on what you are trying to do and I do understand the approach you are taking. Unlike user10689900's completely random post.
I've tried to convey why there is no support for JSP debugging in JDev, but again the main reason is that the current frameworks do not code Java directly into a JSP. The JSP is purely used to position the components on the page aiding visual design and real Java classes in tag libs do the rest.
So the evolution has been:
Servlets (Java classes writing HTML logic and intermingling presentation and business logic)
JSP 1.0 JSP (HTML looking) files where you can add scriptlets enclosed in <% ... %> <- you are here
JSP 1.1 Introduction of Tags and Tag libraries, tag libraries are true java classes and the JSP controls layout and simpler more elegant design.
Really JSP 1.1 is all about building reusable components together in tag libraries that can be reused across pages and improving design by not mixing presentation and business logic code together.
If I had inherited something as you have, my advice (if you can) would be to code up some tag libraries because it would be a good learning experience and would be very helpful for moving to more advanced frameworks like JSF.
regards,
Brenden -
JVM - Microsoft JScript.dll Memory Leak when using Java Web Start
I have posted before a topic related to this matter, although now I have a lot more information about what is going on (although still no fix for it). This is the link to the earlier post, containing JVM HOTSPOT EXCEPTION_ACCESS_VIOLATION
http://forum.java.sun.com/thread.jspa?threadID=5116970&messageID=9402346#9402346
The issue is that after deploying on Windows XP a multi threaded, memory consuming (40 Mb) Java Swing application using Java Web Start, and the standard web page (IE6) reference to the jnlp file to launch the installation from Tomcat, each time this application gets executed by double clicking on the desktop icon (with no IE6 interaction at all), calls start being made between jvm.dll (sun) and jscript.dll (Microsoft IE), these interactions never release the memory they consume, so after a period of using the application it crashes unavoidably. If I execute exactly the same application launching it from command line (no Web Start nor IE6 browser involved), the application works nicely all the time, no problem at all.
My question is: does anyone know why these calls to jscript.dll are even being made during web start application execution? As said, it's a desktop Swing app, it does not need the browser at all, and jscript.dll is a IE dll. Anyone can advice on how these interactions can be eliminated?
I've worked with Microsoft support for two weeks on this issue and all they have said is that it is a Sun problem and they can not help further.
Thanks
Jes�sMicrosoft suggested (among others) two test.
1- Install IE7 and run the application
2-Install Firefox and run the application
Results were:
For 1: The application does not crash, apparently IE7 has a newer version of Jscript.dll that, in Microsoft words, handles better 'bad software', refering to 'bad software' to the java virtual machine I guess. Looking at the performance manager you can see memory goes down as with IE6, but at some point memory is released and the app. does not crash. Anyway is still is uncertain to me why Jscript.dll is getting involved at all. Although this solves the problem is not a viable solution for us, because I work in a corporation where changes such as upgrading the IE version on a large number of desktop can not be justified just because one app. is crashing, and it would take a long validation time anyway before the upgrade can proceed.
For 2: I did install Firefox in a machine and the application still crashed, but it is true I did not uninstall IE6, so the conflictive Jscript.dll was still sitting at system32. Quite possibly the crash would not happen if that dll wasn't there at all.
Thanks
Jes�s -
How to launch a Java WebStart application with older JREs when Java 7u25 is on the client?
How can I launch older versions of my Java WebStart application, that are built and run with Java 7u21 or earlier, even if Java 7u25 is installed locally on the client? Application launch and behaviour must be reliable and consistent.
Background:
As of 7u25 (and later), Java Webstart applications launch with a different class loader than pre-7u25.
My Java Webstart application has supported versions that were built with older versions of the JDK (e.g. Java 5, 6, 7u21 or earlier). These applications run with their required JRE version, enforced through JNLP. Once Java 7u25 is installed locally, these older applications fail to launch, due to classloader differences.
The question is: what is required to run older Java WebStart applications even if 7u25 (or later) Java Webstart is installed locally on the client?I confirm your findings when using shortcuts to try specific versions of JavaWS with 7u25 or later installed:
JRE 5u14 launched and the classloader was as pre 7u25
JRE 6u43 would not launch
JRE 7u21 launched but the classloader was not as pre 7u25
You can launch the shortcut with the JavaWS -verbose option to display a messagebox with valuable information.
I am keeping a close watch on this thread.
Maybe you are looking for
-
How do I fix Camera Raw in CS6? I get "Could not complete request because the file was not found"
How do I fix my Camera Raw in Photoshop CS6? I tried to view an NEF file in CS6, but it said I had to update my Camera Raw to 7.3 (I had 7.0). I went online and the only thing I could find was to download 8.6 so I started with that, and installed som
-
Mail app crashes with large mailbox (25k+ messages)
When connected to an exchange account, large mailboxes, whether split across multiple folders or all in the inbox, cause the mail app to crash occasionally. Nothing is lost, but it is not behavior I am comfortable with and I want to know if there is
-
Hi Experts, Could you please clarify my Doubts , I am trying use the BADI MB_MIGO_BADI and trying to enchance\implement the POST_DOCUMENT method. In which I need to change the movement type if some condition met . I am able to access
-
my query executes well but no results ( I am using a TOAD). What's wrong? select dbms_metadata.get_ddl('VIEW','DBA_SEGMENTS') txt from dual;
-
AIM Documentation- Data Mapping for an Upgrade
Hi All- I am tasked to make a data mapping document for a 10.7 - 12.2 upgrade. I have created CV.040's in the past for conversions, but would this template hold true to a upgrade mapping? I am only mapping oracle base tables that have changed. Not cu