Swap space becomes critically low in the system

Hi,
We are facing a situation where the swap space space  is critically low in the system, there is just 10 % available swap space in the system
And we can't see the swap space being released to the OS .
This is one of the  Application servers we have for R3 system.
SAP environment:
SAP Netweaver 7.01
LINUX RHEL
Oracle 11g
$free -m //outut
As you can see we have 64G physical memory allocated to this application server whereas the configured swap space is around 20G.
This Linux box houses two application server one for R3 and the other for CRM.
Would appreciate if you can tell me what should be done to release the swap space as above shows that 18 gig of 20 is being used right now.
Regards,
Rakesh

Thanks Gaurav..
what you have said makes sense we will check that parameter which determines to what extent swap space should be used between 0 to 100 %.
However, I have found that there are many swaps in the SAP buffer (ST02) that we will be tuning soon in both of the application servers that I had mentioned above,
I have checked the ST06 and couldn't found any page in/out activity.
see below the screen shots of  ST02 and ST06 for both application servers.
Regards,
Rakesh

Similar Messages

  • Swap space/dump devices for large RAM systems

    We are in the process of changing our standards for our server builds and need some input. Currently, we match the swap space on local disk with the size of RAM. I know this isn't necessary for "swap" any more, but we use the swap area as the dump device as well, so we are sure to have enough space there to contain a full memory dump if the server crashes.
    Now we have systems coming in with 128 or even 192 GB of ram. They contain only two disks (146GB), which we mirror. Obviously there isn't enough space on these disks for 192GB of swap or a dedicated dump deivce.
    We need the ability to capture full dumps when we have a crash, probably with a dedicated dump device. Should we start ordering SCSI disk enclosuers for this purpose? Can we make a SAN volume our dump device?
    What is everyone else doing with these larger RAM systems?

    Compressed dumps have been supported in UNIX flavors for many years, so you need to dig further into it. It is supported in linux, so I can't imagine why it wouldn't in Solaris. Tru64UNIX, HP-UX, AIX, and others all provide this capability.
    True, you can't get internal disks bigger than the 146 on Sparc platforms, but you could attach a "boot array", like a storagetek D240 system, or iScsi or fiber attached storage. The only caveat is whether Solaris, in the throes of a crash, still has the drivers loaded that can address an external disk subsystem. That would have to be investigated with SUN. And these arrays support the new 1TB SATA disk drives. We are deploying 6140 shelves with 16TB per shelf. Simply amazing...

  • Lightroom 5.7 just installed with 864GB free disk space.  Receive error message saying disk space is critically low.  ??

    Lightroom 5.7 just installed with 864GB free disk space.  Receive error message saying disk space is critically low.  ??

    I have the same issue with 1.7TB free. This seems to have been an ongoing problem in earlier versions.
    Is there a solution?

  • I have purchased a macbook air with 64 gb hard disk. the available space is only 10gb. With this available space, i can't use the system effectively. kindly advise.

    I have purchased a macbook air with 64 gb hard disk. the available space is only 10gb. With this available space, i can't use the system effectively. kindly advise.

    You should have bought one with a larger SSD. You can regain some space by disabling the sleepimage file:
    To disable safe sleep, run the two following commands in Terminal:
    $ sudo pmset -a hibernatemode 0
    $ sudo nvram "use-nvramrc?"=false
    When done, restart your computer. Now go delete the file "/private/var/vm/sleepimage" to free up some hard drive space. When you put your computer to sleep it, should happen in under five seconds; my MacBook now goes to sleep in two seconds.
    [robg adds: To state the obvious, with safe sleep disabled, a total power loss will wipe out whatever was open on your machine. To enable safe sleep mode again, repeat the above commands, but change hibernatemode 0 on the first line to hibernatemode 3, and =false to =true on the second line. You'll then need to reboot again. Personally, I prefer the safe sleep mode, even with the slower sleep time and hard drive consumption -- even if for no other reason than it's great when changing batteries on a flight.]
    You can also delete unneeded files:
    Freeing Up Space on The Hard Drive
      1. See Lion/Mountain Lion's Storage Display.
      2. You can remove data from your Home folder except for the /Home/Library/ folder.
      3. Visit The XLab FAQs and read the FAQ on freeing up space on your hard drive.
      4. Also see Freeing space on your Mac OS X startup disk.
      5. See Where did my Disk Space go?.
      6. See The Storage Display.
    You must Empty the Trash in order to recover the space they occupied on the hard drive.
    You should consider replacing the drive with a larger one. Check out OWC for drives, tutorials, and toolkits.
    Try using OmniDiskSweeper 1.8 or GrandPerspective to search your drive for large files and where they are located.

  • Swap space for multiple instaces on the same box

    OS: Oracle Enterprise Linux 5 Update 2 (64-bit)
    DB: 10.2.0.4
    I'm fairly new to Linux and have a general question about configuring swap space on a Linux box running 10g. From the 10g: Managing Oracle on Linux for DBAs class, Oracle gives the following recommendations about swap size:
    RAM (Swap)
    <= 2Gb (150% of the RAM size)
    Between 2Gb - 8Gb (Equal to the RAM size)
    8Gb+ (75% of the RAM size)
    It's not clear to me from reading the book if this recommendation is good for all Oracle processes running on the machine, or if it's a per instance value? The box has 16Gb of RAM but there are going to be three instances running on it and I need to verify that 12Gb of swap will support all three without issue or if I'll need to configure 12Gb swap per instance.
    Appreciate any input.

    As noted earlier in this thread, swap space is an operating system property, and has nothing to do with software you want to run on top of it, like an oracle database.
    The function of swap space is to be able cope with memory allocations beyond the amount of physical memory. In most occasions you do not want to use swap space, because it has a severe performance impact (memory blocks have to be exchanged from the swap device to physical memory).
    (please mind that on some unixes (AIX and HPUX if I am not mistaking), swap is pre-allocated for processes which means the swap size is not only depended of operating system and memory size, but also the number of processes. Linux does not do that, it starts to allocate swap pages under memory pressure. There are certain issues with some memory settings in linux (linux kernel memory management is quite automatic) that can get your system to page. (memory overcommitting is one thing))
    In my experience, the amount of memory which is used is depended on the amount of memory on the machine in most cases. Whilst this sound simple, think of it: with 2GB used, probably most DBA's would allocate approximately 1.5GB to the databases on that machine, if your machine got 16GB you probably want approximately 14GB to be used by the databases.
    This means (for me) that I use swapspace with the same size as the amount of memory. That few extra GB's allocated to swap won't cost you or your company an arm and a leg, so have that for not running out of memory.

  • How to get available disk space for a disk in the system ?

    hi all,
    I need to figure out the available disk space for a particular disk before I can write my file to a directory in the disk. I do this to avoid unsufficient disk space problem in my program. May I know how can I do this in Java ? I look at the API function for java.io.File class but there isn't any function to check disk space. I would appreciate any help from fellow Java developers. Thanks a million.
    regards,
    tuck meng

    You cannot do this in purely platform independent way.
    You can use System.exec to run native OS utilities like df and parse the output, but then you have to knwo which platform your program is running on, or build in some automatic detection of OS and have corresponding native calls for each likely one.
    BTW. even if you get a number of free bytes on disk that's larger than the size of the file you want to write this is no guarantee that there will be enough diskspace for your file, because another process might write to the disk in the meantime. So you will have to do proper error handling and recover gracefully from the disk full sitution no matter what.

  • Solaris: The System does not have the required swap space

    I'm trying to install Oracle onto Solaris 9. The installation check fails with the error
    The System does not have the required swap space.
    I've read the install Guide I get the following when I do the memory checks
    bash-2.05# /usr/sbin/prtconf | grep "Memory size"
    Memory size: 1024 Megabytes
    bash-2.05# /usr/sbin/swap -s
    total: 767688k bytes allocated + 117080k reserved = 884768k used, 379040k available
    The install guide doesn't say how to increase the Swap Space. I would like to increase the Swap Space just for the duration of the installation. From the install guide I'll need a Swap Space of 1.5 times the Ram on the machine so I'll need 1.5GB Swap Space. Can someone help me out with the required syntax to increase the Swap File Size?

    Found something on the web that did the trick. Create a few files and then add to your swap.. In case it's of interest to anyone else here's what I had to do
    swap -l (lists files in swap)
    mkdir /tmp/swap/ (create dir for file to add to swap)
    mkfile tmp/swap/swap11 (create file for swap)
    swap -a tmp/swap/swap11 (add file to swap)
    swap -l (check files in swap)

  • GrowableArray.cpp. Out of swap space?

    Hi,
    I need some help interpreting an error log (the complete log is pasted below)
    The application (JBoss 4.2.2) is started on win 2003 with -Xms256m -Xmx1400m -XX:MaxPermSize=256m
    the log shows the following memory usage:
    Heap
    PSYoungGen      total 141760K, used 13076K [0x61650000, 0x6b170000, 0x6b1d0000)
      eden space 124992K, 0% used [0x61650000,0x61650000,0x69060000)
      from space 16768K, 77% used [0x69060000,0x69d25230,0x6a0c0000)
      to   space 16448K, 0% used [0x6a160000,0x6a160000,0x6b170000)
    PSOldGen        total 1274368K, used 1246444K [0x139d0000, 0x61650000, 0x61650000)
      object space 1274368K, 97% used [0x139d0000,0x5fb0b398,0x61650000)
    PSPermGen       total 60032K, used 59667K [0x039d0000, 0x07470000, 0x139d0000)
      object space 60032K, 99% used [0x039d0000,0x07414c88,0x07470000)Some of the threads that I read about the "growableArray.cpp. Out of swap space?" problem indicated that the problem may not be related to my memory settings.
    However, the PSPermGen section is included under "Heap". I always thought the MaxPermsize is in addition to the heap size. Also the total of PSPermGen is significantly lower than specified in MaxPermSize while at the same time the usage is at 99%.
    At the moment I can not really pinpoint what situation exactly is causing the problem since the scenario is rather complex, pushing a lot of data into the system, a lot of clients in parallel running queries and requesting the data etc. But with this particular scenario we can reproduce this error within a couple of hours.
    Would it help to increase the virtual address space of windows to 3GB? Would the jvm actually take advantage of it?
    Here is the complete log
    # An unexpected error has been detected by Java Runtime Environment:
    # java.lang.OutOfMemoryError: requested 1024000 bytes for GrET in C:\BUILD_AREA\jdk6_02\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
    #  Internal Error (414C4C4F434154494F4E0E494E4C494E450E4850500017), pid=4312, tid=11828
    # Java VM: Java HotSpot(TM) Server VM (1.6.0_02-b06 mixed mode)
    # If you would like to submit a bug report, please visit:
    #   http://java.sun.com/webapps/bugreport/crash.jsp
    ---------------  T H R E A D  ---------------
    Current thread (0x6ba32000):  VMThread [id=11828]
    Stack: [0x6bd80000,0x6bdd0000)
    [error occurred during error reporting, step 110, id 0xc0000005]
    VM_Operation (0x7363edd0): parallel gc failed allocation, mode: safepoint, requested by thread 0x6c291800
    ---------------  P R O C E S S  ---------------
    Java Threads: ( => current thread )
      0x77839c00 JavaThread "RMI TCP Connection(434)-172.16.9.236" daemon [_thread_in_native, id=5048]
      0x709ca400 JavaThread "JMS SessionPool Worker-728" daemon [_thread_blocked, id=8176]
      0x78d57400 JavaThread "JMS SessionPool Worker-727" daemon [_thread_blocked, id=15028]
      0x78df1400 JavaThread "JMS SessionPool Worker-726" daemon [_thread_blocked, id=6672]
      0x7b58f400 JavaThread "JMS SessionPool Worker-725" daemon [_thread_blocked, id=7920]
      0x78c1e400 JavaThread "JMS SessionPool Worker-724" daemon [_thread_blocked, id=6744]
      0x78dcd800 JavaThread "JMS SessionPool Worker-723" daemon [_thread_blocked, id=6576]
      0x78dcd400 JavaThread "JMS SessionPool Worker-722" daemon [_thread_blocked, id=7620]
      0x7b899800 JavaThread "JMS SessionPool Worker-721" daemon [_thread_blocked, id=7868]
      0x78e83c00 JavaThread "JMS SessionPool Worker-720" daemon [_thread_blocked, id=7948]
      0x78e0ac00 JavaThread "JMS SessionPool Worker-719" daemon [_thread_blocked, id=5928]
      0x78cbec00 JavaThread "JMS SessionPool Worker-718" daemon [_thread_blocked, id=16232]
      0x78d6ec00 JavaThread "JMS SessionPool Worker-717" daemon [_thread_blocked, id=7240]
      0x7b9d6c00 JavaThread "JMS SessionPool Worker-716" daemon [_thread_blocked, id=16828]
      0x78ad4c00 JavaThread "JMS SessionPool Worker-715" daemon [_thread_blocked, id=10300]
      0x78dd7c00 JavaThread "JMS SessionPool Worker-714" daemon [_thread_blocked, id=15012]
      0x78ec0c00 JavaThread "JMS SessionPool Worker-713" daemon [_thread_blocked, id=10888]
      0x7d6bdc00 JavaThread "JMS SessionPool Worker-712" daemon [_thread_blocked, id=5888]
      0x79f3e400 JavaThread "JMS SessionPool Worker-711" daemon [_thread_blocked, id=14468]
      0x7d6bec00 JavaThread "JMS SessionPool Worker-710" daemon [_thread_blocked, id=16032]
      0x7d6be400 JavaThread "JMS SessionPool Worker-709" daemon [_thread_blocked, id=14572]
      0x7fc30400 JavaThread "JMS SessionPool Worker-708" daemon [_thread_blocked, id=10932]
      0x7d6b8400 JavaThread "JMS SessionPool Worker-707" daemon [_thread_blocked, id=15264]
      0x7d666400 JavaThread "JMS SessionPool Worker-706" daemon [_thread_blocked, id=14580]
      0x7d6b1400 JavaThread "JMS SessionPool Worker-705" daemon [_thread_blocked, id=13996]
      0x78e50000 JavaThread "JMS SessionPool Worker-704" daemon [_thread_blocked, id=14964]
      0x7d6b2400 JavaThread "JMS SessionPool Worker-703" daemon [_thread_blocked, id=14852]
      0x7e981000 JavaThread "JMS SessionPool Worker-702" daemon [_thread_blocked, id=6548]
      0x79800800 JavaThread "JMS SessionPool Worker-701" daemon [_thread_blocked, id=11060]
      0x795b0400 JavaThread "JMS SessionPool Worker-700" daemon [_thread_blocked, id=9112]
      0x78de8800 JavaThread "JMS SessionPool Worker-699" daemon [_thread_blocked, id=4984]
      0x7bc3c800 JavaThread "JMS SessionPool Worker-698" daemon [_thread_blocked, id=6712]
      0x7d81a400 JavaThread "JMS SessionPool Worker-697" daemon [_thread_blocked, id=14808]
      0x7bcf2400 JavaThread "JMS SessionPool Worker-696" daemon [_thread_blocked, id=14660]
      0x7bbbb400 JavaThread "JMS SessionPool Worker-691" daemon [_thread_blocked, id=15036]
      0x7bcaa400 JavaThread "JMS SessionPool Worker-689" daemon [_thread_blocked, id=8568]
      0x788be400 JavaThread "JMS SessionPool Worker-688" daemon [_thread_blocked, id=7712]
      0x78fb0400 JavaThread "JMS SessionPool Worker-686" daemon [_thread_blocked, id=8676]
      0x79047400 JavaThread "JMS SessionPool Worker-685" daemon [_thread_blocked, id=11928]
      0x7bd35400 JavaThread "JMS SessionPool Worker-684" daemon [_thread_blocked, id=8348]
      0x792f9400 JavaThread "JMS SessionPool Worker-678" daemon [_thread_blocked, id=1732]
      0x7a0f8400 JavaThread "JMS SessionPool Worker-677" daemon [_thread_blocked, id=5932]
      0x7fa9b800 JavaThread "JMS SessionPool Worker-676" daemon [_thread_blocked, id=7300]
      0x79f2dc00 JavaThread "JMS SessionPool Worker-666" daemon [_thread_blocked, id=10304]
      0x79902400 JavaThread "JMS SessionPool Worker-653" daemon [_thread_blocked, id=10488]
      0x7f19d400 JavaThread "JMS SessionPool Worker-641" daemon [_thread_blocked, id=11020]
      0x79a83400 JavaThread "JMS SessionPool Worker-633" daemon [_thread_blocked, id=16852]
      0x79d5d800 JavaThread "JMS SessionPool Worker-601" daemon [_thread_blocked, id=12972]
      0x709a2400 JavaThread "WorkerThread#3[172.16.9.236:1441]" [_thread_blocked, id=15300]
      0x6f2fb400 JavaThread "http-172.16.9.236-8080-192" daemon [_thread_blocked, id=1908]
      0x78937400 JavaThread "http-172.16.9.236-8080-191" daemon [_thread_blocked, id=15528]
      0x788f9400 JavaThread "http-172.16.9.236-8080-190" daemon [_thread_blocked, id=14636]
      0x7889f400 JavaThread "http-172.16.9.236-8080-189" daemon [_thread_in_native, id=15664]
      0x7878f800 JavaThread "http-172.16.9.236-8080-188" daemon [_thread_blocked, id=15024]
      0x78ac0400 JavaThread "http-172.16.9.236-8080-187" daemon [_thread_blocked, id=14748]
      0x788f3c00 JavaThread "http-172.16.9.236-8080-186" daemon [_thread_blocked, id=4420]
      0x6f2cbc00 JavaThread "http-172.16.9.236-8080-185" daemon [_thread_blocked, id=14740]
      0x6f789800 JavaThread "http-172.16.9.236-8080-184" daemon [_thread_in_native, id=16184]
      0x788d7800 JavaThread "http-172.16.9.236-8080-183" daemon [_thread_blocked, id=15088]
      0x6f8f0400 JavaThread "http-172.16.9.236-8080-182" daemon [_thread_blocked, id=16208]
      0x6f878400 JavaThread "http-172.16.9.236-8080-181" daemon [_thread_in_native, id=12376]
      0x6f67b400 JavaThread "http-172.16.9.236-8080-180" daemon [_thread_blocked, id=15472]
      0x6f715800 JavaThread "http-172.16.9.236-8080-179" daemon [_thread_blocked, id=17320]
      0x6f808c00 JavaThread "http-172.16.9.236-8080-178" daemon [_thread_blocked, id=16268]
      0x6fd3dc00 JavaThread "http-172.16.9.236-8080-177" daemon [_thread_in_native, id=11148]
      0x6f981c00 JavaThread "http-172.16.9.236-8080-176" daemon [_thread_in_native, id=7184]
      0x6f861c00 JavaThread "http-172.16.9.236-8080-175" daemon [_thread_in_native, id=16180]
      0x6f9b3800 JavaThread "http-172.16.9.236-8080-174" daemon [_thread_blocked, id=16128]
      0x6f9b3400 JavaThread "http-172.16.9.236-8080-173" daemon [_thread_blocked, id=16176]
      0x6f445800 JavaThread "http-172.16.9.236-8080-172" daemon [_thread_in_native, id=6344]
      0x6f3e3800 JavaThread "http-172.16.9.236-8080-171" daemon [_thread_in_native, id=6496]
      0x6e3fa400 JavaThread "http-172.16.9.236-8080-170" daemon [_thread_in_native, id=5056]
      0x70dd3400 JavaThread "http-172.16.9.236-8080-169" daemon [_thread_in_native, id=8736]
      0x6eacf400 JavaThread "http-172.16.9.236-8080-168" daemon [_thread_in_native, id=8716]
      0x6cd32800 JavaThread "JMS SessionPool Worker-120" daemon [_thread_blocked, id=7368]
      0x790bf400 JavaThread "JMS SessionPool Worker-119" daemon [_thread_blocked, id=7596]
      0x7905fc00 JavaThread "JMS SessionPool Worker-118" daemon [_thread_blocked, id=7844]
      0x7905f400 JavaThread "JMS SessionPool Worker-117" daemon [_thread_blocked, id=5160]
      0x790ef400 JavaThread "JMS SessionPool Worker-116" daemon [_thread_blocked, id=936]
      0x7094a800 JavaThread "JMS SessionPool Worker-115" daemon [_thread_blocked, id=13688]
      0x6e5b2400 JavaThread "JMS SessionPool Worker-114" daemon [_thread_blocked, id=7404]
      0x6fa5d400 JavaThread "JMS SessionPool Worker-113" daemon [_thread_blocked, id=10112]
      0x6e66a400 JavaThread "JMS SessionPool Worker-86" daemon [_thread_blocked, id=16188]
      0x6e474400 JavaThread "http-172.16.9.236-8080-167" daemon [_thread_blocked, id=17304]
      0x6e43d400 JavaThread "http-172.16.9.236-8080-166" daemon [_thread_blocked, id=17124]
      0x6e3fe800 JavaThread "http-172.16.9.236-8080-165" daemon [_thread_in_native, id=17276]
      0x6e45bc00 JavaThread "http-172.16.9.236-8080-164" daemon [_thread_in_native, id=17188]
      0x6e45b400 JavaThread "http-172.16.9.236-8080-163" daemon [_thread_blocked, id=13100]
      0x6e483800 JavaThread "http-172.16.9.236-8080-162" daemon [_thread_blocked, id=17932]
      0x6e04ac00 JavaThread "http-172.16.9.236-8080-161" daemon [_thread_in_native, id=17344]
      0x7850e800 JavaThread "http-172.16.9.236-8080-160" daemon [_thread_in_native, id=12496]
      0x706da400 JavaThread "http-172.16.9.236-8080-159" daemon [_thread_in_native, id=12472]
      0x70814400 JavaThread "http-172.16.9.236-8080-158" daemon [_thread_blocked, id=17456]
      0x6e131800 JavaThread "http-172.16.9.236-8080-157" daemon [_thread_blocked, id=12644]
      0x6e105400 JavaThread "http-172.16.9.236-8080-156" daemon [_thread_in_native, id=17404]
      0x6e241400 JavaThread "http-172.16.9.236-8080-155" daemon [_thread_blocked, id=6608]
      0x6f88d400 JavaThread "http-172.16.9.236-8080-154" daemon [_thread_in_native, id=17396]
      0x6ccde800 JavaThread "http-172.16.9.236-8080-153" daemon [_thread_in_native, id=17400]
      0x7099f400 JavaThread "http-172.16.9.236-8080-152" daemon [_thread_in_native, id=12620]
      0x6e250c00 JavaThread "http-172.16.9.236-8080-151" daemon [_thread_blocked, id=17508]
      0x6e3eb400 JavaThread "http-172.16.9.236-8080-150" daemon [_thread_in_native, id=17468]
      0x6f7be800 JavaThread "http-172.16.9.236-8080-149" daemon [_thread_blocked, id=13412]
      0x70b87400 JavaThread "http-172.16.9.236-8080-148" daemon [_thread_blocked, id=17328]
      0x6e394400 JavaThread "http-172.16.9.236-8080-147" daemon [_thread_blocked, id=9076]
      0x6f73dc00 JavaThread "http-172.16.9.236-8080-146" daemon [_thread_blocked, id=12488]
      0x6f73d400 JavaThread "http-172.16.9.236-8080-145" daemon [_thread_blocked, id=12860]
      0x6e0b0400 JavaThread "http-172.16.9.236-8080-144" daemon [_thread_blocked, id=12660]
      0x6df10800 JavaThread "http-172.16.9.236-8080-143" daemon [_thread_blocked, id=17476]
      0x6de63800 JavaThread "http-172.16.9.236-8080-142" daemon [_thread_blocked, id=12540]
      0x6df33400 JavaThread "http-172.16.9.236-8080-141" daemon [_thread_blocked, id=12484]
      0x6c280800 JavaThread "http-172.16.9.236-8080-140" daemon [_thread_in_native, id=11688]
      0x6c236000 JavaThread "http-172.16.9.236-8080-139" daemon [_thread_blocked, id=13900]
      0x6dec1800 JavaThread "http-172.16.9.236-8080-138" daemon [_thread_in_native, id=4988]
      0x6df8d400 JavaThread "http-172.16.9.236-8080-137" daemon [_thread_in_native, id=12548]
      0x6df59400 JavaThread "http-172.16.9.236-8080-136" daemon [_thread_in_native, id=18416]
      0x6de5c800 JavaThread "http-172.16.9.236-8080-135" daemon [_thread_blocked, id=12536]
      0x6dea0400 JavaThread "http-172.16.9.236-8080-134" daemon [_thread_blocked, id=10492]
      0x6dfb0400 JavaThread "http-172.16.9.236-8080-133" daemon [_thread_in_native, id=12888]
      0x6e142800 JavaThread "http-172.16.9.236-8080-132" daemon [_thread_blocked, id=12700]
      0x6de75400 JavaThread "http-172.16.9.236-8080-131" daemon [_thread_in_native, id=12880]
      0x6de49400 JavaThread "http-172.16.9.236-8080-130" daemon [_thread_blocked, id=12608]
      0x6c2c5800 JavaThread "http-172.16.9.236-8080-129" daemon [_thread_blocked, id=12796]
      0x6cd22c00 JavaThread "http-172.16.9.236-8080-128" daemon [_thread_blocked, id=12820]
      0x6cd22400 JavaThread "http-172.16.9.236-8080-127" daemon [_thread_blocked, id=13676]
      0x6cc29c00 JavaThread "http-172.16.9.236-8080-126" daemon [_thread_in_native, id=13024]
      0x6cc29400 JavaThread "http-172.16.9.236-8080-125" daemon [_thread_in_native, id=12464]
      0x789bd800 JavaThread "http-172.16.9.236-8080-124" daemon [_thread_blocked, id=4788]
      0x6de77400 JavaThread "JMS SessionPool Worker-85" daemon [_thread_blocked, id=6556]
      0x6e367800 JavaThread "Thread-159" daemon [_thread_blocked, id=5536]
      0x6c85a800 JavaThread "http-172.16.9.236-8080-123" daemon [_thread_blocked, id=13124]
      0x6c245400 JavaThread "http-172.16.9.236-8080-122" daemon [_thread_in_native, id=5152]
      0x6de7d400 JavaThread "http-172.16.9.236-8080-121" daemon [_thread_blocked, id=13280]
      0x6c97e800 JavaThread "http-172.16.9.236-8080-120" daemon [_thread_blocked, id=13508]
      0x6ccd6400 JavaThread "http-172.16.9.236-8080-119" daemon [_thread_in_native, id=18184]
      0x6ccce400 JavaThread "http-172.16.9.236-8080-118" daemon [_thread_blocked, id=13264]
      0x7042b800 JavaThread "http-172.16.9.236-8080-117" daemon [_thread_blocked, id=9724]
      0x70752400 JavaThread "http-172.16.9.236-8080-116" daemon [_thread_in_native, id=10796]
      0x6cc61400 JavaThread "http-172.16.9.236-8080-115" daemon [_thread_blocked, id=9276]
      0x796e8c00 JavaThread "http-172.16.9.236-8080-114" daemon [_thread_blocked, id=10136]
      0x796e8400 JavaThread "http-172.16.9.236-8080-113" daemon [_thread_in_native, id=4692]
      0x796d0800 JavaThread "http-172.16.9.236-8080-112" daemon [_thread_blocked, id=13476]
      0x79710400 JavaThread "http-172.16.9.236-8080-111" daemon [_thread_in_native, id=1536]
      0x6caa0800 JavaThread "http-172.16.9.236-8080-110" daemon [_thread_in_native, id=13260]
      0x6caa0400 JavaThread "http-172.16.9.236-8080-109" daemon [_thread_blocked, id=8884]
      0x6ca0f800 JavaThread "http-172.16.9.236-8080-108" daemon [_thread_blocked, id=7036]
      0x6ca0f400 JavaThread "http-172.16.9.236-8080-107" daemon [_thread_blocked, id=13156]
      0x6c290400 JavaThread "http-172.16.9.236-8080-106" daemon [_thread_blocked, id=10184]
      0x795c9c00 JavaThread "http-172.16.9.236-8080-105" daemon [_thread_blocked, id=8280]
      0x6f15f800 JavaThread "http-172.16.9.236-8080-104" daemon [_thread_blocked, id=9880]
      0x6f15f400 JavaThread "http-172.16.9.236-8080-103" daemon [_thread_blocked, id=9344]
      0x795e9c00 JavaThread "http-172.16.9.236-8080-102" daemon [_thread_blocked, id=17808]
      0x6de80400 JavaThread "http-172.16.9.236-8080-101" daemon [_thread_in_native, id=14384]
      0x79251400 JavaThread "http-172.16.9.236-8080-100" daemon [_thread_in_native, id=6560]
      0x6cac0400 JavaThread "http-172.16.9.236-8080-99" daemon [_thread_in_native, id=5792]
      0x78d97400 JavaThread "http-172.16.9.236-8080-98" daemon [_thread_blocked, id=8936]
      0x709a6400 JavaThread "http-172.16.9.236-8080-97" daemon [_thread_blocked, id=9340]
      0x793d0c00 JavaThread "http-172.16.9.236-8080-96" daemon [_thread_in_native, id=13880]
      0x7943f800 JavaThread "http-172.16.9.236-8080-95" daemon [_thread_blocked, id=8248]
      0x7940f800 JavaThread "http-172.16.9.236-8080-94" daemon [_thread_in_native, id=9752]
      0x7941f800 JavaThread "http-172.16.9.236-8080-93" daemon [_thread_blocked, id=9212]
      0x6ca5a400 JavaThread "http-172.16.9.236-8080-92" daemon [_thread_blocked, id=8780]
      0x7942fc00 JavaThread "http-172.16.9.236-8080-91" daemon [_thread_in_native, id=18200]
      0x6f5cf400 JavaThread "http-172.16.9.236-8080-90" daemon [_thread_in_native, id=8748]
      0x78b16400 JavaThread "http-172.16.9.236-8080-89" daemon [_thread_blocked, id=9744]
      0x70919c00 JavaThread "http-172.16.9.236-8080-88" daemon [_thread_in_native, id=5564]
      0x6cc2a800 JavaThread "http-172.16.9.236-8080-87" daemon [_thread_blocked, id=16932]
      0x6debc400 JavaThread "JMS SessionPool Worker-84" daemon [_thread_blocked, id=13928]
      0x6cd26400 JavaThread "http-172.16.9.236-8080-86" daemon [_thread_blocked, id=1576]
      0x6de4d400 JavaThread "http-172.16.9.236-8080-85" daemon [_thread_in_native, id=4428]
      0x6de67800 JavaThread "http-172.16.9.236-8080-84" daemon [_thread_blocked, id=13876]
      0x6de67400 JavaThread "http-172.16.9.236-8080-83" daemon [_thread_blocked, id=11440]
      0x6de81c00 JavaThread "http-172.16.9.236-8080-82" daemon [_thread_blocked, id=276]
      0x6dea4c00 JavaThread "http-172.16.9.236-8080-81" daemon [_thread_in_native, id=13836]
      0x6df23800 JavaThread "http-172.16.9.236-8080-80" daemon [_thread_blocked, id=5280]
      0x6f160400 JavaThread "http-172.16.9.236-8080-79" daemon [_thread_blocked, id=17548]
      0x6df23400 JavaThread "http-172.16.9.236-8080-78" daemon [_thread_in_native, id=14276]
      0x6dee4c00 JavaThread "http-172.16.9.236-8080-77" daemon [_thread_blocked, id=5880]
      0x6ffc1800 JavaThread "http-172.16.9.236-8080-76" daemon [_thread_in_native, id=17908]
      0x7042fc00 JavaThread "http-172.16.9.236-8080-75" daemon [_thread_blocked, id=13536]
      0x6f043c00 JavaThread "http-172.16.9.236-8080-74" daemon [_thread_blocked, id=15312]
      0x7050b400 JavaThread "http-172.16.9.236-8080-73" daemon [_thread_in_native, id=4408]
      0x6cb96800 JavaThread "http-172.16.9.236-8080-72" daemon [_thread_blocked, id=4248]
      0x6cbf1400 JavaThread "http-172.16.9.236-8080-71" daemon [_thread_in_native, id=11844]
      0x6cbec800 JavaThread "http-172.16.9.236-8080-70" daemon [_thread_blocked, id=9644]
      0x6ca79800 JavaThread "http-172.16.9.236-8080-69" daemon [_thread_blocked, id=13672]
      0x6cb76c00 JavaThread "http-172.16.9.236-8080-68" daemon [_thread_blocked, id=14304]
      0x6cb97c00 JavaThread "http-172.16.9.236-8080-67" daemon [_thread_blocked, id=18396]
      0x6cb97400 JavaThread "http-172.16.9.236-8080-66" daemon [_thread_blocked, id=17604]
      0x6fea2400 JavaThread "http-172.16.9.236-8080-65" daemon [_thread_blocked, id=17816]
      0x6fea3800 JavaThread "http-172.16.9.236-8080-64" daemon [_thread_in_native, id=8612]
      0x6c8e7400 JavaThread "http-172.16.9.236-8080-63" daemon [_thread_blocked, id=14460]
      0x6c9af800 JavaThread "http-172.16.9.236-8080-62" daemon [_thread_blocked, id=13824]
      0x6c9adc00 JavaThread "http-172.16.9.236-8080-61" daemon [_thread_blocked, id=17972]
      0x6c9ad400 JavaThread "http-172.16.9.236-8080-60" daemon [_thread_blocked, id=740]
      0x6c8e7c00 JavaThread "http-172.16.9.236-8080-59" daemon [_thread_in_native, id=14112]
      0x6c8eb800 JavaThread "http-172.16.9.236-8080-58" daemon [_thread_blocked, id=15376]
      0x6c19b800 JavaThread "http-172.16.9.236-8080-57" daemon [_thread_blocked, id=14164]
      0x6c19c000 JavaThread "http-172.16.9.236-8080-56" daemon [_thread_blocked, id=13916]
      0x6c9dcc00 JavaThread "http-172.16.9.236-8080-55" daemon [_thread_in_native, id=17896]
      0x6c8e8800 JavaThread "http-172.16.9.236-8080-54" daemon [_thread_blocked, id=18364]
      0x6ca58400 JavaThread "http-172.16.9.236-8080-53" daemon [_thread_blocked, id=14104]
      0x6c8eac00 JavaThread "http-172.16.9.236-8080-52" daemon [_thread_in_native, id=18376]
      0x6c9ea800 JavaThread "http-172.16.9.236-8080-51" daemon [_thread_blocked, id=13764]
      0x6c9d1000 JavaThread "http-172.16.9.236-8080-50" daemon [_thread_blocked, id=14156]
      0x6c9ce400 JavaThread "http-172.16.9.236-8080-49" daemon [_thread_blocked, id=9156]
      0x6c8e9800 JavaThread "http-172.16.9.236-8080-48" daemon [_thread_blocked, id=11404]
      0x6c9b3c00 JavaThread "http-172.16.9.236-8080-47" daemon [_thread_blocked, id=18288]
      0x6c9b1000 JavaThread "http-172.16.9.236-8080-46" daemon [_thread_blocked, id=2028]
      0x6c352400 JavaThread "http-172.16.9.236-8080-45" daemon [_thread_in_native, id=14280]
      0x6c351000 JavaThread "http-172.16.9.236-8080-44" daemon [_thread_in_native, id=18280]
      0x6c85b000 JavaThread "http-172.16.9.236-8080-43" daemon [_thread_in_native, id=18152]
      0x6ed96400 JavaThread "http-172.16.9.236-8080-42" daemon [_thread_in_native, id=14260]
      0x6c294c00 JavaThread "http-172.16.9.236-8080-41" daemon [_thread_blocked, id=14144]
      0x6c44bc00 JavaThread "http-172.16.9.236-8080-40" daemon [_thread_blocked, id=8252]
      0x6c2acc00 JavaThread "http-172.16.9.236-8080-39" daemon [_thread_blocked, id=17992]
      0x6c70f000 JavaThread "http-172.16.9.236-8080-38" daemon [_thread_in_native, id=17860]
      0x6c70c000 JavaThread "http-172.16.9.236-8080-37" daemon [_thread_blocked, id=11664]
      0x6c6bf400 JavaThread "http-172.16.9.236-8080-36" daemon [_thread_blocked, id=18400]
      0x6c6bfc00 JavaThread "http-172.16.9.236-8080-35" daemon [_thread_blocked, id=17692]
      0x6c6c0800 JavaThread "http-172.16.9.236-8080-34" daemon [_thread_blocked, id=13640]
      0x6c70f800 JavaThread "http-172.16.9.236-8080-33" daemon [_thread_blocked, id=10276]
      0x6c6c2000 JavaThread "http-172.16.9.236-8080-32" daemon [_thread_in_native, id=13244]
      0x6c70d400 JavaThread "http-172.16.9.236-8080-31" daemon [_thread_in_native, id=17928]
      0x6c6c1000 JavaThread "http-172.16.9.236-8080-30" daemon [_thread_blocked, id=18208]
      0x6c34c400 JavaThread "http-172.16.9.236-8080-29" daemon [_thread_blocked, id=11952]
      0x6ed0a000 JavaThread "http-172.16.9.236-8080-28" daemon [_thread_blocked, id=13192]
      0x6ed09400 JavaThread "http-172.16.9.236-8080-27" daemon [_thread_blocked, id=18136]
      0x6c6bd800 JavaThread "http-172.16.9.236-8080-26" daemon [_thread_blocked, id=18252]
      0x6ed0b000 JavaThread "http-172.16.9.236-8080-25" daemon [_thread_in_native, id=18176]
      0x6c6bc400 JavaThread "http-172.16.9.236-8080-24" daemon [_thread_in_native, id=17676]
      0x6c246400 JavaThread "http-172.16.9.236-8080-23" daemon [_thread_in_native, id=18352]
      0x6c199400 JavaThread "http-172.16.9.236-8080-22" daemon [_thread_in_native, id=18264]
      0x6c6be400 JavaThread "http-172.16.9.236-8080-21" daemon [_thread_in_native, id=18372]
      0x6c247400 JavaThread "http-172.16.9.236-8080-20" daemon [_thread_in_native, id=18256]
      0x6c199c00 JavaThread "http-172.16.9.236-8080-19" daemon [_thread_in_native, id=17688]
      0x6c19f400 JavaThread "http-172.16.9.236-8080-18" daemon [_thread_blocked, id=9956]
      0x6c19e800 JavaThread "http-172.16.9.236-8080-17" daemon [_thread_blocked, id=18260]
      0x6c293400 JavaThread "http-172.16.9.236-8080-16" daemon [_thread_blocked, id=18324]
      0x6fe2b800 JavaThread "http-172.16.9.236-8080-15" daemon [_thread_blocked, id=17920]
      0x6c292000 JavaThread "http-172.16.9.236-8080-14" daemon [_thread_blocked, id=17672]
      0x6c13bc00 JavaThread "http-172.16.9.236-8080-13" daemon [_thread_blocked, id=18268]
      0x6c1ed400 JavaThread "http-172.16.9.236-8080-12" daemon [_thread_blocked, id=13008]
      0x6c1ed800 JavaThread "http-172.16.9.236-8080-11" daemon [_thread_blocked, id=12728]
      0x6c13b400 JavaThread "http-172.16.9.236-8080-10" daemon [_thread_blocked, id=11532]
      0x6c139000 JavaThread "http-172.16.9.236-8080-9" daemon [_thread_blocked, id=17852]
      0x6c138800 JavaThread "http-172.16.9.236-8080-8" daemon [_thread_blocked, id=5320]
      0x6c291000 JavaThread "http-172.16.9.236-8080-7" daemon [_thread_in_native, id=18276]
      0x6c353800 JavaThread "http-172.16.9.236-8080-6" daemon [_thread_blocked, id=18240]
      0x6c291800 JavaThread "http-172.16.9.236-8080-5" daemon [_thread_blocked, id=18328]
      0x6c13cc00 JavaThread "http-172.16.9.236-8080-4" daemon [_thread_blocked, id=18244]
      0x6bc76400 JavaThread "http-172.16.9.236-8080-3" daemon [_thread_blocked, id=17788]
      0x7092e400 JavaThread "JMS SessionPool Worker-83" daemon [_thread_blocked, id=14004]
      0x704c0800 JavaThread "http-172.16.9.236-8080-2" daemon [_thread_blocked, id=17784]
      0x6c0ea400 JavaThread "JMS SessionPool Worker-82" daemon [_thread_blocked, id=17988]
      0x70669400 JavaThread "JMS SessionPool Worker-81" daemon [_thread_blocked, id=18024]
      0x704e0400 JavaThread "JMS SessionPool Worker-79" daemon [_thread_blocked, id=12612]
      0x70d0e800 JavaThread "JMS SessionPool Worker-78" daemon [_thread_blocked, id=6032]
      0x6f507400 JavaThread "JMS SessionPool Worker-77" daemon [_thread_blocked, id=13400]
      0x7048b400 JavaThread "JMS SessionPool Worker-76" daemon [_thread_blocked, id=6996]
      0x6e5f6400 JavaThread "JMS SessionPool Worker-75" daemon [_thread_blocked, id=13004]
      0x78c63800 JavaThread "JMS SessionPool Worker-50" daemon [_thread_blocked, id=8452]
      0x7872e400 JavaThread "JMS SessionPool Worker-49" daemon [_thread_blocked, id=5824]
      0x6fb92400 JavaThread "JMS SessionPool Worker-48" daemon [_thread_blocked, id=11616]
      0x786d0800 JavaThread "JMS SessionPool Worker-47" daemon [_thread_blocked, id=13044]
      0x6f598400 JavaThread "RMI RenewClean-[172.16.9.236:1098]" daemon [_thread_blocked, id=6632]
      0x70c99800 JavaThread "http-172.16.9.236-8080-1" daemon [_thread_blocked, id=10280]
      0x6fb4fc00 JavaThread "JMS SessionPool Worker-28" daemon [_thread_blocked, id=13420]
      0x783ff400 JavaThread "Thread-35" daemon [_thread_blocked, id=11040]
      0x6f6ec000 JavaThread "Thread-34" [_thread_blocked, id=1952]
      0x6e491c00 JavaThread "Timer-11" daemon [_thread_blocked, id=11888]
      0x6fc3ac00 JavaThread "JMS SessionPool Worker-5" daemon [_thread_blocked, id=9332]
      0x6f17f000 JavaThread "JMS SessionPool Worker-4" daemon [_thread_blocked, id=11924]
      0x6f94c800 JavaThread "JMS SessionPool Worker-3" daemon [_thread_blocked, id=7360]
      0x6ec30400 JavaThread "JMS SessionPool Worker-2" daemon [_thread_blocked, id=11376]
      0x6fbaf000 JavaThread "JMS SessionPool Worker-1" daemon [_thread_blocked, id=11280]
      0x6ed46c00 JavaThread "JMS SessionPool Worker-0" daemon [_thread_blocked, id=8244]
      0x6de7ac00 JavaThread "Thread-32" [_thread_blocked, id=9876]
      0x6ef11800 JavaThread "WorkerThread#2[172.16.9.236:1579]" [_thread_blocked, id=9336]
      0x6e825800 JavaThread "WorkerThread#1[172.16.9.236:1686]" [_thread_blocked, id=10964]
      0x6eae0c00 JavaThread "WorkerThread#0[172.16.9.236:1489]" [_thread_blocked, id=11668]
      0x6fe80400 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=11976]
      0x6df8a800 JavaThread "Thread-5" [_thread_blocked, id=12840]
      0x7003a400 JavaThread "ajp-172.16.9.236-8009-Acceptor-0" daemon [_thread_in_native, id=12744]
      0x6f9c0400 JavaThread "http-172.16.9.236-8080-Acceptor-0" daemon [_thread_in_native, id=13788]
      0x6f185800 JavaThread "JBossLifeThread" [_thread_blocked, id=12000]
      0x6f7a9400 JavaThread "Timer-10" daemon [_thread_blocked, id=5576]
      0x6e4ff800 JavaThread "Timer-9" daemon [_thread_blocked, id=7876]
      0x6ded8400 JavaThread "QuartzScheduler_DeadlineScheduler-1_MisfireHandler" [_thread_blocked, id=13756]
      0x6c7ae400 JavaThread "DeadlineScheduler_QuartzSchedulerThread" [_thread_blocked, id=13948]
      0x6c451800 JavaThread "DeadlineScheduler_Worker-5" [_thread_blocked, id=13772]
      0x6c440800 JavaThread "DeadlineScheduler_Worker-4" [_thread_blocked, id=13752]
      0x6c3e9800 JavaThread "DeadlineScheduler_Worker-3" [_thread_blocked, id=13800]
      0x6c461800 JavaThread "DeadlineScheduler_Worker-2" [_thread_blocked, id=13428]
      0x6c607800 JavaThread "DeadlineScheduler_Worker-1" [_thread_blocked, id=13616]
      0x6fada400 JavaThread "Connection Consumer for dest JBossTopic[AuditTopic] id=6" [_thread_blocked, id=9500]
      0x6f5d0400 JavaThread "Thread-24" [_thread_blocked, id=10296]
      0x70472400 JavaThread "Connection Consumer for dest JBossTopic[AuditTopic] id=5" [_thread_blocked, id=8888]
      0x70007800 JavaThread "Thread-23" [_thread_blocked, id=10752]
      0x6fe43400 JavaThread "Connection Consumer for dest JBossTopic[ErrorTopic] id=4" [_thread_blocked, id=1920]
      0x6fe03400 JavaThread "Thread-22" [_thread_blocked, id=13844]
      0x6fdcb800 JavaThread "Connection Consumer for dest JBossTopic[AuditTopic] id=3" [_thread_blocked, id=13516]
      0x6ee22c00 JavaThread "Thread-21" [_thread_blocked, id=13704]
      0x7006e800 JavaThread "Connection Consumer for dest JBossQueue[ActiveActivityQueue] id=2" [_thread_blocked, id=17912]
      0x6f56d800 JavaThread "Thread-20" [_thread_blocked, id=14120]
      0x70294400 JavaThread "Connection Consumer for dest JBossTopic[FreeTextTopic] id=1" [_thread_blocked, id=7636]
      0x6fa6ac00 JavaThread "Thread-19" [_thread_blocked, id=12200]
      0x6edfa800 JavaThread "Timer-8" daemon [_thread_blocked, id=4388]
      0x6cc84400 JavaThread "Timer-7" daemon [_thread_blocked, id=16304]
      0x6bc77800 JavaThread "Timer-6" daemon [_thread_blocked, id=11292]
      0x6bc77400 JavaThread "Timer-5" daemon [_thread_blocked, id=5168]
      0x6dff5400 JavaThread "Timer-4" daemon [_thread_blocked, id=8232]
      0x6e5cc400 JavaThread "Thread-12" [_thread_blocked, id=12792]
      0x6c162800 JavaThread "ConnectionTable.Connection.Receiver [172.16.9.236:7900 - 172.16.9.236:59770]" daemon [_thread_in_native, id=7236]
      0x6c2dd800 JavaThread "ConnectionTable.Connection.Receiver [172.16.9.236:59770 - 172.16.9.236:7900]" daemon [_thread_in_native, id=13608]
      0x6c612c00 JavaThread "MERGE2.FindSubgroups thread (c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        

    uGohde wrote:
    Hi,
    Would it help to increase the virtual address space of windows to 3GB? Would the jvm actually take advantage of it?Forgot to mention this in my last post ...
    No, it wouldn't help. The JDK (VM + library native code) isn't linked with the /LARGEADDRESSAWARE option so windows won't give it a 3G address space. The JDK code has been tested in this mode and works, but the change was not made because it could break existing JNI code which is not expecting to addresses larger than 2G. See
    [bug 5061380|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5061380]

  • Excess Swap Space....

    Hi all,
    Please I would like to find out if large swap space has any effect on the performance of a Server.
    Thanks

    No. Generally not a good idea.
    Oh, details? Sure.
    To begin with you do <u>not</u> want any active paging going on; it kills performance quickly. It is normal to see some memory pages being evicted to paging store; the kernel does some proactive evictions of stale memory pages just in case the room is needed sometime in the future.
    $ top -n1
    top - 07:37:55 up 3 days, 19:58,  1 user,  load average: 0.01, 0.82, 1.26
    Tasks: 302 total,  15 running, 287 sleeping,   0 stopped,   0 zombie
    Cpu(s):  4.5%us,  6.2%sy,  0.1%ni, 88.2%id,  0.8%wa,  0.0%hi,  0.1%si,  0.0%st
    Mem:   6902244k total,  6486860k used,   415384k free,   600968k buffers
    Swap:  8392696k total,    14496k used,  8378200k free,  1290552k cachedThe swap utilization reported by top(1) is misleading because it shows a static picture of how much swap space is being used. The problem is that you can't distinguish between the stale-memory evictions and the unwanted paging activity. Better to use the vmstat(1) tool for this:
    $ vmstat 1 3
    procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
    1  0  14496 411440 601624 1293196    0    0    48   103   28   24  5  6 88  1  0
    61  0  14496 411316 601632 1293196    0    0     0   212 5224 6751  2  4 93  2  0
    5  0  14496 411152 601640 1293188    0    0     0    16 7205 8826  5 11 83  1  0The interesting metric here is the swap-in (si) column, which counts the number of pages read into memory from the paging store. It is OK for this to show some minor values, but should the count increase rapidly for some time you should regard this as a sign of swap activity and fix it.
    Lots of installation documentation suggests that the size of paging store be at least equal to the physical memory size or even larger. On anything much larger than an old desktop this is simply too much space: the system will become unusably slow long before this amount of space is utilized.
    Ancient history: at one time some UNIX ports used the swap space to temporarily store a memory image of a system oops. Every boot, the kernel checked the swap space for a special signature pattern to see if the swap space held such an image.
    Back to the present: having an excessively-large paging store simply wastes disk space. Sometimes the extra paging store can be useful to have because it can postpone a Linux out-of-memory (OOM) kill by the kernel if you have a run-away program leaking memory, but most likely this simply trades an OOM kill for really poor performance. Your mileage may vary, as they say.

  • Problem with swap space

    Hi All,
    Using Oracle 10gR1, Solaris 9.
    prtconf | grep "Memory size"
    Memory size: 16384 Megabytesswap space is 16g
    Pga_aggregate_target = 2G and after checking the v$pga_target_advice, it shows optimal.
    Currently we have a issue of swap space getting used up nearly 100%. The system was out of memory and can't run any application
    Alert logs error;
    ORA-07445: exception encountered: core dump [_aio_free_stack_unlocked()+72] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7B602000] [] []
    ORA-00600: internal error code, arguments: [ksnpost:ksnigb], [], [], [], [], [], [], []
    ORA-04030: out of process memory when trying to allocate 588408 bytes (pga heap,kco buffer)On checking Metalink note, it said its an unhandled Oracle exception; have to increase the swap space.
    AWR excerpt;
    Cache Sizes (end)
    ~~~~~~~~~~~~~~~~~
                   Buffer Cache:     4,848M      Std Block Size:         8K
               Shared Pool Size:     3,985M          Log Buffer:     1,024K
    Load Profile
    ~~~~~~~~~~~~                            Per Second       Per Transaction
                      Redo size:             63,328.92              2,116.68
                  Logical reads:              4,946.83                165.34
                  Block changes:                257.73                  8.61
                 Physical reads:              1,267.50                 42.36
                Physical writes:                 72.44                  2.42
                     User calls:                385.83                 12.90
                         Parses:                 84.22                  2.81
                    Hard parses:                  0.04                  0.00
                          Sorts:                 10.53                  0.35
                         Logons:                  0.14                  0.00
                       Executes:                211.35                  7.06
                   Transactions:                 29.92
      % Blocks changed per Read:    5.21    Recursive Call %:    13.56
    Rollback per transaction %:    0.29       Rows per Sort:    94.72
    Top 5 Timed Events
    ~~~~~~~~~~~~~~~~~~                                        % Total
    Event                                 Waits    Time (s)   DB Time     Wait Class
    row cache lock                       16,498      45,001     43.44    Concurrency
    log file switch (checkpoint in       30,766      30,281     29.23  Configuration
    db file sequential read           2,873,716      10,796     10.42       User I/O
    buffer busy waits                     8,870       8,718      8.42    Concurrency
    db file parallel write                  421       3,848      3.71     System I/O
              -------------------------------------------------------------On increasing the swap space, and restarting the instance, the system was normal.
    Now why does oracle use up all the swap space and what can be the workaround to avoid this in future?

    Verma wrote:
    Using Oracle 10gR1, Solaris 9.
    swap space is 16gOracle 10gR2 swap space recommendations are:
    - 1 GB - 2 GB = 1.5 times the size of RAM
    - 2 GB - 8 GB = equal to the size of RAM
    - Greater than 8GB = 0.75 times the size of RAM
    Currently we have a issue of swap space getting used up nearly 100%. The system was out of memory and can't run any applicationThat is unusual and should not be happening when you have what seems to be an excess of memory available. This could mean some kind of bug (memory leakage for example), or it could mean plain old fashion abuse of memory. Like PL/SQL code that attempts to bulk fetch a few million rows from the database with a single go.
    On checking Metalink note, it said its an unhandled Oracle exception; have to increase the swap space.Wrong IMO (unless you were not following the recommended swap size factor). If you have have a giant swap area..moving the wall a few meters away (by increasing swap) will only enable you to run faster into it - I doubt that this is the intent of that Metalink note.
    On increasing the swap space, and restarting the instance, the system was normal.
    Now why does oracle use up all the swap space and what can be the workaround to avoid this in future?Oracle only use what is needed to perform the client requests that it receives. If the client request a mountain of data to be moved/processed, Oracle will try to oblige. If that mountain topples and crushes server resources, it is not Oracle that is to blame for trying to do exactly what was requested from it.

  • SWAP space usage??

    When I am monitoring my system using TOP, all of my java process is running well over 350M. The Size field is showing 470M and the RES field is showing 350M. But my Xms and Xmx is set to 256M. Is this mean the applications are using SWAP space?

    Not necessarily. The total memory used by the JVM process will always be more than the Java heap size, specified by ms and mx. Memory used by the JVM for internal management, memory allocated in native code and memory allocated for the permanent space are all allocated outside of the Java heap space.
    You could be using swap space if the total memory reserved by all applications and the OS in the system exceeds the physical RAM installed.
    Arunabh

  • Swap Space on Solaris 8

    We just installed Solaris 8 and allocated 2GB of swap space on the first slice of the disk. When I issue a df -k command, this comes out.
    swap 3.5 GB /var/run
    swap 3.5 GB /tmp
    Is this normal? Where did it get the extra 1.5 GB that it allocated for swap? How about the other 3.5 GB where did it all come from? Can somebody shed light on this? Thanks.

    That's normal. To quote from the system administration guide:
    http://docs.sun.com
    �The Solaris environment uses the concept of virtual swap space, a layer between anonymous memory
    pages and the physical storage (or disk-backed swap space) that actually back these pages. A
    system's virtual swap space is equal to the sum of all its physical (disk-backed) swap space plus a
    portion of the currently available physical memory.�
    I.e. you probably have ~ 2GB memory in the machine + the 2 GB disk
    based swap you've configured gives ~3.5GB virtual swap.
    The two mount points /tmp and /var/run are two different filesystems
    that allocate space from the same virtual swap space. They share the
    swap resource, that is, if you put things in /tmp, not only the space
    available in /tmp but also the space available for /var/run decreases.

  • Decreased min free swap space

    Hello everyone,
    due to technical problems, in 07/Dic/2005 we had to replace a disk in one of our cluster server of SAP R/3 Enterprise. We also increased its level security from RAID 0 to RAID 5.
    But today we have detected than since that moment, the min free swap space has decreased considerably, from 12800 KB (approx.) to 1100 KB (approx) !! (with ST06 transaction). The replaced disk has the SAME characteristics of the previous one
    How can we solve this situation and return to original swap space values? Which is the optimal configuration of swap space in SAP and Windows 2003 Server?? We have two nodes in a cluster (one with the application server and one with the database), and each one has 4Gb of memory.
    Current commit charge values of Windows 2003 Server task Manager are:
    Commit charge (K):
    Limit: 7931568
    Peak: 7925732
    This is the SAP answer to the note I opened:
    "Please ask your hardware partner about this. At the moment I do not know anything about the symptom on other customer systems. My opinion is that the workload has been changed, thus increased peak values. Please look at the installation guide "
    Can yoy help me?
    Thanks in advance,
    Óscar Sánchez
    [email protected]

    Hi Oskar,
    My suggestion:
    Activate NT performance monitor and try to identify
    the time when you have peak usage of swap and application that caused it. To my knowledge, even 12800 KB free swap is not acceptable amount (at least, there should be couple of hundreds megabites available). Check your process monitor to see what applications take much memory. If you replaced your disk on the C: drive of the other MSCS node, I don't see any ways that this replacement could affect to the other node swap size. (I am assuming that C: drive is not sharing through the nodes)
    Regards,
    Mike

  • Swap space reduced drastically

    Hi Techies
    We have solaies 10,oracle10g ,BI
    Our production server is working fine but suddenly the swap space will reduced from 259GB to 100GB in 15 min of time span( when i checked in df -h) and then apllication will hang.
    We have a work around like we will restart the application then the swap space will be released to the allocated
    In this due time if i check prststat (process status) IGS process is consuming 150GB of space only.
    what could be the reason for reducing the swap space drastically???
    Is this the problem of application or Operating system
    I have checked the note (724713)
    Thank You
    Haroon

    Thanks for the quick reply
    df -h ( detail)
    Filesystem             size   used  avail capacity  Mounted on
    swap                   258G   394M   258G     1%    /tmp
    swap                   258G    56K   258G     1%    /var/run
    swap                   258G     0K   258G     0%    /dev/vx/dmp
    swap                   258G     0K   258G     0%    /dev/vx/rdmp
    From the above the allocated size swap is reduced drastically when the issues arise.
    There are 57 users connected to the server
    the physical memory is 65 GB
    ( i also checked the note 425207 profile parameter
    em/initial_size_MB =8192
    ztta/roll_extension           2000683008 Byte

  • How to find Oracle is using swap space or not?

    Hi,
    I have a SuSe Linux database server on which 2 Oracle 10g databases and one MySQL server is running.
    I can see a very little swap space is being used in the server while there is almost 100% RAM utlization.
    I want to know if my Oracle databases are using swap or not. If it is using, how much of swap being used currently by each Oracle database?
    Can someone please guide me on how to do that?
    Thanks in advance for your help.
    Regards,
    Murali Mohan

    I did "vmstat 1 10" and here is the output.
    What I want to know how much swap space is used by each of my oracle database while the below output is for total swap usage
    procs -----------memory---------- ---swap-- -----io---- system ----cpu----
    r b swpd free buff cache si so bi bo in cs us sy id wa
    0 0 2366860 36996 81660 9458664 67 44 4160 356 9 13 5 3 82 9
    0 0 2366860 38224 81748 9458576 0 0 0 512 1041 369 0 0 99 1
    1 0 2366860 38224 81748 9458576 0 0 4 80 1053 293 6 0 93 0
    2 0 2366860 38224 81752 9458572 0 0 80 40 1467 956 22 7 70 1
    2 1 2366860 38248 81760 9458564 0 0 0 580 1150 444 32 17 50 1
    1 1 2366860 38124 81808 9458516 0 0 360 1040 1672 1158 28 13 51 8
    2 0 2366860 38124 81832 9458492 0 0 84 288 1380 931 33 10 57 1
    2 0 2366860 36596 81884 9458388 52 0 52 4088 1098 475 36 16 46 1
    2 0 2366860 38188 81916 9458356 0 0 144 1096 1251 570 36 12 49 2
    2 0 2366860 38180 81916 9458356 0 0 68 136 1595 1134 28 9 63 1

Maybe you are looking for

  • Hide grand total field on ALV report, But keep the sub total fields

    Dear Experts, 1) How to hide grand total field on ALV report, But keep the sub total fields. 2) How to populate rate per ton & calculate total price according to the qty in delivering document.

  • Send Using: option not available when replying to emails

    I have multiple e-mail accounts setup on my flip. Yahoo, Gmail, Hotmail, work account on BES. When I reply to any message I got to my Hotmail account, at the top there is a 'Send Using:' and it is set to the account I am currently replying from, in t

  • Wrt54g wireless router

    i just got a wrt54g wireless router.my modem is a dsl modem. verizon is my internet provider. i go through the setup disc and follow everything it says exactly. but when i get up to the point of were it says checking internet connection it says it ca

  • Need a copy of my receipt

    I need a copy of the receipt for our TV. The location we bought it from has us in their system multiple times and did not want to look through them to help. I saw another post where someone emailed the receipt to the person asking. Can someone email

  • Registered array not visible in CAM

    Hi Team I am facing an issue while registering 2500 and 61X0 series storages in CAM installed in my laptop. I am able to ping the controller IP. During the registration process CAM is abe to find out the storage with its name and firmware details. Bu