High CPU on CSS11503 , with one Flow

Hi,
I've seen some high CPU Utilisation on 2 CSS.
I've figured out, that this Utilisation was created, by a mirroring Task from a database Server.
There was one TCP Connection between 2 Servers, whitch transfered about 60 Mbit/s.
But i saw a CPU-Utilisation from about 30-40% on my SCM which passed the Traffic.
That is some thing i do not understand, because the Forwading of this Flow shoud be done in the Flow-Processor, and the CPU Utilisation which is monitored, is the Session Processor.
The Protocol was TCP 1500, which is Used by Tivoli Storage Manager.
Anybody has some Idea, how this makes so much CPU Utilisation?
The Transfer was about 4 hours, and the CPU Utilisation also.
Best regards
Sven

Hello Gilles,
may i try to show you the network design
<-> <-> <-> css <->
So the traffic was hitting 2 css.
One 11506 with 7.50 103
One 11503 with 7.50.103
The traffic was one TCP Connection between Server1 and Server2 (TCP 1500).
There are no ACL and the traffic was not hitting a content rule.
The 11506 handels normal about 30 mbit/s out, and 10 mbit/s in. Average CPU is about 1% over a day.
The 11053 handles normaly about 5mit/ out and 1 mbit/s in. Average CPU is about 4% over a day.
CPU was about 30% higher on both css, during the replicarton (~3 hours)
I did not took any debug or show commands, because i saw it on my mrtg a day later.
Also, the high cpu was only seen on the SCM Module, the second Module (IOM) was normal.
Best regads
Sven

Similar Messages

  • Tecra M2 - hard shutdown on high CPU (turns off with no BSOD)

    Have been experiencing consistent shutdowns when CPU runs at full for a little while.
    - by "shutdowns" I mean complete power cut with no blue screen of death where all is lost and the computer has to be manually restarted.
    - by "high CPU" I mean when the processor locks at full usage such as when you insert a column into a large Excel sheet with errors or when you are RARing a large number of files and you open a video from the network or when you run a virus scan and registry scan at the same time.
    The problem has been occurring regularly for a couple of months and I can now manufacture an occurrence with high predictability. Using the hardware monitor suite 'Everest', it seems the problem happens when the CPU temperature sits at 99 for more than a minute or so.
    It happen with some but never when running from batteries - during attempts to force the error, it will not shutdown if the power cable is pulled out just before the crash point.
    In many instances of the problem, there seems to be a small error message which pops up only a fraction of a second before the computer dies.
    Every forum and help reference I read would suggest the problem is hardware related and that the chip or motherboard is overheating and in need of replacement. While the laptop is nearly 18 months old, I bought a new "everything but the screen and HDD" early this year and was not prepared to accept the explanation without more testing.
    Kantonix Linux works fine and the boot from CD seems to tax the processor as much as the same events that are killing it for me under XP.
    A few weeks back, I installed the WinXP on a second HDD in the hotswappable bay and noted the installation seems to run the computer pretty hot and it did not die during the process. No problems were experienced for while but, a few days after moving the reinstalled HDD into the primary position and having reinstalled many many applications, the error started happening again.
    Last night I caught it just fractions of a second before death, pulling the power and stopping the process and allowing me to see the error message (as referred above) properly:
    - first message was from "TOSHIBA Power Saver" saying "A fatal errorn has occurred x06" (not sure on the last bit)
    - second message was from "THotkey.exe" saying "Cannot load powrprof.dll"
    - many hundreds of instances of the second message followed.
    Due to error messages, I wasn't able to get a screen print.
    Any solutions or even ideas about such would be very much appreciated!
    Message was edited by: DC-Financial

    Hello Markus, thank you for your suggestion and apologies for the lateness of this reply (thought I was supposed to receive email notification (?!?)).
    I agree with your initial assessment "sounds like software related issue" and I would hate to think it is hardware - am die hard Toshiba man! (onto my 5th in as many years) .
    Anywho, I have been through all the power saver options, as well as the CPU options under the Toshiba HWSetup: the system crashes under all situations if properly encouraged.
    NB: Per my understanding, one shouldn't be able to crash a Pentium M, operating within the acceptable environmental conditions and I am working in a room between 16 & 20 degrees.
    It is much more difficult to crash when it has been off for a while (ie. when it is cooler) and is more difficult to crash when on batteries as opposed to AC. Tried switching the RAM during the week and after mistakenly thinking it was all fixed, the problem resurfaced again. While I had the keyboard off, I had a look at the fan and there is a bit of dust on it but not very much. I have a Tecra M3 also, which is only half the age and, compared the fans, the M2 is about twice as dusty [if it were on your dinner table you'd wipe it down but there is not so much that you would imagine it to be a problem].
    I mentioned previously that I was using Everest to monitor outputs from the temperature sensors and that crashes occur when the CPU temperature sits at 99 degrees for a while. Sometimes after turning it on after a crash, Everest is not able to display the CPU temperature. Am very reluctant to believe that the heat sensor on the chip has malfunctioned as all the internals were replaced in January (as was stolen and recovered somewhat worse for the adventure).
    I have had to bite the bullet and move all my work to the Tecra M3 but am not very happy with this (the M3 I have is 1.86Ghz w/ 512MB vs 2.03Ghz w/ 1536MB RAM in the M2). As I'm sure you will appreciate, the cost of the M2 has now already been spent many times over in lost work time but one needs to know if they have an error related to software or if it is hardware, so as not to live in fear of a recurrance.
    I will reinstall the M2 OS as soon as the work lag is caught up again (probably later this week) but really don't know what to try other than crash testing after each piece of software is reinistalled. Note, have already run the Toshiba Hardware Diagnosis console with no errors identified.
    If you have any suggestions or thoughts on what I should try they are greatly appreciated!

  • High CPU usage: Animation with background

    I experience a high CPU usage when animating a shape over a background. If I disable the background or the animation, the CPU usage drops to almost zero. It seems like the combination of background shapes and an animated shape is very costly.
    Is there a way to improve this performance? Can I indicate that the animated shape is always on top? Adding the smooth and cache tags did not improve the performance.
    Thanks in advance.
    import javafx.stage.Stage;
    import javafx.scene.Scene;
    import javafx.animation.Interpolator;
    import javafx.animation.*;
    import javafx.animation.transition.*;
    import javafx.scene.shape.Rectangle;
    import javafx.scene.paint.Color;
    var nBackgrounds = 20;
    var screenWidth = 1024;
    var screenHeight = 800;
    def animatingRectangle = Rectangle{
        width: screenWidth
        height: screenHeight
        fill:Color.BLUE
    def background = for(i in [1..nBackgrounds]) Rectangle{
        width: screenWidth
        height: screenHeight
        fill:Color.RED
    def liveView:Scene = Scene {
        width: screenWidth
        height: screenHeight
        content: [
            background,
            animatingRectangle,
        ] // content
    } // Scene
    Stage {
        title: "Example"
        scene: liveView   
    } // Stage
    var fakeImageTransition:TranslateTransition = TranslateTransition {
        fromY: bind animatingRectangle.translateY
        toY: bind -screenHeight
        duration: 10s
        node: animatingRectangle
        interpolator : Interpolator.EASEOUT
        framerate: 30
        repeatCount:50
    fakeImageTransition.playFromStart();

    I'm having the same issue. I have 4gb of ram and all of it will be in use even with just Safari running. High fan use. When this happens video will stutter.

  • High CPU-Load while no one is working

    Hello everyone,
    we have a customer with a new installation of Business Object 3.1 SP3 with SAP-Integration
    Kit, LCM, Business Explorer on 1 VmWare-Host.
    After installation nobody is working on the system but the CPU-Load is around 30%
    from "fileserver.exe *32" and 10-20% from "CMS.exe *32".
    How can such CPU-Loads be avoided? It's a Windows 2003 Standard Server 64bit.
    Thanks in advance
    Marco

    Hello,
    We are on HP UX and CPU utilization shows almost 80% when no user is accessing the server. Still we havent roll out the solution but can imagine the performance when there would be users simulataneously. Folks, please help me providing document links or tips for this issue.

  • High CPU & GPU temperature with ATI 4870 Graphics card

    Hi I have replaced the original FX1700 card with an ATI 4870 on my S10 and I found the GPU temperature constantly floating at around 58C with no program running and the temperature of the CPU have also gone up to 47C with no programing running and 58C with program running(The temperature of the CPU was fine at under 40C before ATI 4870 was installed). Is this situation normal?
    P.S. I have not replaced the orginal fan came with my S10.
    Thanks very much for your helps
    Solved!
    Go to Solution.

    Hi Erik you are truely intelligent I have tried the method as you suggested and now the temperature of CPU and GPU came down to 45C and 56C at idle respectively. That is really a dramatc improvement  thanks very much!!!

  • 2800s, AIM-VPN-SSL2, vrf aware IPSEC, high CPU low throughput

    We have a couple of new 2821s deployed across a fibre link and they were originally running 12.4 (non T) versions using software encryption. We would get around 8Mb/s throughput. Upgrading to T to use the installed AIM cards we now see the AIM cards in use (show cry isakmp sa det shows then engine as aim vpn), but we still get the same throughput and high CPU. allowing CEF on the interface doubles throughput but with the same high CPU. The only process I can see going high is IP Input. Is this because of vrf aware ipsec - or any other suggestions?

    Hi Nick,
    I am having the same issue. We have a 2851 as a IPSEC VPN headend with an AIM VPN module but we are seeing high CPU usage(80%) with just 4-5mbps worth of traffic. I have an idea that I might have a NAT issue.
    We are currently running, NAT, ZFW, and IPSEC site 2 site VPN on the router.
    When I look at my ZONE firewall policy-map output it is showing all of my VPN traffic as process switched.
    Inspect
    Packet inspection statistics [process switch:fast switch]
    tcp packets: [14809800:0]
    udp packets: [145107:0]
    icmp packets: [20937:12]
    I have disabled the ZFW and still see high cpu although it is a little lower.
    Packets are not fragmented, CEF and fast switching looks to be enabled. I am using a route-map for my nonats. That is the only thing I can think of now.
    I have tried IOS 12.4(20)T3,4 and 12.4(15)T9. Same results.
    Anyone have some ideas?

  • WebKit2WebProcess high cpu usage

    WebKit2WebProcess high cpu usage Safari with windows 7 ... safari unstable ... just started today ... tried reset etc ... problem returns shortly after reset ..

    I love using the Safari Browser compared to any other browser, however the file webkit2webprocess occupies a lot of memory and slows down the browsing, what does this file do and why on earth does it require so much of memory can apple not fix this bug. Have contacted Apple but nobody seems to have an answer to this problem is Apple really working on this issue.

  • FMS 4 configuration / High CPU usage problem

    I am seeing extremly high CPU usage issues with the FMS 4 server.  It seems to be a problem with the FMS Edge process?  If you start and stop the service, the processor goes back to normal operations.
    Looking in the Windows Event logs I am seeing these errors:
    Bad network data; terminating connection : bad chunk version 104 on input stream 00000000077A2E80
    Also:
    Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: Unknown, IP: , App: , Protocol: tunnel) : h o s t = % 2 2 % 2 0 % 2 D F O O B A R % 7 C e c h o % 2 0 % 2 2 % 2 0 S h a r e n a m e % 2 2 % 0 A e c h o % 0 A e c h o % 2 0 % 2 2 % 2 0 % 2 0 S o m e S h a r e % 2 0 % 2 0 D i s k % 2 0 % 2 2 % 6 0 i d % 6 0 % 2 0 % 2 3 % 2 2
    Can someone shed some light on what might be causing these issues?
    Running on a virtual server/ Win2k8R2
    Thanks, Tandrist

    Hi
    Can you please provide us some little more environment information on this issue, like:
      - what were you trying to do when you saw this issue, if there was a live telecast or if that was a simple vod workflow?
    - What publisher application were you using to send livestream to FMS?
    -  were you also doing some kind of recording while doing live streaming??
    - In case of live streaming what audio/video encoding and bitrates was there in use?
    - how many number of connections were there?
    - What amount of cpu and mem usage was there (when you say very high cpu usage).. you can see it using task manager
    It will help if you can also share your fms logs (inside fms_install_folder/logs) so that we can see if something abnormal was going on there..

  • High CPU usage reported on Solaris 8

    Hello,
    In my OEM 10g Grid Control / Hosts view, I have high CPU Util% reported for one of my Solaris servers (pegged at 100% right now); a remote node running the 10g agent.
    However, when I look at the box and use the native Unix commands (uptime, sar, vmstat, iostat, etc), I cannot see why OEM is complaining. When I run "top" on the box, it does show I/O wait @ 97-99%, but that value is kinda bogus, there is no I/O queued or pending, so I'm wondering if the Oracle Agent is somehow using that bogus value of I/O wait to decide on total CPU usage? Anyone seen this, and know a fix?

    I ran 'nmump foo' to get the usage, and it looks like either "osLoad" or "osCpuUsage" would be what it's executing, although the text inside both of those perl scripts says they are only for Linux or Tru64. I'm not sure what to make of the outputs:
    $ nmupm osLoad
    em_result=0.06|0.05|0.06|843677.000000|178|8|279360901.000000|1223751574.000000|
    313790808.000000|98206630.000000|2580027199.000000|8192.000000|4194304.000000|56
    0026113789.000000|4234766618533.000000|1227715348459.000000
    $ nmupm osCpuUsage
    em_result=1|54665464.000000|22348544.000000|597678197.000000|379255063.000000
    em_result=2|93177600.000000|25722959.000000|752622995.000000|182421611.000000
    em_result=3|118210934.000000|29060067.000000|639495186.000000|267178975.000000
    em_result=4|47736833.000000|21075094.000000|590237308.000000|394895925.000000
    I found the metric brower line in emd.properties and uncommented it out, now I'll try to figure out where the 'metric browser' is, and how I see what it's doing...if you happen to catch my reply, and could be a bit more specific about where this is, that'd be great.

  • High CPU usage in cisco 7613 with rsp720-3cxl

    Hi everybody,
    our cisco 7613 has about  4.5 Gbps  Tx/Rx IP traffic in total, and we run ospf with other cisco cloud for routing I list in the following some our router show.what is your idea about our high cpu usage .Is it in normal range with the listed cards and modules.How can I tune the rsp720 and other SIP-200,400,600 for better performances
    why our interrupt rate is high ,and one thing more the total sum of 5sec in separate rows not equal to cpu utilization for five second 50% 
    show proc cpu sor
    CPU utilization for five seconds: 50%/46%; one minute: 54%; five minutes: 59%
     PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
       8   196795220    12640741      15568  1.51%  0.38%  0.26%   0 Check heaps
     224  1048610528  4169501364          0  1.19%  1.45%  1.44%   0 IP Input
      13   374006320  3155162661          0  0.23%  0.26%  0.24%   0 ARP Input
     217   119862004   985030884        121  0.15%  0.32%  0.25%   0 ADJ resolve pro
    c
     185      537716  1825736183          0  0.07%  0.03%  0.02%   0 ACE Tunnel Task
     260     1550992  2983272818          0  0.07%  0.13%  0.15%   0 Ethernet Msec T
    i
     305    38186336    58050485        657  0.07%  0.02%  0.00%   0 XDR mcast
      34       67208    11707798          5  0.07%  0.00%  0.00%   0 IPC Loadometer
      27      232776    57160812          4  0.07%  0.01%  0.00%   0 IPC Periodic Ti
    m
     325    17539200    92894502        188  0.07%  0.15%  0.15%   0 CEF: IPv4 proce
    s
     195     7406636    43782487        169  0.07%  0.00%  0.00%   0 esw_vlan_stat_p
    r
    show ip route summ
    IP routing table name is default (0x0)
    IP routing table maximum-paths is 32
    Route Source    Networks    Subnets     Replicates  Overhead    Memory (bytes)
    static          1           120         0           7620        20812
    connected       0           313         0           18860       53836
    ospf 98         17          4892        0           589020      863984
      Intra-area: 89 Inter-area: 383 External-1: 0 External-2: 0
      NSSA External-1: 0 NSSA External-2: 4437
    bgp 12880       0           1           0           60          172
      External: 1 Internal: 0 Local: 0
    ospf 410        0           269         0           16220       47344
      Intra-area: 1 Inter-area: 0 External-1: 0 External-2: 268
      NSSA External-1: 0 NSSA External-2: 0
    internal        137                                             260544
    Total           155         5595        0           631780      1246692
    sh module
    Mod Ports Card Type                              Model              Serial No.
      1    0  4-subslot SPA Interface Processor-200  7600-SIP-200       
      2    0  4-subslot SPA Interface Processor-400  7600-SIP-400       
      3   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       
      6    1  1-subslot SPA Interface Processor-600  7600-SIP-600       
      7    2  Route Switch Processor 720 (Active)    RSP720-3CXL-GE     
      8    2  Route Switch Processor 720 (Cold)      RSP720-3CXL-GE     
    show ver
    System image file is "bootdisk:c7600rsp72043-adventerprisek9-mz.122-33.SRE2.bin"
    1 SIP-200 controller .
    1 SIP-400 controller (1 Channelized OC3/STM-1).
    1 SIP-600 controller (1 TenGigabitEthernet).
    2 Virtual Ethernet interfaces
    28 Gigabit Ethernet interfaces
    1 Ten Gigabit Ethernet interface
    1 Channelized STM-1 port
    1 Channelized STM-1 port
    show int vlan 1
      Encapsulation ARPA, loopback not set
      Keepalive not supported
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:00, output 00:00:00, output hang never
      Last clearing of "show interface" counters 2d23h
      Input queue: 0/75/2886/1830 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 2380531000 bits/sec, 287383 packets/sec
      5 minute output rate 422133000 bits/sec, 254113 packets/sec
      L2 Switched: ucast: 1200869468 pkt, 101172643240 bytes - mcast: 253599 pkt, 78
    873415 bytes
      L3 in Switched: ucast: 60947040633 pkt, 68919665115039 bytes - mcast: 0 pkt, 0
     bytes mcast
      L3 out Switched: ucast: 52594517004 pkt, 9869168832783 bytes mcast: 0 pkt, 0 b
    ytes
         62147839148 packets input, 69016175499764 bytes, 0 no buffer
         Received 257634 broadcasts (0 IP multicasts)
         0 runts, 0 giants, 15 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
         53647248858 packets output, 10292998021217 bytes, 0 underruns
         0 output errors, 0 interface resets
         0 unknown protocol drops
         0 output buffer failures, 0 output buffers swapped out

    Thank you for your hints and replying
    These are our show ibc in 1 min interval
    Interface information:
            Interface IBC0/0
            5 minute rx rate 20045000 bits/sec, 30183 packets/sec
            5 minute tx rate 47394000 bits/sec, 60212 packets/sec
            19879272237 packets input, 4006174536193 bytes
            19835355282 broadcasts received
            19808585787 packets output, 3981305571968 bytes
            90548 broadcasts sent
            0 Bridge Packet loopback drops
            19756362091 Packets CEF Switched, 1320184 Packets Fast Switched
            0 Packets SLB Switched, 0 Packets CWAN Switched
            Label switched pkts dropped: 0    Pkts dropped during dma: 339549
            Invalid pkts dropped: 0    Pkts dropped(not cwan consumed): 0
            IPSEC pkts dropped: 635184
            Xconnect pkts processed: 0, dropped: 0
            Xconnect pkt reflection drops: 0
            Total paks copied for process level 0
            Total short paks sent in route cache 2605317676
            Total throttle drops 265338    Input queue drops 5831090
            total spd packets classified (120217214 low, 174503 medium, 3073 high)
            total spd packets dropped (339549 low, 0 medium, 0 high)
            spd prio pkts allowed in due to selective throttling (0 med, 0 high)
            IBC resets   = 1; last at 23:52:49.004 Sat Jan 19 2013
    Driver Level Counters: (Cumulative, Zeroed only at Reset)
              Frames          Bytes
      Rx(0)   26537712        3421085217
      Rx(1)   3449063135      2838813650
      Tx(0)   3390340306      2016620276
     Input Drop Frame Count
         Rx0 = 0                Rx1 = 2488435
     Per Queue Receive Errors:
         FRME   OFLW   BUFE   NOENP  DISCRD DISABLE BADCOUNT
     Rx0 0      0      0      0      0        0    0
     Rx1 0      0      0      3633   0        0    0
      Tx Errors/State:
       One Collision Error   = 0            More Collisions       = 0
       No Encap Error        = 0            Deferred Error        = 0
       Loss Carrier Error    = 0            Late Collision Error  = 0
       Excessive Collisions  = 0            Buffer Error          = 0
       Tx Freeze Count       = 0            Tx Intrpt Serv timeout= 1
       Tx Flow State         = FLOW_ON
       Tx Flow Off Count     = 0            Tx Flow On Count      = 0
      Counters collected at Idb:
       Is input throttled    = 0            Throttle Count        = 0
       Rx Resource Errors    = 0            Input Drops           = 2488435
       Input Errors           = 194243
       Output Drops          = 0            Giants/Runts          = 0/0
       Dma Mem Error         = 0            Input Overrun         = 0
    Hash match table for multicast (in use 0, maximum 64 entries):
    show ibc 
    Interface information:
            Interface IBC0/0
            5 minute rx rate 20194000 bits/sec, 30412 packets/sec
            5 minute tx rate 47753000 bits/sec, 60663 packets/sec
            19891125514 packets input, 4007158118761 bytes
            19847185365 broadcasts received
            19820407164 packets output, 3982279276274 bytes
            90576 broadcasts sent
            0 Bridge Packet loopback drops
            19768178233 Packets CEF Switched, 1321008 Packets Fast Switched
            0 Packets SLB Switched, 0 Packets CWAN Switched
            Label switched pkts dropped: 0    Pkts dropped during dma: 339549
            Invalid pkts dropped: 0    Pkts dropped(not cwan consumed): 0
            IPSEC pkts dropped: 635574
            Xconnect pkts processed: 0, dropped: 0
            Xconnect pkt reflection drops: 0
            Total paks copied for process level 0
            Total short paks sent in route cache 2606549061
            Total throttle drops 265338    Input queue drops 5831090
            total spd packets classified (120252754 low, 174531 medium, 3074 high)
            total spd packets dropped (339549 low, 0 medium, 0 high)
            spd prio pkts allowed in due to selective throttling (0 med, 0 high)
            IBC resets   = 1; last at 23:52:49.004 Sat Jan 19 2013
    Driver Level Counters: (Cumulative, Zeroed only at Reset)
              Frames          Bytes
      Rx(0)   26550723        3422835145
      Rx(1)   3461063605      176652699
      Tx(0)   3402319442      3368513724
     Input Drop Frame Count
         Rx0 = 0                Rx1 = 2490155
     Per Queue Receive Errors:
         FRME   OFLW   BUFE   NOENP  DISCRD DISABLE BADCOUNT
     Rx0 0      0      0      0      0        0    0
     Rx1 0      0      0      3633   0        0    0
      Tx Errors/State:
       One Collision Error   = 0            More Collisions       = 0
       No Encap Error        = 0            Deferred Error        = 0
       Loss Carrier Error    = 0            Late Collision Error  = 0
       Excessive Collisions  = 0            Buffer Error          = 0
       Tx Freeze Count       = 0            Tx Intrpt Serv timeout= 1
       Tx Flow State         = FLOW_ON
       Tx Flow Off Count     = 0            Tx Flow On Count      = 0
      Counters collected at Idb:
       Is input throttled    = 0            Throttle Count        = 0
       Rx Resource Errors    = 0            Input Drops           = 2490155
       Input Errors           = 194358
       Output Drops          = 0            Giants/Runts          = 0/0
       Dma Mem Error         = 0            Input Overrun         = 0
    Hash match table for multicast (in use 0, maximum 64 entries):
    and sorry what is your idea about total sum of 5sec in separate rows not equal to cpu utilization for five second 50% 

  • Config Manager Agent - after Hardware Inventory High CPU Usage with WMIPRSVE and very fast empty Battery

    Hi there,
    since a few days there is on some machines (40-60) a high cpu usage on one core (quad core cpu machines) with the WMIPRSVE.EXE if the HARDWARE INVENTORY CYCLE started.
    i try out some tests, read some forum articles and troubleshooting the WMI management but a real problem i doesn´t see.
    in some articles i read that hardware inventory runs about minutes up to more hours but some machines runs longer, someone more as 1 day.
    here an example of mine PC:
    at 8:07 i started Hardware Inventory Cycle, in the InventoryAgent.log i can see that some Collection Namespace are captured.
    after a few minutes there stopped and does nothing round about 5.9 hours or better, after 21436.097 Seconds.
    For any hints i am grateful. :)
    Inventory: *********************** Start of message processing. ***********************
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    Inventory: Message type is InventoryAction InventoryAgent
    18.03.2015 08:09:56 11088 (0x2B50)
    Inventory: Temp directory = C:\WINDOWS\CCM\Inventory\Temp\
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    Inventory: Clearing old collected files. InventoryAgent
    18.03.2015 08:09:56 11088 (0x2B50)
    Inventory: Opening store for action {00000000-0000-0000-0000-000000000001} ...
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    CInvState::VerifyInventoryVersionNumber: Mismatch found for '{00000000-0000-0000-0000-000000000001}': 4.2 vs. 0.0
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    Inventory: Version number mismatch; will do a Full report.
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    Inventory: Action=Hardware, ReportType=ReSync, MajorVersion=5, MinorVersion=0
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    Inventory: Initialization completed in 0.141 seconds
    InventoryAgent 18.03.2015 08:09:56
    11088 (0x2B50)
    Collection: Namespace = \\localhost\root\Microsoft\appvirt\client; Query = SELECT __CLASS, __PATH, __RELPATH, CachedLaunchSize, CachedPercentage, CachedSize, LaunchSize, Name, PackageGUID, TotalSize, Version, VersionGUID FROM Package; Timeout = 600 secs.
    InventoryAgent 18.03.2015 08:09:56
    7836 (0x1E9C)
    Failed to get IWbemService Ptr for \\localhost\root\vm\VirtualServer Namespace: 8004100E
    InventoryAgent 18.03.2015 08:10:02
    7836 (0x1E9C)
    Failed to enumerate instances of VirtualMachine: 8004100E
    InventoryAgent 18.03.2015 08:10:02
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2\sms; Query = SELECT __CLASS, __PATH, __RELPATH, AddressWidth, BrandID, CPUHash, CPUKey, DataWidth, DeviceID, Family, Is64Bit, IsHyperthreadCapable, IsMobile, IsTrustedExecutionCapable, IsVitualizationCapable, Manufacturer,
    MaxClockSpeed, Name, NormSpeed, NumberOfCores, NumberOfLogicalProcessors, PCache, ProcessorId, ProcessorType, Revision, SocketDesignation, Status, SystemName, Version FROM SMS_Processor; Timeout = 600 secs.
    InventoryAgent 18.03.2015 08:10:02
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\CCM\powermanagementagent; Query = SELECT __CLASS, __PATH, __RELPATH, Requester, RequesterInfo, RequesterType, RequestType, Time, UnknownRequester FROM CCM_PwrMgmtLastSuspendError; Timeout = 600 secs.
    InventoryAgent 18.03.2015 08:10:03
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, Availability, Description, DeviceID, Manufacturer, Name, Status FROM Win32_IDEController; Timeout = 600 secs.
    InventoryAgent 18.03.2015 08:10:03
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2\sms; Query = SELECT __CLASS, __PATH, __RELPATH, BinFileVersion, BinProductVersion, Description, ExecutableName, FilePropertiesHash, FilePropertiesHashEx, FileSize, FileVersion, HasPatchAdded, InstalledFilePath, IsSystemFile,
    IsVitalFile, Language, Product, ProductCode, ProductVersion, Publisher FROM SMS_InstalledExecutable; Timeout = 600 secs.
    InventoryAgent 18.03.2015 08:10:03
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, DefaultIPGateway, DHCPEnabled, DHCPServer, DNSDomain, DNSHostName, Index, IPAddress, IPEnabled, IPSubnet, MACAddress, ServiceName FROM Win32_NetworkAdapterConfiguration; Timeout
    = 600 secs. InventoryAgent
    18.03.2015 14:06:43 7836 (0x1E9C)
    Collection: Namespace = \\.\root\Nap; Query = SELECT __CLASS, __PATH, __RELPATH, description, fixupState, friendlyName, id, infoClsid, isBound, percentage, registrationDate, vendorName, version FROM NAP_SystemHealthAgent; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:06:43
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2\sms; Query = SELECT __CLASS, __PATH, __RELPATH, AdditionalProductCodes, CompanyName, ExplorerFileName, FileDescription, FilePropertiesHash, FileSize, FileVersion, FolderPath, LastUsedTime, LastUserName, msiDisplayName,
    msiPublisher, msiVersion, OriginalFileName, ProductCode, ProductLanguage, ProductName, ProductVersion, SoftwarePropertiesHash FROM CCM_RecentlyUsedApps; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:06:43
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, BankLabel, Capacity, Caption, CreationClassName, DataWidth, Description, DeviceLocator, FormFactor, HotSwappable, InstallDate, InterleaveDataDepth, InterleavePosition, Manufacturer,
    MemoryType, Model, Name, OtherIdentifyingInfo, PartNumber, PositionInRow, PoweredOn, Removable, Replaceable, SerialNumber, SKU, Speed, Status, Tag, TotalWidth, TypeDetail, Version FROM Win32_PhysicalMemory; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:07:02
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, Availability, Description, DeviceID, InstallDate, Manufacturer, Name, PNPDeviceID, ProductName, Status FROM Win32_SoundDevice; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:07:02
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, Caption, ClassGuid, ConfigManagerErrorCode, ConfigManagerUserConfig, CreationClassName, Description, DeviceID, Manufacturer, Name, PNPDeviceID, Service, Status, SystemCreationClassName,
    SystemName FROM Win32_USBDevice; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Collection: 62/74 inventory data items successfully inventoried.
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Inventory: Collection Task completed in 21436.097 seconds
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Inventory: 12 Collection Task(s) failed. InventoryAgent
    18.03.2015 14:07:12 7836 (0x1E9C)
    Inventory: Temp report = C:\WINDOWS\CCM\Inventory\Temp\25bf01b2-12fc-4eea-8e97-a51b3c75ba50.xml
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Inventory: Starting reporting task. InventoryAgent
    18.03.2015 14:07:12 7552 (0x1D80)
    Reporting: 4381 report entries created. InventoryAgent
    18.03.2015 14:07:13 7552 (0x1D80)
    Inventory: Reporting Task completed in 1.030 seconds
    InventoryAgent 18.03.2015 14:07:13
    7552 (0x1D80)
    Inventory: Successfully sent report. Destination:mp:MP_HinvEndpoint, ID: {5541A94A-BED9-4132-AE54-110CB6896F02}, Timeout: 80640 minutes MsgMode: Signed, Not Encrypted
    InventoryAgent 18.03.2015 14:07:13
    7552 (0x1D80)
    Inventory: Cycle completed in 21453.570 seconds
    InventoryAgent 18.03.2015 14:07:30
    7552 (0x1D80)
    Inventory: Action completed. InventoryAgent
    18.03.2015 14:07:30 7552 (0x1D80)
    Inventory: ************************ End of message processing. ************************
    InventoryAgent 18.03.2015 14:07:30
    7552 (0x1D80)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, Caption, ClassGuid, ConfigManagerErrorCode, ConfigManagerUserConfig, CreationClassName, Description, DeviceID, Manufacturer, Name, PNPDeviceID, Service, Status, SystemCreationClassName,
    SystemName FROM Win32_USBDevice; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Collection: 62/74 inventory data items successfully inventoried.
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Inventory: Collection Task completed in 21436.097 seconds
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Inventory: 12 Collection Task(s) failed. InventoryAgent
    18.03.2015 14:07:12 7836 (0x1E9C)
    Inventory: Temp report = C:\WINDOWS\CCM\Inventory\Temp\25bf01b2-12fc-4eea-8e97-a51b3c75ba50.xml
    InventoryAgent 18.03.2015 14:07:12
    7836 (0x1E9C)
    Inventory: Starting reporting task. InventoryAgent
    18.03.2015 14:07:12 7552 (0x1D80)
    Reporting: 4381 report entries created. InventoryAgent
    18.03.2015 14:07:13 7552 (0x1D80)
    Inventory: Reporting Task completed in 1.030 seconds
    InventoryAgent 18.03.2015 14:07:13
    7552 (0x1D80)
    Inventory: Successfully sent report. Destination:mp:MP_HinvEndpoint, ID: {5541A94A-BED9-4132-AE54-110CB6896F02}, Timeout: 80640 minutes MsgMode: Signed, Not Encrypted
    InventoryAgent 18.03.2015 14:07:13
    7552 (0x1D80)
    Inventory: Cycle completed in 21453.570 seconds
    InventoryAgent 18.03.2015 14:07:30
    7552 (0x1D80)
    Inventory: Action completed. InventoryAgent
    18.03.2015 14:07:30 7552 (0x1D80)
    Inventory: ************************ End of message processing. ************************
    InventoryAgent 18.03.2015 14:07:30
    7552 (0x1D80)

    InventoryAgent 18.03.2015 08:10:03
    7836 (0x1E9C)
    Collection: Namespace = \\.\root\cimv2; Query = SELECT __CLASS, __PATH, __RELPATH, DefaultIPGateway, DHCPEnabled, DHCPServer, DNSDomain, DNSHostName, Index, IPAddress, IPEnabled, IPSubnet, MACAddress, ServiceName FROM Win32_NetworkAdapterConfiguration; Timeout
    = 600 secs. InventoryAgent
    18.03.2015 14:06:43 7836 (0x1E9C)
    Collection: Namespace = \\.\root\Nap; Query = SELECT __CLASS, __PATH, __RELPATH, description, fixupState, friendlyName, id, infoClsid, isBound, percentage, registrationDate, vendorName, version FROM NAP_SystemHealthAgent; Timeout = 600 secs.
    InventoryAgent 18.03.2015 14:06:43
    7836 (0x1E9C)
    Looks like something in one or both of those wmi queries.  it goes from 8:10:03 to 14:06:43 right around there.  6 hours to do that... 
    try running those queries from wbemtest manually; and see which one just never finishes.
    Standardize. Simplify. Automate.

  • Performance issue with high CPU and IO

    Hi guys,
    I am encountering huge user response time on a production system and I don’t know how to solve it.
    Doing some extra tests and using the instrumentation that we have in the code we concluded that the DB is the bottleneck.
    We generated some AWR reports and noticed the CPU was in top wait events. Also noticed that in a random manner some simple sql take a long time to execute. We activated the sql trace on the system and noticed that for very simple SQLs (unique index access on one table) we have huge exec times. 9s
    In the trace file the huge time we had it in fetch area: 9.1s cpu and elapsed 9.2.
    And no or very small waits for this specific SQL.
    it seems like the bottle neck is on the CPU but at that point there were very few processes running on the DB. Why can we have such a big cpu wait on a simple select? This is a machine with 128 cores. We have quicker responses on machines smaller/busier than this.
    We noticed that we had a huge db_cache_size (12G) and after we scale it down we noticed some improvements but not enough. How can I prove that there is a link between high CPU and big cache_size? (there was not wait involved in SQL execution). what can we do in the case we need big DB cache size?
    The second issue is that I tried to execute an sql on a big table (FTS on a big table. no join). Again on that smaller machine it runs in 30 seconds and on this machine it runs in 1038 seconds.
    Also generated a trace for this SQL on the problematic machine:
    call     count       cpu    elapsed       disk      query    current        rows
    Parse        1      0.00       0.00          0          0          0           0
    Execute      1      0.00       0.00          0          0          0           0
    Fetch        1    402.08    1038.31    1842916    6174343          0           1
    total        3    402.08    1038.32    1842916    6174343          0           1
      db file sequential read                     12419        0.21         40.02
      i/o slave wait                             135475        0.51        613.03
      db file scattered read                     135475        0.52        675.15
      log file switch completion                      5        0.06          0.18
      latch: In memory undo latch                     6        0.00          0.00
      latch: object queue header operation            1        0.00          0.00
    ********************************************************************************The high CPU is present here also but here I have huge wait on db file scattered read.
    Looking at the session with the select the AWG_wait for db scattered read was 0.5. on the other machine it is like 0.07.
    I though this is an IO issue. I did some IO tests at SO level and it seems like the read and writes operation are very fast…much faster than the machine that has the awg_wait smaller. Why the difference in waits?
    One difference between these two DBs is that the problem one has the db block size = 16k and the other one has 8k.
    I received some reports done at OS level on CPU and IO usage on the problematic machine (in normal operations). It seems like the CPU is very used and the IO stays very low.
    On the other machine, the smaller and the faster one, it is other way around.
    What is the problem here? How can I test further? Can I link the high CPU to low/slow IO?
    we have 10G on sun os with ASM.
    Thanks in advance.

    Yes, there are many things you can and should do to isolate this. But first check MOS Poor Performance With Oracle9i and 10g Releases When Using Dynamic Intimate Shared Memory (DISM) [ID 1018855.1] isn't messing you up to start.
    Also, be sure and post exact patch levels for both Oracle and OS.
    Be sure and check all your I/O settings and see what MOS has to say about those.
    Are you using ASSM? See Long running update
    Since it got a little better with shrinking the SGA size, that might indicate (wild speculation here, something like) one of the problems is simply too much thrashing within the SGA, as oracle decides "small" objects being full scanned in memory is faster than range scans (or whatever) from disk, overloading the cpu, not allowing the cpu to ask for other full scans from I/O. Possibly made worse by row level locking, or some other app issue that just does too much cpu.
    You probably have more than one thing wrong. High fetch count might mean you need to adjust the array size on the clients.
    Now that that is all out of the way, if you still haven't found the problem, go through http://oracle-randolf.blogspot.com/2009/02/basic-sql-statement-performance.html
    Edit: Oh, see Solaris 10 memory management conflicts with Automatic PGA Memory Management [ID 460424.1] too.
    Edited by: jgarry on Nov 15, 2011 1:45 PM

  • Since 10.4.6 Update: "ATSServer" with high CPU-usage when opening PDFs

    Since I've updated to 10.4.6 my system is very slow when I open PDFs (in "Preview" or "Acrobat").
    This is caused by the process "ATSServer" which goes through every file in "~/Library/Fonts" (I have over 4.800 files there - but just a view are activated by "FontBook") and so it takes very high CPU-usage.
    I didn't have this problem bevore the update and I didn't make any changes on my font settings.
    I've cleaned the font cache, but it doesn't help.
    Any ideas?
    Ciao
    Mephizo
    PS: Excuse my poor English.
    PowerBook G4 17'' 1.67 GHz   Mac OS X (10.4.6)  
    PowerBook G4 17'' 1.67 GHz   Mac OS X (10.4.6)  

    I haven't noticed any change myself, Dan, but I have the 2 GHz model.
    One thing I HAVE noticed though after OS updates is that they can often result in a fair amount of file and free space fragmentation immediately after installation. OSX's routines will deal with fragmentation of small files , but not of large ones (over 20 Meg), and it won't deal very effectively with free space fragmentation. If your HD is getting full this can cause significant slowdowns, especially where video is involved.
    You should also run DiskUtility and check for any directory and permissions issues.
    Cheers
    Rod

  • Having Issues with High CPU Usage with fluxbox & pekwm

    Greetings,
    I have searched for this issue and it seems that something similar has happened in the past with an old xorg, and it seems that someone is having a similar issue that they can point to kde as being the culrpit. However in my case, this is a fresh install and is not using any kde libs. Please allow me to explain the issue.
    The power supply on my main computer finally gave out, and being incredibly poor at the moment I can not yet replace it. So, I pull out an older computer that I had once set up for my kids, but took it away from them when they were abusing it. I always make backup dvds and such monthly so I didn't have to worry about any lose of data, however I desired to keep my larger hard drives from my main computer. After testing to ensure that the hard drives were fine, I did some minor surgery, and did a fresh install of arch linux onto my back up computer. Since I prefer pekwm, I installed pekwm as my wm of choice, and outfitted it quickly with mpd+sonata, firefox, pcmanfm, and tint2. Then I rebooted into my pekwm 'desktop'.
    It was running sluggish. Firefox was easily maxing the cpu up and beyond 100% and it felt ten times worse than any heavy DE I used in the past. I checked to make sure I had the right video driver installed. As this computer runs a 64MB nVidia GeForce2 MX with TV out video card, I searched nvidia's page and the arch wiki, noting that the driver needed is nvidia-96xx. Well I had already suspected that during install, and thus had installed such driver. I double checked my xorg.config and it has the right driver listed.
    So I checked out some lighter browsers. I tried both Midori and Iron (which is similar to Chrome but without google spyware or whatever). Well both run better than firefox, Midori being the lightest one, but Iron quickly being my favorite. Still, the problem remained. Moving windows caused cpu spikes, opening more than one tab, or more than one program caused cpu spikes and the computer to slow down and sputter, freezing at times.
    So I tried out another wm, fluxbos, which is another of my favorites. Seemed somewhat better but only fractionally, which I consider odd because pekwm has always seemed snappier than fluxbox to me in the past. Running lxtask (still mouse dependent, sorry lol), I was able to take note of the following...
    FLUXBOX
    lxtask 6% CPU usage average
    PCManFM 5% CPU Average
    Fluxbox 1% to 2% cpu usage
    gksu 3% cpu usage
    pidgin 3% to 7% cpu usage
    firefox up to 62% cpu usage
    midori up to 38% cpu usage
    iron up to 50% cpu usage
    mpd 11% cpu uage
    Xorg (with no window movement) 2% cpu usage
    Xorg (moving windows around) up to 80% cpu usage
    PEKWM
    lxtask 6% CPU
    pidgin 7% cpu
    tint2 1% cpu
    pekwm 1% to 5% cpu
    pcmanfm 7% cpu
    firefox up to 85% cpu
    python 1% cpu
    midori up to 38% cpu
    iron up to 25% cpu (odd...)
    Xorg (with no window movement) 1% cpu
    Xorg (when moving windows around) up to 80% cpu
    Both were using around 118 MB RAm and weren't yet touching swap. As I see it I am  thinking xorg  or video driver related, yet I already made sure that I had the correct video driver. Here's my Xorg.conf
    # nvidia-xconfig: X configuration file generated by nvidia-xconfig
    # nvidia-xconfig: version 1.0 (buildmeister@builder63) Thu Jun 25 18:57:07 PDT 2009
    Section "ServerLayout"
    Identifier "Layout0"
    Screen 0 "Screen0" 0 0
    InputDevice "Keyboard0" "CoreKeyboard"
    InputDevice "Mouse0" "CorePointer"
    EndSection
    Section "Files"
    EndSection
    Section "InputDevice"
    # generated from default
    Identifier "Mouse0"
    Driver "mouse"
    Option "Protocol" "auto"
    Option "Device" "/dev/psaux"
    Option "Emulate3Buttons" "no"
    Option "ZAxisMapping" "4 5"
    EndSection
    Section "InputDevice"
    # generated from default
    Identifier "Keyboard0"
    Driver "kbd"
    EndSection
    Section "Monitor"
    Identifier "Monitor0"
    VendorName "Unknown"
    ModelName "Unknown"
    HorizSync 30.0 - 110.0
    VertRefresh 50.0 - 150.0
    Option "DPMS"
    EndSection
    Section "Device"
    Identifier "Device0"
    Driver "nvidia"
    VendorName "NVIDIA Corporation"
    EndSection
    Section "Screen"
    Identifier "Screen0"
    Device "Device0"
    Monitor "Monitor0"
    DefaultDepth 24
    SubSection "Display"
    Depth 24
    Modes "1600x900" "1024x768" "800x600" "640x480"
    EndSubSection
    EndSection
    Here's some specs on this computer, including video cards and such...
    Computer
    Summary
    Computer
    Processor Intel(R) Pentium(R) 4 CPU 1.60GHz
    Memory 1034MB (239MB used)
    Operating System Arch Linux
    User Name mythus (Travis K. Randall)
    Date/Time Thu 08 Oct 2009 05:24:52 PM CDT
    Display
    Resolution 1600x900 pixels
    OpenGL Renderer GeForce2 MX/AGP/SSE2
    X11 Vendor The X.Org Foundation
    Multimedia
    Audio Adapter ICH - Intel 82801BA-ICH2
    Input Devices
    Macintosh mouse button emulation
    AT Translated Set 2 keyboard
    Power Button
    Power Button
    PC Speaker
    Logitech USB Optical Mouse
    Printers
    No printers found
    SCSI Disks
    ATA ST3160212A
    ATA IC35L090AVV207-0
    LITE-ON LTR-16102B
    TSSTcorp CD/DVDW TS-H552D
    Operating System
    Version
    Kernel Linux 2.6.30-ARCH (i686)
    Compiled #1 SMP PREEMPT Wed Sep 9 12:37:32 UTC 2009
    C Library GNU C Library version 2.10.1 (stable)
    Default C Compiler GNU C Compiler version 4.4.1 (GCC)
    Distribution Arch Linux
    Current Session
    Computer Name norova
    User Name mythus (Travis K. Randall)
    Home Directory /home/mythus
    Desktop Environment Unknown (Window Manager: Fluxbox)
    Misc
    Uptime 11 hours, 38 minutes
    Load Average 0.20, 0.38, 0.34
    Kernel Modules
    Loaded Modules
    ipv6 IPv6 protocol stack for Linux
    reiserfs ReiserFS journaled filesystem
    usbhid USB HID core driver
    hid
    arc4 ARC4 Cipher Algorithm
    ecb ECB block cipher algorithm
    snd_seq_dummy ALSA sequencer MIDI-through client
    rt2500pci Ralink RT2500 PCI & PCMCIA Wireless LAN driver.
    snd_seq_oss OSS-compatible sequencer module
    rt2x00pci rt2x00 pci library
    snd_seq_midi_event MIDI byte <-> sequencer event coder
    rt2x00lib rt2x00 library
    snd_seq Advanced Linux Sound Architecture sequencer.
    led_class LED Class Interface
    snd_seq_device ALSA sequencer device management
    input_polldev Generic implementation of a polled input device
    mac80211 IEEE 802.11 subsystem
    snd_intel8x0 Intel 82801AA,82901AB,i810,i820,i830,i840,i845,MX440; SiS 7012; Ali 5455
    uhci_hcd USB Universal Host Controller Interface driver
    snd_pcm_oss PCM OSS emulation for ALSA.
    fan ACPI Fan Driver
    cfg80211 wireless configuration support
    ppdev
    ehci_hcd USB 2.0 'Enhanced' Host Controller (EHCI) Driver
    snd_mixer_oss Mixer OSS emulation for ALSA.
    snd_ac97_codec Universal interface for Audio Codec '97
    nvidia
    lp
    eeprom_93cx6 EEPROM 93cx6 chip driver
    parport_pc PC-style parallel port driver
    ohci1394 Driver for PCI OHCI IEEE-1394 controllers
    parport
    psmouse PS/2 mouse driver
    ac97_bus
    ieee1394
    serio_raw Raw serio driver
    8139too RealTek RTL-8139 Fast Ethernet driver
    e100 Intel(R) PRO/100 Network Driver
    snd_pcm Midlevel PCM code for ALSA.
    pcspkr PC Speaker beeper driver
    battery ACPI Battery Driver
    8139cp RealTek RTL-8139C+ series 10/100 PCI Ethernet driver
    snd_timer ALSA timer interface
    i2c_core I2C-Bus main module
    iTCO_wdt Intel TCO WatchDog Timer Driver
    mii MII hardware support library
    evdev Input driver event char devices
    snd Advanced Linux Sound Architecture driver for soundcards.
    ac ACPI AC Adapter Driver
    iTCO_vendor_support Intel TCO Vendor Specific WatchDog Timer Driver Support
    usbcore
    soundcore Core sound module
    sg SCSI generic (sg) driver
    shpchp Standard Hot Plug PCI Controller Driver
    snd_page_alloc Memory allocator for ALSA system.
    processor ACPI Processor Driver
    thermal ACPI Thermal Zone Driver
    pci_hotplug PCI Hot Plug PCI Core
    intel_agp
    button ACPI Button Driver
    agpgart AGP GART driver
    rtc_cmos Driver for PC-style 'CMOS' RTCs
    rtc_core RTC class support
    rtc_lib
    ext4 Fourth Extended Filesystem
    mbcache Meta block cache (for extended attributes)
    jbd2
    crc16 CRC16 calculations
    sr_mod SCSI cdrom (sr) driver
    cdrom
    sd_mod SCSI disk (sd) driver
    ata_piix SCSI low-level driver for Intel PIIX/ICH ATA controllers
    ata_generic low-level driver for generic ATA
    pata_acpi SCSI low-level driver for ATA in ACPI mode
    libata Library module for ATA devices
    floppy
    scsi_mod SCSI core
    Display
    Display
    Display
    Resolution 1600x900 pixels
    Vendor The X.Org Foundation
    Version 1.6.3.901
    Monitors
    Monitor 0 1600x900 pixels
    Extensions
    BIG-REQUESTS
    Composite
    DAMAGE
    DOUBLE-BUFFER
    DPMS
    DRI2
    GLX
    Generic Event Extension
    MIT-SCREEN-SAVER
    MIT-SHM
    NV-CONTROL
    NV-GLX
    RANDR
    RECORD
    RENDER
    SECURITY
    SHAPE
    SYNC
    X-Resource
    XC-MISC
    XFIXES
    XFree86-DGA
    XFree86-VidModeExtension
    XINERAMA
    XInputExtension
    XKEYBOARD
    XTEST
    XVideo
    OpenGL
    Vendor NVIDIA Corporation
    Renderer GeForce2 MX/AGP/SSE2
    Version 1.5.8 NVIDIA 96.43.13
    Direct Rendering Yes
    Processor
    Processor
    Processor
    Name Intel(R) Pentium(R) 4 CPU 1.60GHz
    Family, model, stepping 15, 1, 2 (Pentium 4)
    Vendor Intel
    Configuration
    Cache Size 256kb
    Frequency 1594.77MHz
    BogoMIPS 3190.44
    Byte Order Little Endian
    Features
    FDIV Bug no
    HLT Bug no
    F00F Bug no
    Coma Bug no
    Has FPU yes
    Cache
    Cache information not available
    Capabilities
    fpu Floating Point Unit
    vme Virtual 86 Mode Extension
    de Debug Extensions - I/O breakpoints
    pse Page Size Extensions (4MB pages)
    tsc Time Stamp Counter and RDTSC instruction
    msr Model Specific Registers
    pae Physical Address Extensions
    mce Machine Check Architeture
    cx8 CMPXCHG8 instruction
    apic Advanced Programmable Interrupt Controller
    sep Fast System Call (SYSENTER/SYSEXIT)
    mtrr Memory Type Range Registers
    pge Page Global Enable
    mca Machine Check Architecture
    cmov Conditional Move instruction
    pat Page Attribute Table
    pse36 36bit Page Size Extensions
    clflush Cache Line Flush instruction
    dts Debug Store
    acpi Thermal Monitor and Software Controlled Clock
    mmx MMX technology
    fxsr FXSAVE and FXRSTOR instructions
    sse SSE instructions
    sse2 SSE2 (WNI) instructions
    ss Self Snoop
    ht HyperThreading
    tm Thermal Monitor
    up smp kernel running on up
    pebs Precise-Event Based Sampling
    bts Branch Trace Store
    Memory
    Memory
    Total Memory 1034084 kB
    Free Memory 94276 kB
    Buffers 40536 kB
    Cached 700112 kB
    Cached Swap 0 kB
    Active 170788 kB
    Inactive 726844 kB
    Active(anon) 74112 kB
    Inactive(anon) 88348 kB
    Active(file) 96676 kB
    Inactive(file) 638496 kB
    Unevictable 12 kB
    Mlocked 12 kB
    High Memory 139144 kB
    Free High Memory 252 kB
    Low Memory 894940 kB
    Free Low Memory 94024 kB
    Virtual Memory 2931852 kB
    Free Virtual Memory 2931852 kB
    Dirty 12 kB
    Writeback 0 kB
    AnonPages 156996 kB
    Mapped 57392 kB
    Slab 24260 kB
    SReclaimable 18864 kB
    SUnreclaim 5396 kB
    PageTables 1376 kB
    NFS_Unstable 0 kB
    Bounce 0 kB
    WritebackTmp 0 kB
    CommitLimit 3448892 kB
    Committed_AS 404212 kB
    VmallocTotal 122880 kB
    VmallocUsed 27648 kB
    VmallocChunk 52368 kB
    DirectMap4k 32760 kB
    DirectMap4M 876544 kB
    Benchmarks
    CPU Blowfish
    CPU Blowfish
    This Machine 1595 MHz 50.176
    Intel(R) Celeron(R) M processor 1.50GHz (null) 26.1876862
    PowerPC 740/750 (280.00MHz) (null) 172.816713
    CPU CryptoHash
    CPU CryptoHash
    This Machine 1595 MHz 11.071
    CPU Fibonacci
    CPU Fibonacci
    This Machine 1595 MHz 74.202
    Intel(R) Celeron(R) M processor 1.50GHz (null) 8.1375674
    PowerPC 740/750 (280.00MHz) (null) 58.07682
    CPU N-Queens
    CPU N-Queens
    This Machine 1595 MHz 132.704
    FPU FFT
    FPU FFT
    This Machine 1595 MHz 104.630
    FPU Raytracing
    FPU Raytracing
    This Machine 1595 MHz 212.639
    Intel(R) Celeron(R) M processor 1.50GHz (null) 40.8816714
    PowerPC 740/750 (280.00MHz) (null) 161.312647
    So what I am wanting help with is finding the solution to the cpu usage so that I can use this computer at least until I can afford a new power supply or whatever. I am open to any good suggestions, though I must state I am not really interested in tiling wm's at the moment. I am just not a true minimalist.
    Thanks in advance for any and all help. I understand that there is a lot of info in this post, but it is my hope that with this info the problem can be solved. If there is info I didn't provide that is needed, please let me know.

    also check that you have
    session.screen0.opaqueMove false
    in your .fluxbox/init  this will probably help a bit if it is currently true,
    Last edited by Cyrusm (2009-01-26 13:52:13)

  • High CPU usage with Coherence 3.5.0 cluster node

    We're noticing sustained high CPU usage on one of our Coherence nodes - this happens after running a job that pretty intensively interacts with the cache. The jobs complete, however we still see the Coherence process using approximately the equivalent of one core's worth of CPU. I took a couple of thread dumps about fifteen minutes apart, and the only noticeable differences are within the PacketListener1 and PacketListenerN threads - they're holding locks on different DatagramPacket instances, which suggests to be that this may be the piece of code that is looping.
    There are two other cache nodes in this particular cluster - both of them on a different machine - and neither of them are exhibiting the same CPU utilization.
    Environment:
    Coherence 3.5.0
    Java - BEA JRockit(R) (build R27.6.3-40_o-112056-1.6.0_11-20090318-2103-linux-x86_64, compiled mode)
    Has anyone encountered this scenario before?
    Here's a copy of the two thread dumps:
    Thread Dump 1 -
    ===== FULL THREAD DUMP ===============
    Thu Feb 24 21:45:00 2011
    BEA JRockit(R) R27.6.3-40_o-112056-1.6.0_11-20090318-2103-linux-x86_64
    "Main Thread" id=1 idx=0x4 tid=18143 prio=5 alive, in native, waiting
    -- Waiting for notification on: java/lang/Class@0x43587b58[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/net/DefaultCacheServer.main(DefaultCacheServer.java:80)
    ^-- Lock released while waiting: java/lang/Class@0x43587b58[fat lock]
    at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.doWork(ZzzzCoherenceServerStartup.java:29)
    at com/zzzz/util/runner/ZzzzRunnerBase.run(ZzzzRunnerBase.java:23)
    at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.main(ZzzzCoherenceServerStartup.java:16)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "(Signal Handler)" id=2 idx=0x8 tid=18144 prio=5 alive, in native, daemon
    "(GC Main Thread)" id=3 idx=0xc tid=18145 prio=5 alive, in native, daemon
    "(GC Worker Thread 1)" id=? idx=0x10 tid=18146 prio=5 alive, in native, daemon
    "(GC Worker Thread 2)" id=? idx=0x14 tid=18147 prio=5 alive, in native, daemon
    "(GC Worker Thread 3)" id=? idx=0x18 tid=18148 prio=5 alive, in native, daemon
    "(GC Worker Thread 4)" id=? idx=0x1c tid=18149 prio=5 alive, in native, daemon
    "(Code Generation Thread 1)" id=4 idx=0x20 tid=18150 prio=5 alive, in native, native_waiting, daemon
    "(Code Optimization Thread 1)" id=5 idx=0x24 tid=18151 prio=5 alive, in native, native_waiting, daemon
    "(VM Periodic Task)" id=6 idx=0x28 tid=18152 prio=10 alive, in native, daemon
    "Finalizer" id=7 idx=0x2c tid=18153 prio=8 alive, in native, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
        at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
        at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
        at java/lang/Thread.run(Thread.java:619)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "Reference Handler" id=8 idx=0x30 tid=18154 prio=10 alive, in native, native_waiting, daemon
        at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
        at java/lang/ref/Reference.access$100(Reference.java:11)
        at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "(Sensor Event Thread)" id=9 idx=0x34 tid=18155 prio=5 alive, in native, daemon
    "Logger@9217551 3.5/459" id=12 idx=0x38 tid=18158 prio=3 alive, in native, waiting, daemon
        -- Waiting for notification on: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketListener1" id=14 idx=0x3c tid=18159 prio=8 alive, in native, daemon
    at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[recursive]
    at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
    at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[thin lock]
    ^-- Holding lock: java/net/DatagramPacket@0x73aa4e80[thin lock]
    ^-- Holding lock: java/net/DatagramSocket@0x4cb21338[thin lock]
    at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
    at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketReceiver" id=15 idx=0x40 tid=18160 prio=7 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketReceiver.onWait(PacketReceiver.CDB:2)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketPublisher" id=16 idx=0x44 tid=18161 prio=6 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketPublisher.onWait(PacketPublisher.CDB:2)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketSpeaker" id=17 idx=0x48 tid=18162 prio=8 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/queue/ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
    at com/tangosol/coherence/component/util/queue/ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
    at com/tangosol/coherence/component/util/Queue.remove(Queue.CDB:1)
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketListenerN" id=18 idx=0x4c tid=18163 prio=8 alive, in native, daemon
    at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[recursive]
    at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
    at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[thin lock]
    ^-- Holding lock: java/net/DatagramPacket@0x73e59890[thin lock]
    ^-- Holding lock: java/net/MulticastSocket@0x4cb22178[thin lock]
    at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
    at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "Cluster|Member(Id=2, Timestamp=2011-02-23 16:25:36.488, Address=10.10.100.77:8095, MachineId=25677, Location=site:emtzzzz.com,machine:njdev04,process:18142,member:cldev201~carboncachelauncher~181)" id=19 idx=0x50 tid=18164 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "TcpRingListener" id=20 idx=0x54 tid=18165 prio=6 alive, in native, daemon
    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
    ^-- Holding lock: java/net/SocksSocketImpl@0x4cb227d8[thin lock]
    at java/net/ServerSocket.implAccept(ServerSocket.java:453)
    at java/net/ServerSocket.accept(ServerSocket.java:421)
    at com/tangosol/coherence/component/net/socket/TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
    at com/tangosol/coherence/component/util/daemon/TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
    at com/tangosol/coherence/component/util/daemon/TcpRingListener.onNotify(TcpRingListener.CDB:9)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "Invocation:Management:EventDispatcher" id=22 idx=0x58 tid=18166 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher.onWait(Service.CDB:7)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "Invocation:Management" id=23 idx=0x5c tid=18167 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "DistributedCache" id=25 idx=0x60 tid=18168 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    ===== END OF THREAD DUMP ===============
    Thread Dump 2 -
    ===== FULL THREAD DUMP ===============
    Thu Feb 24 21:27:53 2011
    BEA JRockit(R) R27.6.3-40_o-112056-1.6.0_11-20090318-2103-linux-x86_64
    "Main Thread" id=1 idx=0x4 tid=18143 prio=5 alive, in native, waiting
    -- Waiting for notification on: java/lang/Class@0x43587b58[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/net/DefaultCacheServer.main(DefaultCacheServer.java:80)
    ^-- Lock released while waiting: java/lang/Class@0x43587b58[fat lock]
    at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.doWork(ZzzzCoherenceServerStartup.java:29)
    at com/zzzz/util/runner/ZzzzRunnerBase.run(ZzzzRunnerBase.java:23)
    at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.main(ZzzzCoherenceServerStartup.java:16)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "(Signal Handler)" id=2 idx=0x8 tid=18144 prio=5 alive, in native, daemon
    "(GC Main Thread)" id=3 idx=0xc tid=18145 prio=5 alive, in native, native_waiting, daemon
    "(GC Worker Thread 1)" id=? idx=0x10 tid=18146 prio=5 alive, in native, daemon
    "(GC Worker Thread 2)" id=? idx=0x14 tid=18147 prio=5 alive, in native, daemon
    "(GC Worker Thread 3)" id=? idx=0x18 tid=18148 prio=5 alive, in native, daemon
    "(GC Worker Thread 4)" id=? idx=0x1c tid=18149 prio=5 alive, in native, daemon
    "(Code Generation Thread 1)" id=4 idx=0x20 tid=18150 prio=5 alive, in native, native_waiting, daemon
    "(Code Optimization Thread 1)" id=5 idx=0x24 tid=18151 prio=5 alive, in native, native_waiting, daemon
    "(VM Periodic Task)" id=6 idx=0x28 tid=18152 prio=10 alive, in native, daemon
    "Finalizer" id=7 idx=0x2c tid=18153 prio=8 alive, in native, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
        at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
        at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
        at java/lang/Thread.run(Thread.java:619)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "Reference Handler" id=8 idx=0x30 tid=18154 prio=10 alive, in native, native_waiting, daemon
        at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
        at java/lang/ref/Reference.access$100(Reference.java:11)
        at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "(Sensor Event Thread)" id=9 idx=0x34 tid=18155 prio=5 alive, in native, daemon
    "Logger@9217551 3.5/459" id=12 idx=0x38 tid=18158 prio=3 alive, in native, waiting, daemon
        -- Waiting for notification on: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketListener1" id=14 idx=0x3c tid=18159 prio=8 alive, in native, daemon
    at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[recursive]
    at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
    at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[thin lock]
    ^-- Holding lock: java/net/DatagramPacket@0x73a80930[thin lock]
    ^-- Holding lock: java/net/DatagramSocket@0x4cb21338[thin lock]
    at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
    at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketReceiver" id=15 idx=0x40 tid=18160 prio=7 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketReceiver.onWait(PacketReceiver.CDB:2)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketPublisher" id=16 idx=0x44 tid=18161 prio=6 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketPublisher.onWait(PacketPublisher.CDB:2)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketSpeaker" id=17 idx=0x48 tid=18162 prio=8 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/queue/ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
    ^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
    at com/tangosol/coherence/component/util/queue/ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
    at com/tangosol/coherence/component/util/Queue.remove(Queue.CDB:1)
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "PacketListenerN" id=18 idx=0x4c tid=18163 prio=8 alive, in native, daemon
    at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[recursive]
    at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
    at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
    ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[thin lock]
    ^-- Holding lock: java/net/DatagramPacket@0x5e55a240[thin lock]
    ^-- Holding lock: java/net/MulticastSocket@0x4cb22178[thin lock]
    at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
    at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "Cluster|Member(Id=2, Timestamp=2011-02-23 16:25:36.488, Address=10.10.100.77:8095, MachineId=25677, Location=site:emtzzzz.com,machine:njdev04,process:18142,member:cldev201~carboncachelauncher~181)" id=19 idx=0x50 tid=18164 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "TcpRingListener" id=20 idx=0x54 tid=18165 prio=6 alive, in native, daemon
    at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
    at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
    ^-- Holding lock: java/net/SocksSocketImpl@0x4cb227d8[thin lock]
    at java/net/ServerSocket.implAccept(ServerSocket.java:453)
    at java/net/ServerSocket.accept(ServerSocket.java:421)
    at com/tangosol/coherence/component/net/socket/TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
    at com/tangosol/coherence/component/util/daemon/TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
    at com/tangosol/coherence/component/util/daemon/TcpRingListener.onNotify(TcpRingListener.CDB:9)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "Invocation:Management:EventDispatcher" id=22 idx=0x58 tid=18166 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher.onWait(Service.CDB:7)
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "Invocation:Management" id=23 idx=0x5c tid=18167 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    "DistributedCache" id=25 idx=0x60 tid=18168 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)[optimized]
    at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
    at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
    ^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
    at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace
    ===== END OF THREAD DUMP ===============

    Charlie, Cameron,
    Thanks for following up. To answer Cameron's question - yes, running top shows that it is this process using a significant amount of the CPU. I captured a JRA this morning setting nativesamples=true.
    I see the following when I look at the Hot Methods - let me know if there is something else that would shed some more light on this issue.
    Method     Percent(%)     #Samples     Optimized     Native     Method Id.
    libjvm.so#mmDetGcFollowReferencesWorkers     28.575     2,062     0     1     0x2AE6B82EC2C1
    libpthread.so.0#__recvfrom_nocancel     26.012     1,877     0     1     0x3CC1A0D689
    libpthread.so.0#__accept_nocancel     26.012     1,877     0     1     0x3CC1A0D4A9
    libjvm.so#mmBalanceGetReference     4.975     359     0     1     0x2AE6B817A65B
    libjvm.so#objIterStepFieldFromBitmaskWord     4.753     343     0     1     0x2AE6B82A6B34
    libjvm.so#objIterStep     3.88     280     0     1     0x2AE6B82A6F8D
    libjvm.so#objIterInitInner     2.744     198     0     1     0x2AE6B82A71B7
    libjvm.so#mmBalanceStoreReference     1.414     102     0     1     0x2AE6B817A781
    libjvm.so#objIterSetupFields     0.637     46     0     1     0x2AE6B82A6B08
    libjvm.so#nativeInnerLockNotLinked     0.236     17     0     1     0x2AE6B829DE76
    libjvm.so#mmWorkPacketPoolGet     0.125     9     0     1     0x2AE6B817A4F5
    libjvm.so#qBitSetIsBitSetInWord     0.069     5     0     1     0x2AE6B82C8AA6
    libjvm.so#nativeInnerUnlockNotLinked     0.069     5     0     1     0x2AE6B829E028
    libjvm.so#mmWorkPacketPoolAdd     0.042     3     0     1     0x2AE6B817A48E
    libjvm.so#objPoolMarkAllHard     0.042     3     0     1     0x2AE6B82A4923
    libjvm.so#objIterSetupArray     0.028     2     0     1     0x2AE6B82A6BDB
    libjvm.so#markAcquired     0.028     2     0     1     0x2AE6B829DBD3
    libjvm.so#mmPointerMatrixTryInsertAtIndex     0.028     2     0     1     0x2AE6B82BEFEB
    libjvm.so#mmPointerMatrixClear     0.028     2     0     1     0x2AE6B82BF61C
    libjvm.so#objPoolMarkWeakConcurrently     0.028     2     0     1     0x2AE6B82A5358
    libjvm.so#qBitSetFindFirstNonClearedWord     0.014     1     0     1     0x2AE6B82C888E
    libjvm.so#utilCounterToNanos     0.014     1     0     1     0x2AE6B8313B53
    libjvm.so#waitForLockIfNeededAndMarkAcquired     0.014     1     0     1     0x2AE6B829DC30
    libjvm.so#vmtLinkData     0.014     1     0     1     0x2AE6B8325B2E
    libjvm.so#mmBitsIsObjectMarkedGrey     0.014     1     0     1     0x2AE6B82ED7E9
    libjvm.so#qBitSetFindLastSetBitInWord     0.014     1     0     1     0x2AE6B82C8854
    libjvm.so#mark_writebarriers     0.014     1     0     1     0x2AE6B8250EEA
    libjvm.so#signalNextInLockQueueIfNeeded     0.014     1     0     1     0x2AE6B829DF52
    libjvm.so#mmGetUsingMatrixes     0.014     1     0     1     0x2AE6B819F33C
    libjvm.so#ptGetThreadId     0.014     1     0     1     0x2AE6B82BD9ED
    libc.so.6#memset     0.014     1     0     1     0x3CC0A7A000
    libjvm.so#setupNodeForSelf     0.014     1     0     1     0x2AE6B829DD87
    libc.so.6#_int_free     0.014     1     0     1     0x3CC0A714E0
    libjvm.so#mmAddChunkToList     0.014     1     0     1     0x2AE6B830977C
    libjvm.so#vmtiUnlinkData     0.014     1     0     1     0x2AE6B8325B4A
    libjvm.so#nativeLockInSuspendCritical     0.014     1     0     1     0x2AE6B829E0E3
    libjvm.so#mmSweepHeapPart     0.014     1     0     1     0x2AE6B830989A
    libjvm.so#mmBalanceWorkSetSwapPackets     0.014     1     0     1     0x2AE6B817A647
    libc.so.6#_int_malloc     0.014     1     0     1     0x3CC0A71E80
    libjvm.so#charToJlcType     0.014     1     0     1     0x2AE6B8253633
    unknown#unknown functions     0     0     0     1     0x2AE6B8597090
    Running with nativemethods=false, I see the following under hot methods
    Method     Percent(%)     #Samples     Optimized     Native     Method Id.
    jrockit.vm.Locks.monitorExitSecondStage(Object)     50     1     1     0     0x161D30D0
    com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify()     50     1     1     0     0x169D2C70
    com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onNotify()     0     0     1     0     0x1551C050
    com.tangosol.coherence.component.util.queue.ConcurrentQueue.checkFlush(int)     0     0     1     0     0x151C2180
    java.lang.Thread.run()     0     0     0     0     0x14E98EB0
    jrockit.vm.RNI.c2java(long, long, long, long, long)     0     0     0     0     0x14C07830
    unknown#unknown functions     0     0     0     1     0x2AE6B8597090
    com.tangosol.coherence.component.util.Daemon.run()     0     0     0     0     0x15CFEBC0
    com.tangosol.coherence.component.util.queue.ConcurrentQueue.add(Object)     0     0     1     0     0x15ED6B00
    com.tangosol.coherence.component.util.queue.ConcurrentQueue.onAddElement()     0     0     1     0     0x151C9C20
    com.tangosol.coherence.component.util.queue.ConcurrentQueue.flush(boolean)     0     0     1     0     0x1618CAD0
    Thanks,
    Jason

Maybe you are looking for