Performance degrading CPU utilization 100%

Hello,
RHEL 4
Oracle 10.2.0.4
Attached to a DAS (partition is 91% full) RAID 5
Over the past few weeks my production database performance has majorly degraded. I have not made any application, OS, or database changes (I was on vacation!). I have started troubleshooting, but need some more tips as to what else I can check.
My users run a query against the database, and for a table with only 40,000 rows, it will take about 2 minutes before the results return. For a table with 12 million records, it takes about 10 minutes or more for the query to complete. If I run a script that counts/displays a total record count for each table in the database as well as a total count of all records in the database (~15,000,000 records total), the script either takes about 45 minutes to complete or sometimes it just never completes. The Linux partition on my DAS is currently 91% full. I do not have Flashback or auditing enabled.
These are some things I tried/observed:
I shut down all applications/servers/connections to the database and then restarted the database. After starting the database, I monitored the DAS interface, and the CPU utilization spiked to 100% and never goes down, even with no users/application trying to connect to the database. The alert.log file contains these errors:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code arguments: [ttcdrv-recursivecall]
ORA-03135: connection lost contact
ORA-06512: at "CTXSYS.SYNCRN", line 1
The database still starts, but the performance is bad. From the error above and after checking performance in EM, I see there are a lot of sync index jobs running by each of the schemas and the db sequential file read is high. There is a job to resync the indexes every 5 minutes. I am going to try disabling these jobs tihs afternoon to see what happens with the CPU utilization. If it helps, I will try adjusting the job from running every 5 minutes to something like every 30 minutes. Is there a way to defrag the CONTEXT indexes? REBUILD?
I'm not sure if I am running down the right path or not. Does anyone have any other suggestions as to what I can check? My SGA_TARGET is currently set to 880M and the SGA_MAX_SIZE is 2032M. Would it also help for me to increase the SGA_TARGET to the SGA_MAX_SIZE; thus increasing the amount of space allocated to the buffer cache? I have ASMM enabled and currently this is what is allocated:
Shared Pool = 18.2%
Buffer Cache = 61.8%
Large Pool = 16.4%
Java Pool = 1.8%
Other = 1.8%
I also ran ADDM and these were the results of my Performance Analysis:
34.7% The throughput of the I/O subsystem was significantly lower than expected (when I clicked on this it said to either implement ASM or stripe using SAME methodology...we are already using RAID5)
31% SQL statements consuming significant database time were found (I cannot make application code changes, and my database consists entirely of INSERT statements...there are never any deletes or updates. I see that the updates that are being made were by the index resyncing job to the various DR$ tables)
18% Individual database segments responsible for significant user I/O wait were found
15.9% Individual SQL statements responsible for significant user I/O wait were found
8.4% PL/SQL execution consumed significant database time
I also recently ran a SHRINK on all possible tablespace as recommended in EM, but that did not seem to help either.
Please let me know if I can provide any other pertinent information to solve the poor I/O problem. I am leaning toward thinking it has to do with the index sync job stepping on itself...the job cannot complete in 5 minutes before it tries to kick off again...but I could be completely wrong! What else can I check to figure out why I have 100% CPU utilization, with no users/applications connected? Thank you!
Mimi
Edited by: Mimi Miami on Jul 25, 2009 10:22 AM

Tables/Indexes last analyzed today.
I figured out that it was the Oracle Text indexes synching to frequently that was causing the problem. I disabled all the jobs that kicked off those indexes and my CPU utilization dropped to almost 0%. I will work on tuning the interval/re-enabling the indexes for my dynamic datasources.
Thank you for everyone's suggestions!
Mimi

Similar Messages

ASA 5545 NIPS CPU utilization 100%

Hi,
We are having two pair of NIPS SSP and working in Active and standby mode.
All four Devices (ASA 5545 and ASA 5525 SSP IPS) showing CPU utilization 100% on both active and standby devices.
Memory usage is also showing very highly utilized.
Kindly help me resovle this issue.
Regards,
Dheeraj

Hi Dheeraj,
I don't think this is an issue. CPU 100% utilization is reported on every IPS (appliance or ASA software module) and it should be normal.
If you want to see if your device is getting more traffic than it can process, you should look at inspection load.
I hope this will help and it would be nice if someone from Cisco team could reply to this.

JVM 1.4.x IE 6 CPU utilization 100% when issue window.print

I am using JVM 1.4.2_04, IE 6 and WinXP Pro.
When visiting internet application having applet initialized, and click the PRINT link (window.print()), suppose the print dialog box will be shown, however, the CPU utilization draw up to 100% and the dialog cannot be shown over 10 minutes.
I had tried with 1.4.1_x, the same result found.
If using 1.3.1_x on the same application, no problem found, can anyone help.

case solved, which is due to the bug_id 4942346

Monitoring IP Gateway Activity hangs workstation and rise Servers CPU utilization 100%

Hi..
I have nw5.1 sp6 and BM 3.7 sp3 with last patches..installed with C.
Johnson's BM Book.
used services/proxies HTTP, FTP ,Generic TCP and Socks
Only one acces rule with indexed logging for denyed URL-s..
all works well except socks gateway sometimes. Sometimes, sometimes with no
reason, if you look real time activity of IP Gateway
(Nwadmin-tools-Bordermanager-IP Gateway) workstation hangs and server CPU
utilization rise to 100%. and only way reboot workstation and reload
proxy.nlm on server..
By the way the same is for Proxy cache monitor...
is any ideas ?
Thanks,
Lab.

May be.. Actually, i have pervasive upgraded to sp4 + postfixes
for other reasons and have some bti.cfg settings changed..
Thanks, i will look in this way..
Lab
BTW, what does -c parameter mean?
> I suspect your btrieve database hasn't been loaded with the optimal
> parameters.
> When loading btrieve make sure that you use the -u=1 -c parameters.
> >
>
>
> --
> Cat
> NSC Volunteer Sysop

CPU Utilization 100%

Hi,
I have developed an application to view images from an IP camera. By this I can simualtaneously view images from about 25 cameras. The problem is that the CPU Usage increases as the no of player increases. My Player is JPanel. which continuously paints the images from camera. The method 'paintImage' is called from another thread's run method. This thread is responsible for taking jpeg images from IP camera.
Here is the code for this.
public void paintImage(Image image, int fps) {
     try {
          int width = this.getWidth();
          addToBuffer(image);
          currentImage = image;
          Graphics graphics = this.getGraphics();
          if (isRunning && graphics != null) {
               graphics.drawImage(image, 0, 0, getWidth(), getHeight(), this);
               if(border ==true){
                    graphics.setColor(Color.RED);
          graphics.drawRect(0,0,getWidth()-1, getHeight()-1);
               graphics.setColor(Color.white);
               graphics.setFont(new Font("verdana", Font.ITALIC, 12));
               graphics.drawString("FPS : " + fps, width-60, 13);
               this.fps = fps;
               if (isRandomRecord) {
                    graphics.setColor(new Color(0, 255, 0));
                    graphics.fillArc((getWidth() - 10), 5, 10, 10, 0, 360);
     } catch (Exception e) {
          e.printStackTrace();
Can someone please help me to solve this problem so that the CPU usage can be reduced.

   Yes, this is expected.
Since Web/Business tier databases do not have any guaranteed DTUs/resource limits associated with them, we normalize the percentage values in terms of the amount of resources available to a database in S2 performance level. DTU and other resource (CPU/IO/Log
etc) consumption information in terms of an S2 database level will allow you to normalize the current consumption of your Web/Business databases in terms of new tier databases and see where they fit better for migration as needed. Since Web
and Business databases run in a shared pool (i.e. sharing resources with other customer loads in the same machine), the resource availability can vary based on who else is running on the same machine. If there are not many significant other active loads
on the machine where your database is running, it is possible for you to get significantly more than 100% of an S2 performance level. Similarly, it is possible to see resource availability to fall much lower than 100% for an active database, if there are many
other active workloads are running on the same machine.

IE performance issue (CPU rate = 100%, IE CPU rate 90%)

When I open a report in active studio or active viewer or IE, the IE cpu rate > 90%. The system becomes very slow.
The report contains 4 views, containing 1 line chart view (3 lines, time limit within 1 minute, 1 new row every 1 second).
CPU is 2GHZ, mem is 2G.
Any one knows how to improve the performance of IE? Thanks.

Hi Leon,
What version of BAM are you using? What OS? What version of IE?
What else is running on your machine? Are all the BAM servers on this machine? Oracle Database? Any other products? How much memory is still free at the time this occurs?
Regards, Stephen

Xcode 4.4 cpu utilization 100%

Since upgrading to XCode 4.4, I am seeing it use > 100% CPU. I've taken a sample, most threads appear to be spending almost all their time in mach_msg_trap (partial sample below). Known issue? Other than editing some rather simple C++ code, xcode is downloading and installing libraries, not an activity that I'd expect to spin my fans up to top speed.
Is this a known issue? Is there a workaround?
Sampling process 39159 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling Xcode (pid 39159) every 1 millisecond
Process:         Xcode [39159]
Path:            /Applications/Xcode.app/Contents/MacOS/Xcode
Load Address:    0x101ca9000
Identifier:      com.apple.dt.Xcode
Version:         4.4 (1484)
Build Info:      IDEApplication-1484000000000000~4
Code Type:       X86-64 (Native)
Parent Process: launchd [106]
Date/Time:       2012-07-28 14:42:37.585 +0200
OS Version:      Mac OS X 10.7.4 (11E53)
Report Version: 7
Call graph:
    2121 Thread_804492   DispatchQueue_1: com.apple.main-thread (serial)
    + 2121 ??? (in Xcode) load address 0x101ca9000 + 0x1dc0 [0x101caadc0]
    +   2121 NSApplicationMain (in AppKit) + 867 [0x7fff87516244]
    +     2121 -[NSApplication run] (in AppKit) + 470 [0x7fff8729a071]
    +       2121 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] (in AppKit) + 135 [0x7fff8729d735]
    +         2121 _DPSNextEvent (in AppKit) + 659 [0x7fff8729de31]
    +           2121 BlockUntilNextEventMatchingListInMode (in HIToolbox) + 62 [0x7fff8c12160e]
    +             2121 ReceiveNextEventCommon (in HIToolbox) + 355 [0x7fff8c121781]
    +               2121 RunCurrentEventLoopInMode (in HIToolbox) + 277 [0x7fff8c11a4d3]
    +                 2119 CFRunLoopRunSpecific (in CoreFoundation) + 230 [0x7fff8f6ac486]
    +                 ! 2100 __CFRunLoopRun (in CoreFoundation) + 1204 [0x7fff8f6acc74]
    +                 ! : 2100 __CFRunLoopServiceMachPort (in CoreFoundation) + 188 [0x7fff8f6a450c]
    +                 ! :   2100 mach_msg (in libsystem_kernel.dylib) + 73 [0x7fff8976fd71]
    +                 ! :     2100 mach_msg_trap (in libsystem_kernel.dylib) + 10 [0x7fff8977067a]

I believe the NSLog is a different problem.
NSLog printing of a very large string can cause a beachball with XCode 4.4 eating up a very high percentage of CPU Cycles. (For example see ... http://www.openradar.me/11972490) ... I experience this on my old MacMini running XCode 4.4
I have a new MacBook air and I am seeing the high CPU usage causing my fans to spin up so high that I can hear them on my normally silent machine, but with no beachball. Quitting Xcode and reopening resolves the issue on my MaxBook Air.

IWS 6.0 100% CPU utilization hanging- very urgent

Hi,
We are using Iplanet Web server 6.0 on windows-2000 SP2.The problem we are facing is after 10 concurrent users have logged in the CPU utilization shoots up to 100% and we have to reboot the systesm
Our billing Application is affected very much due to this.
Can anybody throw some light on this?
Thanks in advance.

Hi,
Are you using any plugin with iWS. Please let me know your config file. Mean while please check tunning parameters of solaris for Performance bench mark.
http://docs.iplanet.com/docs/manuals/enterprise/50/tuning/perf6.htm#17580
Regards,
Dakshin.

Load testing and 100% CPU Utilization - Multiple JVMs?

Hi,
Problem:
While stress testing the application with 20 simultaneous users, unix is 100% utilized and there is degradation in response times.
One component of the application is invoking a unix script which is executing a java program after setting CLASSPATH, PATH etc. I guess thus each execution will be invoking a jvm. At the end of the program it will do a System.Exit(). This java program generates around 15-60 barcode images. It takes 2 secs normally.
When, 5 and 10 concurrent users executing this unix script for a steady state time of 30 minutes is also working fine even through response time degrades to 4 secs.
With 20 concurrent users, unix box with 4 CPUs is 100% utilized and response time degrades to 11 secs or so.
We need to do this through unix script only because it is executed from database tier (oracle reports - rdfs) and java stored procs doesn't allow awt operations (image file creation).
I repeated this testing using a java class which just loops for 2 secs. With 20 concurrent users, it was also degrading to 6 secs or so and fluctuating between 100% CPU utilization.
Any pointers on what we should be analyzing more and how should we try to solve this issue.
Thanks,
Ayyappa

Stupid forums made me change the ``screen name.'' Whatever...
Anyway, it is resource intensive to invoke a new JVM per request. You can do several things but it comes down to the fact you will want to keep ONE JVM running in the background awaiting requests to do work. These requests can come from several mechanisms, such as polling a database table, sending requests over a local TCP socket, listening in on a Unix fifo file, etc, etc.
When the shell script is exec'd, connect to your JVM (for example) by opening a socket to it, enter your request for work, and wait for the response, the JVM app will listen and accept socket requests, thread off and process and then return data. Something in this form will be substantially more scalable that what is currently being done.

SDK-based Management Pack causes 100% CPU utilization for HealtService.exe

We developed an MP that uses SDK to create SCOM objects and insert a large number of performance counters ~ 2000 every 5 minute interval.
On SCOM side the MP has about 2000 instances of UnitMonitor (only ~20 per class but there are lots of actual objects). In these monitors we use
DataSource based on Microsoft.SystemCenter.TargetEntitySdkPerformanceDataProvider. All worked fine for a year but lately added a bunch of new objects/counters and CPU utilization for the MonitoringHost.exe started to spike
to 100% a few seconds after the Performance Counters were posted via SDK. The spike lasts for up to 3 minutes. No DB or Network spikes observed. We suspect SCOM does not deal efficiently with SDK Performance data - as if a separate SdkPerformanceDataProvider
is started for every UnitMonitor when the counters are posted rather than having cooked-down - one per Target instance.
Can anyone shed some light on this? I suspect only the Microsoft engineers would know.
Our environment is like this:
SCOM 2012 R2 UR4
Windows Server 2012 8 Core, 16 GB Ram
Thanks,
Dave

Hi Dave,
is there a way that you can reduce the number of performance insertions? Regardless of even fixing the performance issue, the impact from a database standpoint may be something to consider in the long term. My point is that fixing one issue, will take us
to the next one, etc etc.
I would reduce the number of insertions, only to the either insertion of the "most critical" or "top #". Also, aren't those counters already captured out of the box?
hth
Jose

100% CPU utilization on svchost.exe or Automatic Updates service

We have upgraded from WSUS 2.0 to 3.0 SP1 and now have few Windows XP SP2 PCs that are extremely slow because the CPU is at 100% utilization running a process called "svchost.exe." If I go into services and stop and disable the "Automatic Updates" service the CPU drop to normal almost instantly. I tried forcing a reinstall of the Windows Update Agent. After I enable the "Automatic Updates" service the machine works fine for a day, than after a reboot it goes back to 100% CPU utilization. We need this fixed so we can get these computers updates.

Hi Ryan / Folks,
NO CA Products here - but I have had the same issues with Microsoft updates!
Here's the install path I used during my experience :
Cold install of XP with SP1 on the PC (Full factory system restore/rebuild).
Acer - Semperon 1.8GHz + 1GB RAM - 8Mb ADSL connection to Internet
XP SP2
Windows updates OK - used to install IE7
Reason - I found IE is compromised if you go straight to XP SP3
XP SP3
Next Office 2003 Pro
Switch to Microsoft updates - Custom Updates - SVCHost issue - Still checking for updates after 15 minutes
Switch back to Windows updates - Custom Updates - NO SVCHost Issue - Checking complete after 2 minutes
Tried both Microsoft fixes mentioned above
http://support.microsoft.com/kb/927891
Same report back - i.e. SP3 newer etc.
http://support.microsoft.com/kb/943144 - Method 2
Seems to install ok
Reboot PC
Swich back to manual Microsoft Updates and all is not really rosey as the initial "checking updates" scan can take at least 5 minutes with SVCHost at better than 90% CPU usage. So I believe the issue is not fixed, but it is just about useable.
Interestingly, no issues on my work LAN where I am the systems manager - 20 PCs and 8 servers using WSUS. All units are up to date and no SVCHost issues.
So.... No fix yet here, however my solution is as follows:
On the problematic PC I decided to switch back to Automatic Windows Updates. This keeps the PC up to date with all Operating System patches. Performance is not affected. I have decided that I will manually switch to the Microsoft update system once every couple of weeks or so to catch the updates for Office etc. I'll just have to set the updates scan running over a quiet period I suppose.
Plan B = Manual Office Updates http://office.microsoft.com/en-gb/downloads/default.aspx - Left Pane - Office Updates.
Hope this sheds some light.
Regards,
Knaphie

Which Performance Counter tells the CPU Utilization

Hi,
I am new to Performance testing and wanted to know, Which Performance Counter tells the CPU Utilization of any Application hosted on Win 2K8 server. Request you to explain in details how to achieve this ?
Thanks, Anoop Singh

this will help you understand for sure
http://social.technet.microsoft.com/wiki/contents/articles/12984.understanding-processor-processor-time-and-process-processor-time.aspx
Understanding Processor (% Processor Time) and Process (%Processor Time)
Sanket. J Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

JRun 100% CPU Utilization - Source: Attack

When I came in this morning, I found my server (HP ML370, 2x
Xeon 3.06MHz, 4GB RAM, Windows 2003, CF7, Java 1.4.2_05) extremely
slow. Checking out the Task Manager, I found that JRun was taking
up 100% of my CPU utilization. Yikes!
First I thought it was a JVM problem, so I went about
applying the various tweaks: No improvement.
Rebooted: No improvement.
Went through the CF and JVM logs: Nothing of importance
Finally looked at the 404 error log: Blam! Ton of entries
(where usually it's pretty empty). It appears that since midnight
my website was under some sort of attack from 84.32.147.110 (RIPE
network machine). Refer to the attached code - this is a single URL
request (the /directory names/ are correct, but not that deep on my
server - appears he's adding them one after another in a DOS attach
or something).
What is this? Once I tracked it down to this machine, I
blocked access from our firewall. Yes, even now, hours later, he's
still trying to hammer our site. Once I blocked him JRun went back
down to around 3-15% (leaning more towards the 3%) and everything
went to normal.
I've written an abuse complaint to his ISP in Lithuania (I'm
not holding by breath).
What else can I do to avoid future attacks?
Has anyone heard of anything like this?
Help!!!!

The only difference I can discern is that on the machine that doesn't work, the root drive is E
hmmmm. if you have the drive letter for the root drive that high, it's probably worth checking to see if you're getting a Windows drive letter confusion. for troubleshooting advice on that, see:
Windows confuses iPod with network drive or hard drive and may keep iPod from mounting or songs may seem to disappear

Need Help! CPU shows 100% constantly in performance monitor

We have a new Dell t620 server running server 2012 std. running AD DS, dhcp, dns and the performance monitor shows constant 100% CPU load, while the processes in task manager can show very little CPU useage. By the way this is a Intel Xeon E5-2620 6 core
CPU, with 32GB ram which shows normally using only about 2GB ram at the same time. If I do a shutdown and restart I am getting the same 100% CPU load.
Does anyone know what might be causing this behavior?
We have other identical servers which show usually zero to low CPU load, performing the same functions.
Is this a known problem with server 2012 std.?

This server is now in service on a network. So I cannot shut it down during business day.
What is really strange is when I open task manager it shows almost all the CPU useage in APPS, which consist of task manager and resource monitor, with task manager sometimes even going to 100% CPU useage.
While memory, harddisk and ethernet are showing like low useage.
Do you all think it could be memory or a bad CPU? This is a 1 week old Dell server.
Does anyone know if there is a good memory checker that can be run from windows while server is in service?
Thank you for your assistance.

High CPU utilization with JDesktopPane.OUTLINE_DRAG_MODE

Hello there,
since I updated from Java SDK 1.4.0 to 1.4.1_01 I recognized a problem with MDI Java applications using a JDesktopPane with JInternalFrames. When the drag mode of the internal frames is set to OUTLINE_DRAG_MODE, which should have a better performance than the LIVE_DRAG_MODE the cpu utilization goes nearly up to 100% an the drag of the frame is quite slow.
Does anybody else experience this problem?
(The problem exists in the application I develop and also in the IDE Netbeans, I use for development)
I am not sure if this is the right place for my problem, so if there is a better one to post it to, please tell me.
Thanks
R�diger

Hi,
I've also noticed this. It happens on Windows 2000 with 1.4.1, but not with version 1.4.0. Have you found a solution yet?
Martin

Performance degrading CPU utilization 100%

Similar Messages

Maybe you are looking for