Performance issues - Log file parallel write
Hi there,
Since a few months I have big performance issues with my Oracle 11.2.0.1.0.
If I look in the Enterprise manager (in blocking sessions) I see al lot of "log file paralles writes" and a lot of "log file sync" .
We have configured an active data guard environment and are using ASM.
We are not stressing out the database with heavy queries or commits or something, but sometimes during the day this happens on non specific times...
We've investigated everything (performance to SAN / heavy queries / oracle problems etc etc) and we really don't know what to do anymore so i thought.. let's try a post on the Forum.....
Perhaps someone had similar things?
Thanks,
BR
Mark
mwevromans wrote:
See blow a tail of alertlog.
Tue Apr 24 15:12:17 2012
Thread 1 cannot allocate new log, sequence 194085
Checkpoint not complete
Current log# 1 seq# 194084 mem# 0: +DATA/kewillprd/onlinelog/group_1.262.712516155
Current log# 1 seq# 194084 mem# 1: +FRA/kewillprd/onlinelog/group_1.438.756466165
LGWR: Standby redo logfile selected to archive thread 1 sequence 194085
LGWR: Standby redo logfile selected for thread 1 sequence 194085 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 194085 (LGWR switch)
Current log# 2 seq# 194085 mem# 0: +DATA/kewillprd/onlinelog/group_2.264.712516155
Current log# 2 seq# 194085 mem# 1: +FRA/kewillprd/onlinelog/group_2.418.756466215
Tue Apr 24 15:12:21 2012
Archived Log entry 388061 added for thread 1 sequence 194084 ID 0x90d7aa62 dest 1:
Tue Apr 24 15:14:09 2012
Thread 1 cannot allocate new log, sequence 194086
Checkpoint not complete
Current log# 2 seq# 194085 mem# 0: +DATA/kewillprd/onlinelog/group_2.264.712516155
Current log# 2 seq# 194085 mem# 1: +FRA/kewillprd/onlinelog/group_2.418.756466215
LGWR: Standby redo logfile selected to archive thread 1 sequence 194086
LGWR: Standby redo logfile selected for thread 1 sequence 194086 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 194086 (LGWR switch)
Current log# 3 seq# 194086 mem# 0: +DATA/kewillprd/onlinelog/group_3.266.712516155
Current log# 3 seq# 194086 mem# 1: +FRA/kewillprd/onlinelog/group_3.435.756466241
Tue Apr 24 15:14:14 2012
Archived Log entry 388063 added for thread 1 sequence 194085 ID 0x90d7aa62 dest 1:
Tue Apr 24 15:16:46 2012
Thread 1 cannot allocate new log, sequence 194087
Checkpoint not complete
Current log# 3 seq# 194086 mem# 0: +DATA/kewillprd/onlinelog/group_3.266.712516155
Current log# 3 seq# 194086 mem# 1: +FRA/kewillprd/onlinelog/group_3.435.756466241
Thread 1 cannot allocate new log, sequence 194087
Private strand flush not complete
Current log# 3 seq# 194086 mem# 0: +DATA/kewillprd/onlinelog/group_3.266.712516155
Current log# 3 seq# 194086 mem# 1: +FRA/kewillprd/onlinelog/group_3.435.756466241
LGWR: Standby redo logfile selected to archive thread 1 sequence 194087
LGWR: Standby redo logfile selected for thread 1 sequence 194087 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 194087 (LGWR switch)
Current log# 1 seq# 194087 mem# 0: +DATA/kewillprd/onlinelog/group_1.262.712516155
Current log# 1 seq# 194087 mem# 1: +FRA/kewillprd/onlinelog/group_1.438.756466165
Tue Apr 24 15:16:54 2012
Archived Log entry 388065 added for thread 1 sequence 194086 ID 0x90d7aa62 dest 1:
Tue Apr 24 15:18:59 2012
Thread 1 cannot allocate new log, sequence 194088
Checkpoint not complete
Current log# 1 seq# 194087 mem# 0: +DATA/kewillprd/onlinelog/group_1.262.712516155
Current log# 1 seq# 194087 mem# 1: +FRA/kewillprd/onlinelog/group_1.438.756466165
Thread 1 cannot allocate new log, sequence 194088
Private strand flush not complete
Current log# 1 seq# 194087 mem# 0: +DATA/kewillprd/onlinelog/group_1.262.712516155
Current log# 1 seq# 194087 mem# 1: +FRA/kewillprd/onlinelog/group_1.438.756466165
LGWR: Standby redo logfile selected to archive thread 1 sequence 194088
LGWR: Standby redo logfile selected for thread 1 sequence 194088 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 194088 (LGWR switch)
Current log# 2 seq# 194088 mem# 0: +DATA/kewillprd/onlinelog/group_2.264.712516155
Current log# 2 seq# 194088 mem# 1: +FRA/kewillprd/onlinelog/group_2.418.756466215
Tue Apr 24 15:19:06 2012
Archived Log entry 388067 added for thread 1 sequence 194087 ID 0x90d7aa62 dest 1:
Tue Apr 24 15:22:00 2012
Thread 1 cannot allocate new log, sequence 194089
Checkpoint not complete
Current log# 2 seq# 194088 mem# 0: +DATA/kewillprd/onlinelog/group_2.264.712516155
Current log# 2 seq# 194088 mem# 1: +FRA/kewillprd/onlinelog/group_2.418.756466215
Thread 1 cannot allocate new log, sequence 194089
Private strand flush not complete
Current log# 2 seq# 194088 mem# 0: +DATA/kewillprd/onlinelog/group_2.264.712516155
Current log# 2 seq# 194088 mem# 1: +FRA/kewillprd/onlinelog/group_2.418.756466215
LGWR: Standby redo logfile selected to archive thread 1 sequence 194089
LGWR: Standby redo logfile selected for thread 1 sequence 194089 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 194089 (LGWR switch)
Current log# 3 seq# 194089 mem# 0: +DATA/kewillprd/onlinelog/group_3.266.712516155
Current log# 3 seq# 194089 mem# 1: +FRA/kewillprd/onlinelog/group_3.435.756466241
Tue Apr 24 15:19:06 2012
Archived Log entry 388069 added for thread 1 sequence 194088 ID 0x90d7aa62 dest 1:Hi
1st switch time ==> Tue Apr 24 15:18:59 2012
2nd switch time ==> Tue Apr 24 15:19:06 2012
3rd switch time ==> Tue Apr 24 15:19:06 2012
Redo log file switch has good impact on the performance of the database. Frequent log switches may lead to the slowness of the database . Oracle documents suggests to resize the redolog files so that log switches happen more like every 15-30 min (roughly depending on the architecture and recovery requirements).
AS i check the alertlog file and find that the log are switchinh very fequent which is one of the reason that you are getting checkpoint not complete message . i have face this issue many times and i generally increase the size of the logfile and set the archive_lag_time parameter as i have suggested above . If you further want to go root cause and more details then above guys will help you more because i don't have much experience in database tunning . If you looking for aworkarounf then you must go through it .
Good Luck
--neeraj
Similar Messages
-
Wait Events "log file parallel write" / "log file sync" during CREATE INDEX
Hello guys,
at my current project i am performing some performance tests for oracle data guard. The question is "How does a LGWR SYNC transfer influences the system performance?"
To get some performance values, that i can compare i just built up a normal oracle database in the first step.
Now i am performing different tests like creating "large" indexes, massive parallel inserts/commits, etc. to get the bench mark.
My database is an oracle 10.2.0.4 with multiplexed redo log files on AIX.
I am creating an index on a "normal" table .. i execute "dbms_workload_repository.create_snapshot()" before and after the CREATE INDEX to get an equivalent timeframe for the AWR report.
After the index is built up (round about 9 GB) i perform an awrrpt.sql to get the AWR report.
And now take a look at these values from the AWR
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
log file parallel write 10,019 .0 132 13 33.5
log file sync 293 .7 4 15 1.0
......How can this be possible?
Regarding to the documentation
-> log file sync: http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3120
Wait Time: The wait time includes the writing of the log buffer and the post.-> log file parallel write: http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3104
Wait Time: Time it takes for the I/Os to complete. Even though redo records are written in parallel, the parallel write is not complete until the last I/O is on disk.This was also my understanding .. the "log file sync" wait time should be higher than the "log file parallel write" wait time, because of it includes the I/O and the response time to the user session.
I could accept it, if the values are close to each other (maybe round about 1 second in total) .. but the different between 132 seconds and 4 seconds is too noticeable.
Is the behavior of the log file sync/write different when performing a DDL like CREATE INDEX (maybe async .. like you can influence it with the initialization parameter COMMIT_WRITE??)?
Do you have any idea how these values come about?
Any thoughts/ideas are welcome.
Thanks and RegardsSurachart Opun (HunterX) wrote:
Thank you for Nice Idea.
In this case, How can we reduce "log file parallel write" and "log file sync" waited time?
CREATE INDEX with NOLOGGINGA NOLOGGING can help, can't it?Yes - if you create index nologging then you wouldn't be generating that 10GB of redo log, so the waits would disappear.
Two points on nologging, though:
<ul>
it's "only" an index, so you could always rebuild it in the event of media corruption, but if you had lots of indexes created nologging this might cause an unreasonable delay before the system was usable again - so you should decide on a fallback option, such as taking a new backup of the tablespace as soon as all the nologging operatons had completed.
If the database, or that tablespace, is in +"force logging"+ mode, the nologging will not work.
</ul>
Don't get too alarmed by the waits, though. My guess is that the +"log file sync"+ waits are mostly from other sessions, and since there aren't many of them the other sessions are probably not seeing a performance issue. The +"log file parallel write"+ waits are caused by your create index, but they are happeninng to lgwr in the background which is running concurrently with your session - so your session is not (directly) affected by them, so may not be seeing a performance issue.
The other sessions are seeing relatively high sync times because their log file syncs have to wait for one of the large writes that you have triggered to complete, and then the logwriter includes their (little) writes with your next (large) write.
There may be a performance impact, though, from the pure volume of I/O. Apart from the I/O to write the index you have LGWR writting (N copies) of the redo for the index and ARCH is reading and writing the completed log files caused by the index build. So the 9GB of index could easily be responsible for vastly more I/O than the initial 9GB.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan -
Hi,
on 11g R2. I have the following :
SQL> select total_waits, time_waited from v$system_event where event='log file parallel write';
TOTAL_WAITS TIME_WAITED
74144 28100Is it too much or not ? These valuses , to what should be compared ?
Thank you.Hi,
thanks to all.
Here is checkpoint frequency from my alertlog :
Wed May 9 23:01:14 2012
Thread 1 advanced to log sequence 14 (LGWR switch)
Current log# 2 seq# 14 mem# 0: /index01/bases/MYDB/redo02.log
Thu May 10 02:14:27 2012
Thread 1 cannot allocate new log, sequence 15
Private strand flush not complete
Current log# 2 seq# 14 mem# 0: /index01/bases/MYDB/redo02.log
Thu May 10 02:14:29 2012
Thread 1 advanced to log sequence 15 (LGWR switch)
Current log# 3 seq# 15 mem# 0: /index01/bases/MYDB/redo03.log
Thu May 10 02:14:29 2012
ALTER SYSTEM ARCHIVE LOG
Thu May 10 02:14:29 2012
Thread 1 advanced to log sequence 16 (LGWR switch)
Current log# 1 seq# 16 mem# 0: /index01/bases/MYDB/redo01.log No I do not know log advisor. Where is it ?
Karan,
thank for your interesting remarks.
I usually have created DBs without enough attention to REDOLOG file size. I decided to verify on the most recent of my DBs. I found that query and issued it. For me it is not enough but I wanted to understand.
Regards. -
Log file sync vs log file parallel write probably not bug 2669566
This is a continuation of a previous thread about ‘log file sync’ and ‘log file parallel write’ events.
Version : 9.2.0.8
Platform : Solaris
Application : Oracle Apps
The number of commits per second ranges between 10 and 30.
When querying statspack performance data the calculated average wait time on the event ‘log file sync’ is on average 10 times the wait time for the ‘log file parallel write’ event.
Below just 2 samples where the ratio is even about 20.
"snap_time" " log file parallel write avg" "log file sync avg" "ratio
11/05/2008 10:38:26 8,142 156,343 19.20
11/05/2008 10:08:23 8,434 201,915 23.94
So the wait time for a ‘log file sync’ is 10 times the wait time for a ‘log file parallel write’.
First I thought that I was hitting bug 2669566.
But then Jonathan Lewis is blog pointed me to Tanel Poder’s snapper tool.
And I think that it proves that I am NOT hitting this bug.
Below is a sample of the output for the log writer.
-- End of snap 3
HEAD,SID, SNAPSHOT START ,SECONDS,TYPE,STATISTIC , DELTA, DELTA/SEC, HDELTA, HDELTA/SEC
DATA, 4, 20081105 10:35:41, 30, STAT, messages sent , 1712, 57, 1.71k, 57.07
DATA, 4, 20081105 10:35:41, 30, STAT, messages received , 866, 29, 866, 28.87
DATA, 4, 20081105 10:35:41, 30, STAT, background timeouts , 10, 0, 10, .33
DATA, 4, 20081105 10:35:41, 30, STAT, redo wastage , 212820, 7094, 212.82k, 7.09k
DATA, 4, 20081105 10:35:41, 30, STAT, redo writer latching time , 2, 0, 2, .07
DATA, 4, 20081105 10:35:41, 30, STAT, redo writes , 867, 29, 867, 28.9
DATA, 4, 20081105 10:35:41, 30, STAT, redo blocks written , 33805, 1127, 33.81k, 1.13k
DATA, 4, 20081105 10:35:41, 30, STAT, redo write time , 652, 22, 652, 21.73
DATA, 4, 20081105 10:35:41, 30, WAIT, rdbms ipc message ,23431084, 781036, 23.43s, 781.04ms
DATA, 4, 20081105 10:35:41, 30, WAIT, log file parallel write , 6312957, 210432, 6.31s, 210.43ms
DATA, 4, 20081105 10:35:41, 30, WAIT, LGWR wait for redo copy , 18749, 625, 18.75ms, 624.97us
When adding the DELTA/SEC (which is in micro seconds) for the wait events it always roughly adds up to a million micro seconds.
In the example above 781036 + 210432 = 991468 micro seconds.
This is the case for all the snaps taken by snapper.
So I think that the wait time for the ‘log file parallel write time’ must be more or less correct.
So I still have the question “Why is the ‘log file sync’ about 10 times the time of the ‘log file parallel write’?”
Any clues?Yes that is true!
But that is the way I calculate the average wait time = total wait time / total waits
So the average wait time for the event 'log file sync' per wait should be near the wait time for the 'llog file parallel write' event.
I use the query below:
select snap_id
, snap_time
, event
, time_waited_micro
, (time_waited_micro - p_time_waited_micro)/((snap_time - p_snap_time) * 24) corrected_wait_time_h
, total_waits
, (total_waits - p_total_waits)/((snap_time - p_snap_time) * 24) corrected_waits_h
, trunc(((time_waited_micro - p_time_waited_micro)/((snap_time - p_snap_time) * 24))/((total_waits - p_total_waits)/((snap_time - p_snap_time) * 24))) average
from (
select sn.snap_id, sn.snap_time, se.event, se.time_waited_micro, se.total_waits,
lag(sn.snap_id) over (partition by se.event order by sn.snap_id) p_snap_id,
lag(sn.snap_time) over (partition by se.event order by sn.snap_time) p_snap_time,
lag(se.time_waited_micro) over (partition by se.event order by sn.snap_id) p_time_waited_micro,
lag(se.total_waits) over (partition by se.event order by sn.snap_id) p_total_waits,
row_number() over (partition by event order by sn.snap_id) r
from perfstat.stats$system_event se, perfstat.stats$snapshot sn
where se.SNAP_ID = sn.SNAP_ID
and se.EVENT = 'log file sync'
order by snap_id, event
where time_waited_micro - p_time_waited_micro > 0
order by snap_id desc; -
'log file sync' versus 'log file prallel write'
I have been asked to run an artificial test that performs a large number of small insert-only transactions with a high degree (200) of parallelism. The COMMITS were not inside a PL/SQL loop so a 'log file sync' (LFS) event occured each COMMIT. I have measured the average 'log file parallel write' (LFPW) time by running the following PL/SQL queries at the beginning and end of a 10 second period:
SELECT time_waited,
total_waits
INTO wait_start_lgwr,
wait_start_lgwr_c
FROM v$system_event e
WHERE event LIKE 'log%parallel%';
SELECT time_waited,
total_waits
INTO wait_end_lgwr,
wait_end_lgwr_c
FROM v$system_event e
WHERE event LIKE 'log%parallel%';
I took the difference in TIME_WAITED and divided it by the difference in TOTAL_WAITS.
I did the same thing for LFS.
What I expected was that the LFS time would be just over 50% more than the LFPW time: when the thread commits it has to wait for the previous LFPW to complete (on average half way through) and then for its own.
Now I know there is a lot of CPU related stuff that goes on in LGWR but I 'reniced' it to a higher priority and could observe that it was then spending 90% of its time in LFPW, 10% ON CPU and no time idle. Total system CPU time averaged only 25% on this 64 'processor' machine.
What I saw was that the LFS time was substantially more than the LFPW time. For example, on one test LFS was 18.07ms and LFPW was 6.56ms.
When I divided the number of bytes written each time by the average 'commit size' it seems that LGWR is writing out data for only about one third of the average number of transactions in LFS state (rather than the two thirds that I would have expected). When the COMMIT was changed to COMMIT WORK NOWAIT the size of each LFPW increased substantially.
These observations are at odds with my understanding of how LGWR works. My understanding is that when LGWR completes one LFPW it begins a new one with the entire contents of the log buffer at that time.
Can anybody tell me what I am missing?
P.S. Same results in database versions 10.2 Sun M5000 and 11.2 HP G7s.I have been asked to run an artificial test that performs a large number of small insert-only transactions with a high degree (200) of parallelism. The COMMITS were not inside a PL/SQL loop so a 'log file sync' (LFS) event occured each COMMIT. I have measured the average 'log file parallel write' (LFPW) time by running the following PL/SQL queries at the beginning and end of a 10 second period:
SELECT time_waited,
total_waits
INTO wait_start_lgwr,
wait_start_lgwr_c
FROM v$system_event e
WHERE event LIKE 'log%parallel%';
SELECT time_waited,
total_waits
INTO wait_end_lgwr,
wait_end_lgwr_c
FROM v$system_event e
WHERE event LIKE 'log%parallel%';
I took the difference in TIME_WAITED and divided it by the difference in TOTAL_WAITS.
I did the same thing for LFS.
What I expected was that the LFS time would be just over 50% more than the LFPW time: when the thread commits it has to wait for the previous LFPW to complete (on average half way through) and then for its own.
Now I know there is a lot of CPU related stuff that goes on in LGWR but I 'reniced' it to a higher priority and could observe that it was then spending 90% of its time in LFPW, 10% ON CPU and no time idle. Total system CPU time averaged only 25% on this 64 'processor' machine.
What I saw was that the LFS time was substantially more than the LFPW time. For example, on one test LFS was 18.07ms and LFPW was 6.56ms.
When I divided the number of bytes written each time by the average 'commit size' it seems that LGWR is writing out data for only about one third of the average number of transactions in LFS state (rather than the two thirds that I would have expected). When the COMMIT was changed to COMMIT WORK NOWAIT the size of each LFPW increased substantially.
These observations are at odds with my understanding of how LGWR works. My understanding is that when LGWR completes one LFPW it begins a new one with the entire contents of the log buffer at that time.
Can anybody tell me what I am missing?
P.S. Same results in database versions 10.2 Sun M5000 and 11.2 HP G7s. -
Hi,
From my statspack report one of the top wait events is control file parallel write.
Wait Time
Event Time Percentage Avg. wait
Control file 11000 3.61% 11.68
parallel write
How can I tune the control file parallel write event?
Right now for this instance I have control file multiplexed onto
3 different drives L. M. N
ThanksIf you are doing excessive log swaps, you could be generating too many checkpoints. See how many log swaps you have done in v$loghist. It is also possible to reduce the number of writes to log files by adding the /*+ APPEND */ hint and the "nologging" to insert statements to reduce the amount of log files filled.
I have also combined update and delete statements to generate fewer writes to the log files.
You can recreate the log files larger with :
alter database drop logfile group 3;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 3 '/oracle/oradata/CURACAO9/redo03.log' size 500m reuse;
ALTER DATABASE ADD LOGFILE member '/oracle/oradata/CURACAO9/redo03b.log' reuse to GROUP 3;
alter system switch logfile;
alter system checkpoint;
alter database drop logfile group 2; -- and do group 2 etc. -
Performance issues logging on to Workspace 11.1.2.1
Hi All,
We have a distributed install of 11.1.2.1 (2 HSS, 3 PLN, 3HFM, 3 Workspace, 2 FDM servers) and facing significant performance issues:
- Logging on to the Workspace as an external user takes 5 min
- Logging on to the AAS console as an external user takes over 2 min
- Opening HBRs in AAS takes a couple of minutes
- Logging on to HSS as an external user takes over a min
Logging on as a native user takes around 30 seconds.
We have disabled compoments we do not use under the Workspace server properties and applied patch Set Exception (PSE): 13327628 which addressed the following perfomance issues:
• 12913216 – Intermittent login error is observed while attempting to log into EPM System Workspace. Workspace displays the error message “You must supply a valid User Name and Password to log onto the system.” However, user can log in by clicking OK.
• 13341789 – Poor login performance (delay of 3-5 minutes) is observed at the first user login if Workspace web application has been inactive for an hour. Subsequent login performance is not impacted.
• 13388864 – In deployments where a firewall is configured to time out idle applications (for example, after 30 minutes) users can login once, but subsequent login times out.
In Shared Services we have also set Evict Interval and Allowed Idle Connection Time to 5 mins.
Is there anything else we could try to improve performance?
Thanks for your help.
Regards
SebIf you already have deactivated the non used services from WS, it's weird. As I can see you may be using an external directory for authenticating, check on these.
* have you checked the response time from the external directory... sometimes for example in Active directory the user hierarchy is too complex to navigate on it. you can use an LDAP tester to see
* Are you using SSL? if yes, try wtihout SSL
* For workspace Start the non used services, and activate back the services in the WS to see if there it improves with all the apps up ( this will help you to narrow the debugging)
* go directly to java based servers, for instance in the shared services server go directly to the port 28080 for interop, instead of going by http server and check if you can login quick or not. This will help you also to isolate the issue (if it's related or not to Workspace).
Hope this help u to narrow your search
Motor -
OPM과 OM ISSUE 해결을 위해 LOG FILE을 생성 및 검색하는 방법
제품 : MFG_OM
작성날짜 : 2005-11-09
OPM과 OM ISSUE 해결을 위해 LOG FILE을 생성 및 검색하는 방법
===========================================================
PURPOSE
Customer에게 Oralce Processing Manufacturing(OPM)/Order Management(OM)
issue 해결을 위해 Log file을 생성하고 검색하는 방법을 알고 이를 고객에게
전달할 수 있다.
Explanation
OPM 고객이 Order process 중 fail이 발생 했을 경우,fails 된 Order process
에 따라 우리는 관련된 log file을 받기를 원한다.
예를 들면;
a) Order pad에 문제가 있다면 사용자는 Tools > Debug을 'On'으로 setting
함으로써 log file을 생성할 수 있다.
이는 Order pad 자체에서 행할수 있고 좀 더 자세한 사항은 아래의 step2 에
설명되어져 있다.
b) 만약 pick release가 fails 된다면 OPM support는 'pick release
generation' concurrent manager log file을 요구하게 된다.
좀 더 자세한 사항은 아래 step4 에 설명되어져 있다.
c) 만약 Transact process move order form(manual allocation screen)의
allocation 작업이나 pick confirm이 fails 이라면 support는 생성되는
OPMLOG***를 받기를 원할 것이다.
좀 더 자세한 사항은 아래 step5 에 설명되어져 있다.
d) 만약 Order Management/Inventory Interface가 fail 이라면 support는
'Interface Trip Stop' concurrent 실행중에 생성된 log file을 받기를 원할
것이다. 자세한 사항은 step5 에 설명되어져 있다.
OM/OPM issues를 위한 Log files의 생성 및 검색 방법
1. utl_file_dir parameter 값을 얻기 위해 아래의 sql을 실행한다.
select value
from v$parameter
where name like 'utl_file_dir';
2. OM: Debug Directory profile option 값으로 step1의 결과값이 setting
되어져 있는지 확인한다.
3. Order pad에서 log file 생성:
1) OM: Debug level profile을 '5'로 setting
2) Order pad에서 Tools > Debug를 ON, lof file name을 적어 놓는다.
3) Issue를 재현하고 step1 결과로 나온 경로에서 log file을 검색한다.
4. Pick release concurrent log file을 생성하기 위해서는 profile option
OM: Debug level을 '5' setting 후 pick release를 concurrent manager
mode로 수행한다.
Application의 view requests 화면에서 log file을 검색할 수 있다.
5. Transact move order form에서 'manual allocation' 진행시의 log file을
생성하기 위해서 아래의 steps을 따른다.
1) Step2에서 지정한 경로에서 'OPMLOG****' file을 remove 한다.
2) Manual allocation을 수행한다.
3) 'OPMLOG***' log file이 생성될 것이며, '****'는 session id 값이다.
Session id 값은 application 화면의 Help > Tools > Examine의
$Profiles/db_session_id 위에서 찾을 수 있다.
4) Pick confirm에 관련된 log는 2)번 step을 제외하고, pick confirm을
수행하여 생성할 수 있다.
6. Interface Trip Stop(ITS)가 실행되는 중의 log file을 생성하기 위해서는
OM: Debug level profile option 값을 Interface Trip Stop을 실행하기
전에 '5'로 setting 한다.
Interface Trip Stop을 실행하기전에 'Defer' checkbox가 uncheck 되어
있는지 확인해야 하며, 이 log file은 Application의 view requests 화면
에서 검색할 수 있다.
Reference Documents
Note 290069.1 -
Apex/OHS issue:log files in opmn consume disk space
Hi.
I have installed OHS from Companion CD (10g Rel1) and the new Apex 3.0.
There seems to be a problem because log files in opmn/logs are eating up my disk space. Apparently there's something missing or not configured yet.
The exact error message is:
07/04/12 10:34:13 [4] Falta el factor de formato de la conexión local 0,127.0.0.1,6100
The contents of etc/hosts is:
[oracle@caman bin]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
172.17.1.8 caman.rioturbio.com.ve caman
172.17.1.1 tepuy.rioturbio.com.ve tepuy
OS info:
[oracle@caman bin]$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
Please help ...! Thanks in advance ....!Hi,
I believe this is due to a conflict with something else using that port number (6100). One solution would be to edit the opmn config file and change the port from 6100 to something else (such as 6101). -
Performance issues in file download servlet
FileInputStream in = null;
ServletOutputStream sos = res.getOutputStream();
File f = new File(path);
if(f.exists()) {
in = new FileInputStream(f);
int length = (int)f.length();
byte[] input = new byte[length];
res.setContentType ("application/octet-stream");
res.setHeader("Content-Disposition", "attachment; filename=\""+fileName+"\"");
res.setContentLength(length);
if(length != 0) {
while ((in != null) && ((length = in.read(input)) != -1)) { // read
sos.write(input,0, input.length); // write
}Hi experts, the above is a block of code that sends a file to a client machine through servlet. One problem I am having now is that it frequently cause the server to run out of memory. Is it because of the way I read the FileInputStream is not efficient?
byte[] input = new byte[length];
if(length != 0) {
while ((in != null) && ((length = in.read(input)) != -1)) { // read
sos.write(input,0, input.length); // write
}It seems that my code reads an entire file into memory before it writes anything to the client, so when there are many people downloading files, the server will run out of memory. Is it true?
I am not very familiar with the read() and write() methods in the inputstream and output stream classes. So I hope someone here can tell me whether the above code is indeed the cause of the problem. If not, then I can troubleshoot my other servlets and JSPs.
Thanks!try not to create a byte[] with the file size
try to make it a fix one
e.g.new byte[1024]
treat it as a package to sendThanks. But is 1024 too small a size for sending files throught the internet? What is the best size to used? -
Performance Issue: Wait event "log file sync" and "Execute to Parse %"
In one of our test environments users are complaining about slow response.
In statspack report folowing are the top-5 wait events
Event Waits Time (cs) Wt Time
log file parallel write 1,046 988 37.71
log file sync 775 774 29.54
db file scattered read 4,946 248 9.47
db file parallel write 66 248 9.47
control file parallel write 188 152 5.80
And after runing the same application 4 times, we are geting Execute to Parse % = 0.10. Cursor sharing is forced and query rewrite is enabled
When I view v$sql, following command is parsed frequently
EXECUTIONS PARSE_CALLS
SQL_TEXT
93380 93380
select SEQ_ORDO_PRC.nextval from DUAL
Please suggest what should be the method to troubleshoot this and if I need to check some more information
Regards,
Sudhanshu BhandariWell, of course, you probably can't eliminate this sort of thing entirely: a setup such as yours is inevitably a compromise. What you can do is make sure your log buffer is a good size (say 10MB or so); that your redo logs are large (at least 100MB each, and preferably large enough to hold one hour or so of redo produced at the busiest time for your database without filling up); and finally set ARCHIVE_LAG_TARGET to something like 1800 seconds or more to ensure a regular, routine, predictable log switch.
It won't cure every ill, but that sort of setup often means the redo subsystem ceases to be a regular driver of foreground waits. -
Log file sync top event during performance test -av 36ms
Hi,
During the performance test for our product before deployment into product i see "log file sync" on top with Avg wait (ms) being 36 which i feel is too high.
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
log file sync 208,327 7,406 36 46.6 Commit
direct path write 646,833 3,604 6 22.7 User I/O
DB CPU 1,599 10.1
direct path read temp 1,321,596 619 0 3.9 User I/O
log buffer space 4,161 558 134 3.5 ConfiguratAlthough testers are not complaining about the performance of the appplication , we ,DBAs, are expected to be proactive about the any bad signals from DB.
I am not able to figure out why "log file sync" is having such slow response.
Below is the snapshot from the load profile.
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 108127 16-May-13 20:15:22 105 6.5
End Snap: 108140 16-May-13 23:30:29 156 8.9
Elapsed: 195.11 (mins)
DB Time: 265.09 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,168M 1,136M Std Block Size: 8K
Shared Pool Size: 1,120M 1,168M Log Buffer: 16,640K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 1.4 0.1 0.02 0.01
DB CPU(s): 0.1 0.0 0.00 0.00
Redo size: 607,512.1 33,092.1
Logical reads: 3,900.4 212.5
Block changes: 1,381.4 75.3
Physical reads: 134.5 7.3
Physical writes: 134.0 7.3
User calls: 145.5 7.9
Parses: 24.6 1.3
Hard parses: 7.9 0.4
W/A MB processed: 915,418.7 49,864.2
Logons: 0.1 0.0
Executes: 85.2 4.6
Rollbacks: 0.0 0.0
Transactions: 18.4Some of the top background wait events:
^LBackground Wait Events DB/Inst: Snaps: 108127-108140
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
log file parallel write 208,563 0 2,528 12 1.0 66.4
db file parallel write 4,264 0 785 184 0.0 20.6
Backup: sbtbackup 1 0 516 516177 0.0 13.6
control file parallel writ 4,436 0 97 22 0.0 2.6
log file sequential read 6,922 0 95 14 0.0 2.5
Log archive I/O 6,820 0 48 7 0.0 1.3
os thread startup 432 0 26 60 0.0 .7
Backup: sbtclose2 1 0 10 10094 0.0 .3
db file sequential read 2,585 0 8 3 0.0 .2
db file single write 560 0 3 6 0.0 .1
log file sync 28 0 1 53 0.0 .0
control file sequential re 36,326 0 1 0 0.2 .0
log file switch completion 4 0 1 207 0.0 .0
buffer busy waits 5 0 1 116 0.0 .0
LGWR wait for redo copy 924 0 1 1 0.0 .0
log file single write 56 0 1 9 0.0 .0
Backup: sbtinfo2 1 0 1 500 0.0 .0During a previous perf test , things didnt look this bad for "log file sync. Few sections from the comparision report(awrddprt.sql)
{code}
Workload Comparison
~~~~~~~~~~~~~~~~~~~ 1st Per Sec 2nd Per Sec %Diff 1st Per Txn 2nd Per Txn %Diff
DB time: 0.78 1.36 74.36 0.02 0.07 250.00
CPU time: 0.18 0.14 -22.22 0.00 0.01 100.00
Redo size: 573,678.11 607,512.05 5.90 15,101.84 33,092.08 119.13
Logical reads: 4,374.04 3,900.38 -10.83 115.14 212.46 84.52
Block changes: 1,593.38 1,381.41 -13.30 41.95 75.25 79.38
Physical reads: 76.44 134.54 76.01 2.01 7.33 264.68
Physical writes: 110.43 134.00 21.34 2.91 7.30 150.86
User calls: 197.62 145.46 -26.39 5.20 7.92 52.31
Parses: 7.28 24.55 237.23 0.19 1.34 605.26
Hard parses: 0.00 7.88 100.00 0.00 0.43 100.00
Sorts: 3.88 4.90 26.29 0.10 0.27 170.00
Logons: 0.09 0.08 -11.11 0.00 0.00 0.00
Executes: 126.69 85.19 -32.76 3.34 4.64 38.92
Transactions: 37.99 18.36 -51.67
First Second Diff
1st 2nd
Event Wait Class Waits Time(s) Avg Time(ms) %DB time Event Wait Class Waits Time(s) Avg Time
(ms) %DB time
SQL*Net more data from client Network 2,133,486 1,270.7 0.6 61.24 log file sync Commit 208,355 7,407.6
35.6 46.57
CPU time N/A 487.1 N/A 23.48 direct path write User I/O 646,849 3,604.7
5.6 22.66
log file sync Commit 99,459 129.5 1.3 6.24 log file parallel write System I/O 208,564 2,528.4
12.1 15.90
log file parallel write System I/O 100,732 126.6 1.3 6.10 CPU time N/A 1,599.3
N/A 10.06
SQL*Net more data to client Network 451,810 103.1 0.2 4.97 db file parallel write System I/O 4,264 784.7 1
84.0 4.93
-direct path write User I/O 121,044 52.5 0.4 2.53 -SQL*Net more data from client Network 7,407,435 279.7
0.0 1.76
-db file parallel write System I/O 986 22.8 23.1 1.10 -SQL*Net more data to client Network 2,714,916 64.6
0.0 0.41
{code}
*To sum it sup:
1. Why is the IO response getting such an hit during the new perf test? Please suggest*
2. Does the number of DB writer impact "log file sync" wait event? We have only one DB writer as the number of cpu on the host is only 4
{code}
select *from v$version;
BANNER
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
PL/SQL Release 11.1.0.7.0 - Production
CORE 11.1.0.7.0 Production
TNS for HPUX: Version 11.1.0.7.0 - Production
NLSRTL Version 11.1.0.7.0 - Production
{code}
Please let me know if you would like to see any other stats.
Edited by: Kunwar on May 18, 2013 2:20 PM1. A snapshot interval of 3 hours always generates meaningless results
Below are some details from the 1 hour interval AWR report.
Platform CPUs Cores Sockets Memory(GB)
HP-UX IA (64-bit) 4 4 3 31.95
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 108129 16-May-13 20:45:32 140 8.0
End Snap: 108133 16-May-13 21:45:53 150 8.8
Elapsed: 60.35 (mins)
DB Time: 140.49 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,168M 1,168M Std Block Size: 8K
Shared Pool Size: 1,120M 1,120M Log Buffer: 16,640K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 2.3 0.1 0.03 0.01
DB CPU(s): 0.1 0.0 0.00 0.00
Redo size: 719,553.5 34,374.6
Logical reads: 4,017.4 191.9
Block changes: 1,521.1 72.7
Physical reads: 136.9 6.5
Physical writes: 158.3 7.6
User calls: 167.0 8.0
Parses: 25.8 1.2
Hard parses: 8.9 0.4
W/A MB processed: 406,220.0 19,406.0
Logons: 0.1 0.0
Executes: 88.4 4.2
Rollbacks: 0.0 0.0
Transactions: 20.9
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
log file sync 73,761 6,740 91 80.0 Commit
log buffer space 3,581 541 151 6.4 Configurat
DB CPU 348 4.1
direct path write 238,962 241 1 2.9 User I/O
direct path read temp 487,874 174 0 2.1 User I/O
Background Wait Events DB/Inst: Snaps: 108129-108133
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
log file parallel write 61,049 0 1,891 31 0.8 87.8
db file parallel write 1,590 0 251 158 0.0 11.6
control file parallel writ 1,372 0 56 41 0.0 2.6
log file sequential read 2,473 0 50 20 0.0 2.3
Log archive I/O 2,436 0 20 8 0.0 .9
os thread startup 135 0 8 60 0.0 .4
db file sequential read 668 0 4 6 0.0 .2
db file single write 200 0 2 9 0.0 .1
log file sync 8 0 1 152 0.0 .1
log file single write 20 0 0 21 0.0 .0
control file sequential re 11,218 0 0 0 0.1 .0
buffer busy waits 2 0 0 161 0.0 .0
direct path write 6 0 0 37 0.0 .0
LGWR wait for redo copy 380 0 0 0 0.0 .0
log buffer space 1 0 0 89 0.0 .0
latch: cache buffers lru c 3 0 0 1 0.0 .0 2 The log file sync is a result of commit --> you are committing too often, maybe even every individual record.
Thanks for explanation. +Actually my question is WHY is it so slow (avg wait of 91ms)+3 Your IO subsystem hosting the online redo log files can be a limiting factor.
We don't know anything about your online redo log configuration
Below is my redo log configuration.
GROUP# STATUS TYPE MEMBER IS_
1 ONLINE /oradata/fs01/PERFDB1/redo_1a.log NO
1 ONLINE /oradata/fs02/PERFDB1/redo_1b.log NO
2 ONLINE /oradata/fs01/PERFDB1/redo_2a.log NO
2 ONLINE /oradata/fs02/PERFDB1/redo_2b.log NO
3 ONLINE /oradata/fs01/PERFDB1/redo_3a.log NO
3 ONLINE /oradata/fs02/PERFDB1/redo_3b.log NO
6 rows selected.
04:13:14 perf_monitor@PERFDB1> col FIRST_CHANGE# for 999999999999999999
04:13:26 perf_monitor@PERFDB1> select *from v$log;
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIME
1 1 40689 524288000 2 YES INACTIVE 13026185905545 18-MAY-13 01:00
2 1 40690 524288000 2 YES INACTIVE 13026185931010 18-MAY-13 03:32
3 1 40691 524288000 2 NO CURRENT 13026185933550 18-MAY-13 04:00Edited by: Kunwar on May 18, 2013 2:46 PM -
Log file sequential read and RFS ping/write - among Top 5 event
I have situation here to discuss. In a 3-node RAC setup which is Logical standby DB; one node is showing high CPU utilization around 40~50%. The CPU utilization was less than 20% 10 days back but from 9th oldest day it jumped and consistently shows the double figure. I ran AWR reports on all three nodes and found one node with high CPU utilization and shows below tops events-
EVENT WAITS TIME(S) AVG WAIT(MS) %TOTAL CALL TIME WAIT CLASS
CPU time 5,802 34.9
RFS ping 15 5,118 33,671 30.8 Other
Log file sequential read 234,831 5,036 21 30.3 System I/O
Sql*Net more data from
client 24,171 1,087 45 6.5 Network
Db file sequential read 130,939 453 3 2.7 User I/O
Findings:-
On AWR report(file attached) for node= sipd207; we can see that "RFS PING" wait event takes 30% of the waits and "log file sequential read" wait event takes 30% of the waits that occurs in database.
Environment :- (Oracle- 10.2.0.4.0, O/S - AIX .3)
1)other node awr shows "log file sync" - is it due to oversized log buffer?
2)Network wait events can be reduced by tweaking SDU & TDU values based on MDU.
3) Why ARCH processes taking much to archives filled redo logs; is it issue with slow disk I/O?
Regards
WORKLOAD REPOSITORY report for<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<DB Name DB Id Instance Inst Num Release RAC Host
XXXPDB 4123595889 XXX2p2 2 10.2.0.4.0 YES sipd207
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 1053 04-Apr-11 18:00:02 59 7.4
End Snap: 1055 04-Apr-11 20:00:35 56 7.5
Elapsed: 120.55 (mins)
DB Time: 233.08 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 3,728M 3,728M Std Block Size: 8K
Shared Pool Size: 4,080M 4,080M Log Buffer: 14,332K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 245,392.33 10,042.66
Logical reads: 9,080.80 371.63
Block changes: 1,518.12 62.13
Physical reads: 7.50 0.31
Physical writes: 44.00 1.80
User calls: 36.44 1.49
Parses: 25.84 1.06
Hard parses: 0.59 0.02
Sorts: 12.06 0.49
Logons: 0.05 0.00
Executes: 295.91 12.11
Transactions: 24.43
% Blocks changed per Read: 16.72 Recursive Call %: 94.18
Rollback per transaction %: 4.15 Rows per Sort: 53.31
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 99.92 In-memory Sort %: 100.00
Library Hit %: 99.83 Soft Parse %: 97.71
Execute to Parse %: 91.27 Latch Hit %: 99.79
Parse CPU to Parse Elapsd %: 15.69 % Non-Parse CPU: 99.95
Shared Pool Statistics Begin End
Memory Usage %: 83.60 84.67
% SQL with executions>1: 97.49 97.19
% Memory for SQL w/exec>1: 97.10 96.67
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 4,503 32.2
RFS ping 168 4,275 25449 30.6 Other
log file sequential read 183,537 4,173 23 29.8 System I/O
SQL*Net more data from client 21,371 1,009 47 7.2 Network
RFS write 25,438 343 13 2.5 System I/O
RAC Statistics DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
Begin End
Number of Instances: 3 3
Global Cache Load Profile
~~~~~~~~~~~~~~~~~~~~~~~~~ Per Second Per Transaction
Global Cache blocks received: 0.78 0.03
Global Cache blocks served: 1.18 0.05
GCS/GES messages received: 131.69 5.39
GCS/GES messages sent: 139.26 5.70
DBWR Fusion writes: 0.06 0.00
Estd Interconnect traffic (KB) 68.60
Global Cache Efficiency Percentages (Target local+remote 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer access - local cache %: 99.91
Buffer access - remote cache %: 0.01
Buffer access - disk %: 0.08
Global Cache and Enqueue Services - Workload Characteristics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg global enqueue get time (ms): 0.5
Avg global cache cr block receive time (ms): 0.9
Avg global cache current block receive time (ms): 1.0
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.1
Global cache log flushes for cr blocks served %: 2.9
Avg global cache cr block flush time (ms): 4.6
Avg global cache current block pin time (ms): 0.0
Avg global cache current block send time (ms): 0.1
Global cache log flushes for current blocks served %: 0.1
Avg global cache current block flush time (ms): 5.0
Global Cache and Enqueue Services - Messaging Statistics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg message sent queue time (ms): 0.1
Avg message sent queue time on ksxp (ms): 0.6
Avg message received queue time (ms): 0.0
Avg GCS message process time (ms): 0.0
Avg GES message process time (ms): 0.1
% of direct sent messages: 31.57
% of indirect sent messages: 5.17
% of flow controlled messages: 63.26
Time Model Statistics DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> Total time in database user-calls (DB Time): 13984.6s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 7,270.6 52.0
DB CPU 4,503.1 32.2
parse time elapsed 506.7 3.6
hard parse elapsed time 497.8 3.6
sequence load elapsed time 152.4 1.1
failed parse elapsed time 19.5 .1
repeated bind elapsed time 3.4 .0
PL/SQL execution elapsed time 0.7 .0
hard parse (sharing criteria) elapsed time 0.3 .0
connection management call elapsed time 0.3 .0
hard parse (bind mismatch) elapsed time 0.0 .0
DB time 13,984.6 N/A
background elapsed time 869.1 N/A
background cpu time 276.6 N/A
Wait Class DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
System I/O 529,934 .0 4,980 9 3.0
Other 582,349 37.4 4,611 8 3.3
Network 279,858 .0 1,009 4 1.6
User I/O 54,899 .0 317 6 0.3
Concurrency 136,907 .1 58 0 0.8
Cluster 60,300 .0 41 1 0.3
Commit 80 .0 10 130 0.0
Application 6,707 .0 3 0 0.0
Configuration 17,528 98.5 1 0 0.1
Wait Events DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
RFS ping 168 .0 4,275 25449 0.0
log file sequential read 183,537 .0 4,173 23 1.0
SQL*Net more data from clien 21,371 .0 1,009 47 0.1
RFS write 25,438 .0 343 13 0.1
db file sequential read 54,680 .0 316 6 0.3
DFS lock handle 97,149 .0 214 2 0.5
log file parallel write 104,808 .0 157 2 0.6
db file parallel write 143,905 .0 149 1 0.8
RFS random i/o 25,438 .0 86 3 0.1
RFS dispatch 25,610 .0 56 2 0.1
control file sequential read 39,309 .0 55 1 0.2
row cache lock 130,665 .0 47 0 0.7
gc current grant 2-way 35,498 .0 23 1 0.2
wait for scn ack 50,872 .0 20 0 0.3
enq: WL - contention 6,156 .0 14 2 0.0
gc cr grant 2-way 16,917 .0 11 1 0.1
log file sync 80 .0 10 130 0.0
Log archive I/O 3,986 .0 9 2 0.0
control file parallel write 3,493 .0 8 2 0.0
latch free 2,356 .0 6 2 0.0
ksxr poll remote instances 278,473 49.4 6 0 1.6
enq: XR - database force log 2,890 .0 4 1 0.0
enq: TX - index contention 325 .0 3 11 0.0
buffer busy waits 4,371 .0 3 1 0.0
gc current block 2-way 3,002 .0 3 1 0.0
LGWR wait for redo copy 9,601 .2 2 0 0.1
SQL*Net break/reset to clien 6,438 .0 2 0 0.0
latch: ges resource hash lis 23,223 .0 2 0 0.1
enq: WF - contention 32 6.3 2 62 0.0
enq: FB - contention 660 .0 2 2 0.0
enq: PS - contention 1,088 .0 2 1 0.0
library cache lock 869 .0 1 2 0.0
enq: CF - contention 671 .1 1 2 0.0
gc current grant busy 1,488 .0 1 1 0.0
gc current multi block reque 1,072 .0 1 1 0.0
reliable message 618 .0 1 2 0.0
CGS wait for IPC msg 62,402 100.0 1 0 0.4
gc current block 3-way 998 .0 1 1 0.0
name-service call wait 18 .0 1 57 0.0
cursor: pin S wait on X 78 100.0 1 11 0.0
os thread startup 16 .0 1 53 0.0
enq: RO - fast object reuse 193 .0 1 3 0.0
IPC send completion sync 652 99.2 1 1 0.0
local write wait 194 .0 1 3 0.0
gc cr block 2-way 534 .0 0 1 0.0
log file switch completion 17 .0 0 20 0.0
SQL*Net message to client 258,483 .0 0 0 1.5
undo segment extension 17,282 99.9 0 0 0.1
gc cr block 3-way 286 .7 0 1 0.0
enq: TM - contention 76 .0 0 4 0.0
PX Deq: reap credit 15,246 95.6 0 0 0.1
kksfbc child completion 5 100.0 0 49 0.0
enq: TT - contention 141 .0 0 2 0.0
enq: HW - contention 203 .0 0 1 0.0
RFS create 2 .0 0 115 0.0
rdbms ipc reply 339 .0 0 1 0.0
PX Deq Credit: send blkd 452 20.1 0 0 0.0
gcs log flush sync 128 32.8 0 2 0.0
latch: cache buffers chains 128 .0 0 1 0.0
library cache pin 441 .0 0 0 0.0
Wait Events DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)We only apply on one node in a cluster so I would expect that the node running SQL Apply would have much higher usage and waits. Is this what you are asking?
Larry -
Performance issue showing read by other session Event
Hi All,
we are having a severe performance issue in my database when we are running batch jobs.
This was a new database(11.2.0.2) and we are testing the performance by running some batch jobs. These batch jobs included some inserts and updates.
I am seeing read by other session in top 5 timed events and cache buffers chains in Latch Miss Sources section.
Please help me to solve this out.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5605 29-Feb-12 03:00:27 63 4.5
End Snap: 5614 29-Feb-12 12:00:47 63 4.3
Elapsed: 540.32 (mins)
DB Time: 1,774.23 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 3.3 0.8 0.02 0.05
DB CPU(s): 1.1 0.3 0.01 0.02
Redo size: 55,763.8 13,849.3
Logical reads: 23,906.6 5,937.4
Block changes: 325.7 80.9
Physical reads: 665.6 165.3
Physical writes: 40.4 10.0
User calls: 60.7 15.1
Parses: 10.6 2.6
Hard parses: 1.1 0.3
W/A MB processed: 0.6 0.2
Logons: 0.1 0.0
Executes: 151.2 37.6
Rollbacks: 0.0 0.0
Transactions: 4.0
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.94 Redo NoWait %: 100.00
Buffer Hit %: 97.90 In-memory Sort %: 100.00
Library Hit %: 98.06 Soft Parse %: 90.16
Execute to Parse %: 92.96 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 76.71 % Non-Parse CPU: 98.57
Shared Pool Statistics Begin End
Memory Usage %: 89.38 87.96
% SQL with executions>1: 97.14 95.15
% Memory for SQL w/exec>1: 96.05 92.46
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 14,092,706 65,613 5 61.6 User I/O
DB CPU 34,819 32.7
read by other session 308,534 1,260 4 1.2 User I/O
direct path read 97,454 987 10 .9 User I/O
db file scattered read 71,870 910 13 .9 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
0.43 0.36 13.7 0.6 9.7 85.7
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.5
% of busy CPU for Instance: 94.2
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 506.5 629.1
% Host Mem used for SGA+PGA: 7.28 7.53
Time Model Statistics
-> Total time in database user-calls (DB Time): 106453.8s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 105,531.1 99.1
DB CPU 34,818.8 32.7
parse time elapsed 714.7 .7
hard parse elapsed time 684.8 .6
PL/SQL execution elapsed time 161.9 .2
PL/SQL compilation elapsed time 44.2 .0
connection management call elapsed time 16.9 .0
hard parse (sharing criteria) elapsed time 10.2 .0
hard parse (bind mismatch) elapsed time 9.4 .0
sequence load elapsed time 2.9 .0
repeated bind elapsed time 0.5 .0
failed parse elapsed time 0.0 .0
DB time 106,453.8
background elapsed time 1,753.9
background cpu time 61.7
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 3,704,415
IDLE_TIME 22,203,740
IOWAIT_TIME 2,517,864
NICE_TIME 3
SYS_TIME 145,696
USER_TIME 3,557,758
LOAD 0 0
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 358,813,045,760
VM_OUT_BYTES 29,514,830,848
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 03:00:27 0.4 N/A N/A N/A N/A N/A
29-Feb 04:00:35 1.4 11.9 11.2 0.6 88.1 14.3
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
29-Feb 06:00:48 1.5 14.0 13.5 0.6 86.0 12.3
29-Feb 07:01:00 1.8 16.3 15.8 0.5 83.7 10.4
29-Feb 08:00:12 2.6 23.2 22.5 0.6 76.8 12.6
29-Feb 09:00:26 1.3 16.6 16.0 0.5 83.4 5.7
29-Feb 10:00:33 1.2 13.8 13.3 0.5 86.2 2.0
29-Feb 11:00:43 1.3 14.5 14.0 0.5 85.5 3.8
29-Feb 12:00:47 0.4 4.9 4.2 0.7 95.1 10.6
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.9% of Total DB time 106,453.79 (s)
-> Total FG Wait Time: 69,415.64 (s) DB CPU time: 34,818.79 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 14,693,843 0 69,222 5 65.0
DB CPU 34,819 32.7
Commit 40,629 0 119 3 0.1
System I/O 26,504 0 57 2 0.1
Network 1,945,010 0 11 0 0.0
Other 125,200 99 4 0 0.0
Application 2,673 0 2 1 0.0
Concurrency 3,059 0 1 0 0.0
Configuration 31 19 0 15 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 14,092,706 0 65,613 5 108.0 61.6
read by other session 308,534 0 1,260 4 2.4 1.2
direct path read 97,454 0 987 10 0.7 .9
db file scattered read 71,870 0 910 13 0.6 .9
db file parallel read 35,001 0 372 11 0.3 .3
log file sync 40,629 0 119 3 0.3 .1
control file sequential re 26,504 0 57 2 0.2 .1
direct path read temp 14,499 0 49 3 0.1 .0
direct path write temp 9,186 0 28 3 0.1 .0
SQL*Net message to client 1,923,973 0 5 0 14.7 .0
SQL*Net message from dblin 1,056 0 5 5 0.0 .0
Disk file operations I/O 8,848 0 2 0 0.1 .0
ASM file metadata operatio 36 0 2 54 0.0 .0
SQL*Net break/reset to cli 2,636 0 1 1 0.0 .0
ADR block file read 472 0 1 1 0.0 .0
os thread startup 8 0 1 74 0.0 .0
SQL*Net more data to clien 17,656 0 1 0 0.1 .0
asynch descriptor resize 123,852 100 0 0 0.9 .0
local write wait 110 0 0 4 0.0 .0
utl_file I/O 55,635 0 0 0 0.4 .0
log file switch (private s 8 0 0 52 0.0 .0
cursor: pin S wait on X 2 0 0 142 0.0 .0
enq: KO - fast object chec 13 0 0 20 0.0 .0
PX Deq: Slave Session Stat 248 0 0 1 0.0 .0
enq: RO - fast object reus 18 0 0 11 0.0 .0
latch: cache buffers chain 2,511 0 0 0 0.0 .0
latch: shared pool 195 0 0 1 0.0 .0
CSS initialization 12 0 0 8 0.0 .0
PX qref latch 54 100 0 2 0.0 .0
SQL*Net more data from cli 995 0 0 0 0.0 .0
SQL*Net more data from dbl 300 0 0 0 0.0 .0
kksfbc child completion 1 100 0 56 0.0 .0
library cache: mutex X 244 0 0 0 0.0 .0
PX Deq: Signal ACK RSG 124 0 0 0 0.0 .0
undo segment extension 6 100 0 7 0.0 .0
PX Deq: Signal ACK EXT 124 0 0 0 0.0 .0
library cache load lock 3 0 0 9 0.0 .0
ADR block file write 45 0 0 1 0.0 .0
CSS operation: action 12 0 0 2 0.0 .0
reliable message 28 0 0 1 0.0 .0
CSS operation: query 72 0 0 0 0.0 .0
latch: row cache objects 14 0 0 1 0.0 .0
enq: SQ - contention 17 0 0 0 0.0 .0
latch free 32 0 0 0 0.0 .0
buffer busy waits 52 0 0 0 0.0 .0
enq: PS - contention 16 0 0 0 0.0 .0
enq: TX - row lock content 6 0 0 1 0.0 .0
SQL*Net message to dblink 1,018 0 0 0 0.0 .0
cursor: pin S 23 0 0 0 0.0 .0
latch: cache buffers lru c 8 0 0 0 0.0 .0
SQL*Net message from clien 1,923,970 0 944,508 491 14.7
jobq slave wait 66,732 100 33,334 500 0.5
Streams AQ: waiting for me 6,481 100 32,412 5001 0.0
wait for unread message on 32,858 98 32,411 986 0.3
PX Deq: Execution Msg 1,448 0 190 131 0.0
PX Deq: Execute Reply 1,196 0 74 62 0.0
HS message to agent 228 0 4 19 0.0
single-task message 42 0 4 97 0.0
PX Deq Credit: send blkd 904 0 2 3 0.0
PX Deq Credit: need buffer 205 0 1 3 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
PX Deq: Table Q Normal 4,291 0 1 0 0.0
PX Deq: Join ACK 124 0 0 1 0.0
PX Deq: Parse Reply 124 0 0 0 0.0
KSV master wait 256 0 0 0 0.0
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
ASM map operation freeli kffmTranslate2 0 2 0
DML lock allocation ktadmc 0 2 0
FOB s.o list latch ksfd_allfob 0 2 2
In memory undo latch ktiFlushMe 0 5 0
In memory undo latch ktichg: child 0 3 0
PC and Classifier lists No latch 0 6 0
Real-time plan statistic keswxAddNewPlanEntry 0 20 20
SQL memory manager worka qesmmIRegisterWorkArea:1 0 1 1
active service list kswslogon: session logout 0 23 12
active service list kswssetsvc: PX session swi 0 6 1
active service list kswsite: service iterator 0 1 0
archive process latch kcrrgpll 0 3 3
cache buffers chains kcbgtcr_2 0 1,746 573
cache buffers chains kcbgtcr: fast path (cr pin 0 1,024 2,126
cache buffers chains kcbgcur_2 0 60 8
cache buffers chains kcbchg1: kslbegin: bufs no 0 16 3
cache buffers chains kcbgtcr: fast path 0 14 20
cache buffers chains kcbzibmlt: multi-block rea 0 10 0
cache buffers chains kcbrls_2 0 9 53
cache buffers chains kcbgtcr: kslbegin shared 0 8 1
cache buffers chains kcbrls_1 0 7 84
cache buffers chains kcbgtcr: kslbegin excl 0 6 14
cache buffers chains kcbnew: new latch again 0 6 0
cache buffers chains kcbzgb: scan from tail. no 0 6 0
cache buffers chains kcbzwb 0 5 8
cache buffers chains kcbgcur: fast path (shr) 0 3 0
cache buffers chains kcbget: pin buffer 0 3 0
cache buffers chains kcbzhngcbk2_1 0 1 0
cache buffers lru chain kcbzgws 0 19 0
cache buffers lru chain kcbo_link_q 0 3 0
call allocation ksuxds 0 14 10
call allocation ksudlp: top call 0 2 3
enqueue hash chains ksqgtl3 0 2 1
enqueue hash chains ksqrcl 0 1 2
enqueues ksqgel: create enqueue 0 1 0
object queue header oper kcbo_unlink_q 0 5 2
object queue header oper kcbo_sw_buf 0 2 0
object queue header oper kcbo_link_q 0 1 2
object queue header oper kcbo_switch_cq 0 1 2
object queue header oper kcbo_switch_mq_bg 0 1 4
parallel query alloc buf kxfpbalo 0 1 1
process allocation ksucrp:1 0 2 0
process queue reference kxfpqrsnd 0 1 0
qmn task queue latch kwqmnmvtsks: delay to read 0 1 0
redo allocation kcrfw_redo_gen: redo alloc 0 17 0
row cache objects kqreqd: reget 0 6 0
row cache objects kqrpre: find obj 0 6 13
row cache objects kqrso 0 2 0
row cache objects kqreqd 0 1 2
row cache objects kqrpre: init complete 0 1 1
shared pool kghalo 0 199 106
shared pool kghupr1 0 39 109
shared pool kghfre 0 18 19
shared pool kghalp 0 7 29
space background task la ktsj_grab_task 0 21 27
Mutex Sleep Summary
-> ordered by number of sleeps desc
Wait
Mutex Type Location Sleeps Time (ms)
Library Cache kglhdgn2 106 338 12
Library Cache kgllkc1 57 259 10
Library Cache kgllkdl1 85 123 21
Cursor Pin kkslce [KKSCHLPIN2] 70 286
Library Cache kglget2 2 31 1
Library Cache kglhdgn1 62 31 2
Library Cache kglpin1 4 26 1
Library Cache kglpnal1 90 18 0
Library Cache kglpndl1 95 15 2
Library Cache kgllldl2 112 6 0
Library Cache kglini1 32 1 0
-------------------------------------------------------------Thanks in advance.Hi,
Thanks for reply.
I provided one hour report.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5606 29-Feb-12 04:00:35 63 3.7
End Snap: 5607 29-Feb-12 05:00:41 63 3.6
Elapsed: 60.11 (mins)
DB Time: 382.67 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 6.4 0.8 0.03 0.03
DB CPU(s): 1.0 0.1 0.00 0.00
Redo size: 84,539.3 10,425.6
Logical reads: 23,345.6 2,879.1
Block changes: 386.5 47.7
Physical reads: 1,605.0 197.9
Physical writes: 7.1 0.9
User calls: 233.9 28.9
Parses: 4.0 0.5
Hard parses: 0.1 0.0
W/A MB processed: 0.1 0.0
Logons: 0.1 0.0
Executes: 210.9 26.0
Rollbacks: 0.0 0.0
Transactions: 8.1
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.62 Redo NoWait %: 100.00
Buffer Hit %: 95.57 In-memory Sort %: 100.00
Library Hit %: 99.90 Soft Parse %: 98.68
Execute to Parse %: 98.10 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 32.08 % Non-Parse CPU: 99.90
Shared Pool Statistics Begin End
Memory Usage %: 89.25 89.45
% SQL with executions>1: 96.79 97.52
% Memory for SQL w/exec>1: 95.67 96.56
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 3,054,464 17,002 6 74.0 User I/O
DB CPU 3,748 16.3
read by other session 199,603 796 4 3.5 User I/O
direct path read 46,301 439 9 1.9 User I/O
db file scattered read 21,113 269 13 1.2 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
1.45 1.67 13.2 0.6 15.8 86.2
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.0
% of busy CPU for Instance: 94.7
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 513.5 467.7
% Host Mem used for SGA+PGA: 7.29 7.20
Time Model Statistics
-> Total time in database user-calls (DB Time): 22960.5s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 22,835.9 99.5
DB CPU 3,748.4 16.3
parse time elapsed 15.4 .1
hard parse elapsed time 14.3 .1
PL/SQL execution elapsed time 7.5 .0
PL/SQL compilation elapsed time 6.0 .0
connection management call elapsed time 1.6 .0
sequence load elapsed time 0.4 .0
hard parse (sharing criteria) elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 22,960.5
background elapsed time 238.1
background cpu time 4.9
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 396,506
IDLE_TIME 2,483,725
IOWAIT_TIME 455,495
NICE_TIME 0
SYS_TIME 16,163
USER_TIME 380,052
LOAD 1 2
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 95,646,943,232
VM_OUT_BYTES 1,686,059,008
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 04:00:35 1.4 N/A N/A N/A N/A N/A
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.6% of Total DB time 22,960.46 (s)
-> Total FG Wait Time: 18,651.75 (s) DB CPU time: 3,748.35 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 3,327,253 0 18,576 6 80.9
DB CPU 3,748 16.3
Commit 23,882 0 69 3 0.3
System I/O 1,035 0 3 3 0.0
Network 842,393 0 2 0 0.0
Other 10,120 99 0 0 0.0
Configuration 3 0 0 58 0.0
Application 264 0 0 1 0.0
Concurrency 1,482 0 0 0 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 3,054,464 0 17,002 6 104.5 74.0
read by other session 199,603 0 796 4 6.8 3.5
direct path read 46,301 0 439 9 1.6 1.9
db file scattered read 21,113 0 269 13 0.7 1.2
log file sync 23,882 0 69 3 0.8 .3
db file parallel read 4,727 0 68 14 0.2 .3
control file sequential re 1,035 0 3 3 0.0 .0
SQL*Net message to client 840,792 0 2 0 28.8 .0
direct path read temp 95 0 2 18 0.0 .0
local write wait 79 0 0 4 0.0 .0
Disk file operations I/O 870 0 0 0 0.0 .0
ASM file metadata operatio 4 0 0 50 0.0 .0
log file switch (private s 3 0 0 58 0.0 .0
ADR block file read 36 0 0 3 0.0 .0
enq: RO - fast object reus 5 0 0 16 0.0 .0
latch: cache buffers chain 1,465 0 0 0 0.1 .0
SQL*Net break/reset to cli 256 0 0 0 0.0 .0
asynch descriptor resize 10,059 100 0 0 0.3 .0
SQL*Net more data to clien 1,510 0 0 0 0.1 .0
enq: KO - fast object chec 3 0 0 8 0.0 .0
SQL*Net more data from cli 91 0 0 0 0.0 .0
latch: shared pool 14 0 0 0 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
reliable message 8 0 0 0 0.0 .0
direct path write temp 1 0 0 2 0.0 .0
SQL*Net message from clien 840,794 0 68,885 82 28.8
jobq slave wait 7,365 100 3,679 499 0.3
Streams AQ: waiting for me 721 100 3,605 5000 0.0
wait for unread message on 3,648 98 3,603 988 0.1
KSV master wait 20 0 0 0 0.0
Background Wait Events
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
log file parallel write 29,353 0 83 3 1.0 34.8
db file parallel write 5,753 0 17 3 0.2 6.9
db file sequential read 1,638 0 15 9 0.1 6.1
control file sequential re 5,142 0 13 2 0.2 5.4
os thread startup 140 0 8 58 0.0 3.4
control file parallel writ 1,440 0 8 6 0.0 3.4
log file sequential read 304 0 8 26 0.0 3.3
db file scattered read 214 0 2 9 0.0 .8
ASM file metadata operatio 1,199 0 1 1 0.0 .3
direct path write 35 0 0 6 0.0 .1
direct path read 41 0 0 5 0.0 .1
kfk: async disk IO 6 0 0 9 0.0 .0
Disk file operations I/O 1,266 0 0 0 0.0 .0
ADR block file read 16 0 0 2 0.0 .0
read by other session 3 0 0 8 0.0 .0
Log archive I/O 2 0 0 10 0.0 .0
log file sync 3 0 0 5 0.0 .0
asynch descriptor resize 341 100 0 0 0.0 .0
CSS initialization 1 0 0 6 0.0 .0
log file single write 4 0 0 1 0.0 .0
latch: redo allocation 3 0 0 1 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
LGWR wait for redo copy 45 0 0 0 0.0 .0
CSS operation: query 6 0 0 0 0.0 .0
CSS operation: action 1 0 0 1 0.0 .0
SQL*Net message to client 420 0 0 0 0.0 .0
rdbms ipc message 47,816 39 61,046 1277 1.6
DIAG idle wait 7,200 100 7,200 1000 0.2
Space Manager: slave idle 1,146 98 5,674 4951 0.0
class slave wait 284 0 3,983 14026 0.0
dispatcher timer 61 100 3,660 60006 0.0
Streams AQ: qmn coordinato 258 50 3,613 14003 0.0
Streams AQ: qmn slave idle 130 0 3,613 27789 0.0
Streams AQ: waiting for ti 7 71 3,608 515430 0.0
wait for unread message on 3,605 100 3,606 1000 0.1
pmon timer 1,201 100 3,604 3001 0.0
smon timer 15 73 3,603 240207 0.0
ASM background timer 754 0 3,602 4777 0.0
shared server idle wait 120 100 3,601 30006 0.0
SQL*Net message from clien 554 0 4 7 0.0
KSV master wait 101 0 0 2 0.0
Wait Event Histogram
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 52 73.1 1.9 9.6 13.5 1.9
ADR block file write 10 100.0
ADR file lock 12 100.0
ARCH wait for archivelog l 3 100.0
ASM file metadata operatio 1203 97.3 .5 .7 .3 .2 .9
CSS initialization 1 100.0
CSS operation: action 1 100.0
CSS operation: query 6 83.3 16.7
Disk file operations I/O 2118 95.4 4.5 .1
LGWR wait for redo copy 45 100.0
Log archive I/O 2 100.0
SQL*Net break/reset to cli 256 99.6 .4
SQL*Net message to client 839.9 100.0 .0
SQL*Net more data from cli 91 100.0
SQL*Net more data to clien 1503 100.0
asynch descriptor resize 10.4K 100.0
buffer busy waits 2 100.0
control file parallel writ 1440 5.7 35.1 24.0 16.3 12.0 5.5 1.5
control file sequential re 6177 69.4 7.5 5.9 8.1 7.1 1.7 .3
db file parallel read 4727 1.7 3.2 3.2 10.1 46.6 33.3 1.8
db file parallel write 5755 42.3 21.3 18.6 11.2 4.6 1.4 .5
db file scattered read 21.5K 8.4 4.3 11.9 18.9 26.3 25.3 4.9
db file sequential read 3053. 28.7 15.1 11.1 17.9 21.5 5.4 .3 .0
direct path read 46.3K 9.9 8.8 18.5 21.7 22.8 15.7 2.7
direct path read temp 95 9.5 9.5 23.2 49.5 8.4
direct path write 35 11.4 31.4 17.1 22.9 11.4 2.9 2.9
direct path write temp 1 100.0
enq: KO - fast object chec 3 66.7 33.3
enq: RO - fast object reus 5 20.0 20.0 20.0 20.0 20.0
kfk: async disk IO 6 50.0 16.7 16.7 16.7
latch free 3 100.0
latch: cache buffers chain 1465 100.0
latch: cache buffers lru c 1 100.0
latch: object queue header 2 100.0
latch: redo allocation 3 33.3 33.3 33.3
latch: row cache objects 2 100.0
latch: shared pool 15 93.3 6.7
local write wait 79 35.4 34.2 21.5 8.9
log file parallel write 29.4K 47.8 21.7 11.9 9.9 6.8 1.6 .3
log file sequential read 304 6.3 3.0 3.6 10.2 23.4 24.3 29.3
log file single write 4 25.0 75.0
log file switch (private s 3 100.0
log file sync 23.9K 40.9 28.0 12.9 9.7 6.7 1.5 .3
os thread startup 140 100.0
read by other session 199.6 37.1 19.9 12.9 13.1 13.8 3.1 .2
reliable message 8 100.0
ASM background timer 755 2.9 .4 .1 .1 .3 .1 .3 95.8
DIAG idle wait 7196 100.0
KSV master wait 121 88.4 2.5 3.3 2.5 .8 .8 1.7
SQL*Net message from clien 840.1 97.1 1.8 .5 .2 .2 .1 .0 .1
Space Manager: slave idle 1147 .1 .5 99.4
Streams AQ: qmn coordinato 258 49.6 .4 50.0
Streams AQ: qmn slave idle 130 .8 99.2
Streams AQ: waiting for me 721 100.0
Streams AQ: waiting for ti 7 28.6 42.9 28.6
class slave wait 283 39.9 2.5 2.5 3.5 4.9 9.2 15.2 22.3
dispatcher timer 60 100.0
jobq slave wait 7360 .0 .0 .0 99.9
pmon timer 1201 100.0
rdbms ipc message 47.8K 2.7 31.6 17.4 1.1 1.1 .9 20.9 24.3
Wait Event Histogram DB/Inst: I2KPROD/I2KPROD Snaps: 5606-5607
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
shared server idle wait 120 100.0
smon timer 16 6.3 93.8
wait for unread message on 7250 .1 99.9
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
In memory undo latch ktichg: child 0 1 0
active service list kswslogon: session logout 0 2 0
cache buffers chains kcbgtcr_2 0 1,123 483
cache buffers chains kcbgtcr: fast path (cr pin 0 496 1,131
cache buffers chains kcbrls_2 0 5 6
cache buffers chains kcbgcur_2 0 4 0
cache buffers chains kcbgtcr: fast path 0 3 1
cache buffers chains kcbzwb 0 2 4
cache buffers chains kcbchg1: kslbegin: bufs no 0 1 0
cache buffers chains kcbnew: new latch again 0 1 0
cache buffers chains kcbrls_1 0 1 6
cache buffers chains kcbzgb: scan from tail. no 0 1 0
cache buffers lru chain kcbzgws 0 1 0
object queue header oper kcbo_switch_cq 0 1 0
object queue header oper kcbo_switch_mq_bg 0 1 2
redo allocation kcrfw_redo_gen: redo alloc 0 3 0
row cache objects kqrpre: find obj 0 1 1
row cache objects kqrso 0 1 0
shared pool kghalo 0 13 3
shared pool kghupr1 0 4 15
shared pool kghalp 0 1 0
space background task la ktsj_grab_task 0 2 2
------------------------------------------------------------- -
45 min long session of log file sync waits between 5000 and 20000 ms
45 min long log file sync waits between 5000 and 20000 ms
Encountering a rather unusual performance issue. Once every 4 hours I am seeing a 45 minute long log file sync wait event being reported using Spotlight on Oracle. For the first 30 minutes the event wait is for approx 5000 ms, followed by an increase to around 20000 ms for the next 15 min before rapidly dropping off and normal operation continues for the next 3 hours and 15 minutes before the cycle repeats itself. The issue appears to maintain it's schedule independently of restarting the database. Statspack reports do not show an increase in commits or executions or any new sql running during the time the issue is occuring. We have two production environments both running identicle applications with similar usage and we do not see the issue on the other system. I am leaning towards this being a hardware issue, but the 4 hour interval regardless of load on the database has me baffled. If it were a disk or controller cache issue one would expect to see the interval change with database load.
I cycle my redo logs and archive them just fine with log file switches every 15-20 minutes. Even during this unusally long and high session of log file sync waits I can see that the redo log files are still switching and are being archived.
The redo logs are on a RAID 10, we have 4 redo logs at 1 GB each.
I've run statspack reports on hourly intervals around this event:
Top 5 Wait Events
~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
log file sync 756,729 2,538,034 88.47
db file sequential read 208,851 153,276 5.34
log file parallel write 636,648 129,981 4.53
enqueue 810 21,423 .75
log file sequential read 65,540 14,480 .50
And here is a sample while not encountering the issue:
Top 5 Wait Events
~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
log file sync 953,037 195,513 53.43
log file parallel write 875,783 83,119 22.72
db file sequential read 221,815 63,944 17.48
log file sequential read 98,310 18,848 5.15
db file scattered read 67,584 2,427 .66
Yes I know I am already tight on I/O for my redo even during normal operations yet, my redo and archiving works just fine for 3 hours and 15 minutes (11 to 15 log file switches). These normal switches result in a log file sync wait of about 5000 ms for about 45 seconds while the 1GB redo log is being written and then archived.
I welcome any and all feedback.
Message was edited by:
acyoung1
Message was edited by:
acyoung1Lee,
log_buffer = 1048576 we use a standard of 1 MB for our buffer cache, we've not altered the setting. It is my understanding that Oracle typically recommends that you not exceed 1MB for the log_buffer, stating that a larger buffer normally does not increase performance.
I would agree that tuning the log_buffer parameter may be a place to consider; however, this issue last for ~45 minutes once every 4 hours regardless of database load. So for 3 hours and 15 minutes during both peak usage and low usage the buffer cache, redo log and archival processes run just fine.
A bit more information from statspack reports:
Here is a sample while the issue is occuring.
Snap Id Snap Time Sessions
Begin Snap: 661 24-Mar-06 12:45:08 87
End Snap: 671 24-Mar-06 13:41:29 87
Elapsed: 56.35 (mins)
Cache Sizes
~~~~~~~~~~~
db_block_buffers: 196608 log_buffer: 1048576
db_block_size: 8192 shared_pool_size: 67108864
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 615,141.44 2,780.83
Logical reads: 13,241.59 59.86
Block changes: 2,255.51 10.20
Physical reads: 144.56 0.65
Physical writes: 61.56 0.28
User calls: 1,318.50 5.96
Parses: 210.25 0.95
Hard parses: 8.31 0.04
Sorts: 16.97 0.08
Logons: 0.14 0.00
Executes: 574.32 2.60
Transactions: 221.21
% Blocks changed per Read: 17.03 Recursive Call %: 26.09
Rollback per transaction %: 0.03 Rows per Sort: 46.87
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 98.91 In-memory Sort %: 100.00
Library Hit %: 98.89 Soft Parse %: 96.05
Execute to Parse %: 63.39 Latch Hit %: 99.87
Parse CPU to Parse Elapsd %: 90.05 % Non-Parse CPU: 85.05
Shared Pool Statistics Begin End
Memory Usage %: 89.96 92.20
% SQL with executions>1: 76.39 67.76
% Memory for SQL w/exec>1: 72.53 63.71
Top 5 Wait Events
~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
log file sync 756,729 2,538,034 88.47
db file sequential read 208,851 153,276 5.34
log file parallel write 636,648 129,981 4.53
enqueue 810 21,423 .75
log file sequential read 65,540 14,480 .50
And this is a sample during "normal" operation.
Snap Id Snap Time Sessions
Begin Snap: 671 24-Mar-06 13:41:29 88
End Snap: 681 24-Mar-06 14:42:57 88
Elapsed: 61.47 (mins)
Cache Sizes
~~~~~~~~~~~
db_block_buffers: 196608 log_buffer: 1048576
db_block_size: 8192 shared_pool_size: 67108864
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 716,776.44 2,787.81
Logical reads: 13,154.06 51.16
Block changes: 2,627.16 10.22
Physical reads: 129.47 0.50
Physical writes: 67.97 0.26
User calls: 1,493.74 5.81
Parses: 243.45 0.95
Hard parses: 9.23 0.04
Sorts: 18.27 0.07
Logons: 0.16 0.00
Executes: 664.05 2.58
Transactions: 257.11
% Blocks changed per Read: 19.97 Recursive Call %: 25.87
Rollback per transaction %: 0.02 Rows per Sort: 46.85
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 99.02 In-memory Sort %: 100.00
Library Hit %: 98.95 Soft Parse %: 96.21
Execute to Parse %: 63.34 Latch Hit %: 99.90
Parse CPU to Parse Elapsd %: 96.60 % Non-Parse CPU: 84.06
Shared Pool Statistics Begin End
Memory Usage %: 92.20 88.73
% SQL with executions>1: 67.76 75.40
% Memory for SQL w/exec>1: 63.71 68.28
Top 5 Wait Events
~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
log file sync 953,037 195,513 53.43
log file parallel write 875,783 83,119 22.72
db file sequential read 221,815 63,944 17.48
log file sequential read 98,310 18,848 5.15
db file scattered read 67,584 2,427 .66
Maybe you are looking for
-
Use of warehouse for 'as of' balance custom fields
we captured custom fields on each account/contact for current balance, current deposits, current trade volume we are updating these fields daily based on the transactional system integration how/will i be able to report on the day over day #'s or mon
-
Authorization Group Material Master
Hello, i am trying to restrict a user from accessing a single material. I am using Authorization Object M_MATE_MAT for this. When I use transaction mm03 to go on the material master I see a field for the Authorization Group but when I go to the same
-
Updating iphoto to 6.0 on other mac from new install disk?
Hi, I have a new mac book and would like to install iphoto 6.0 on my older mac. Can I simply use the newer install disk to get the latest version of iphoto? Or will I have to now buy a full package of iLife? Thanks in advance
-
Greetings: I have been given an xml file that originated in a low cost xml editor. When I open the document in Frame 9, the document has extraneous spaces before and after various tags. The most bothersome are those that appear after start tags. For
-
How do you change 64 bit to 32 bit
How do you change the setting of 64 bit to 32 bit? My Norton program can't run everything on 64.