High enqueue wait
The oracle version is 9.2.0.7. The total memory of machine is 16g. I change sga max size from 2g to 7g. I changed the db cache size from 1.5g to 6g because i was getting law buffer hit.I increased processes parameter to 1000 from 700 and sessions parameter to 1000 from 700.Now I am seeing the high enqueue wait in the database like 90% in statspack.
What could be the reason for this
There are no rows return from those queries
But i got this
select * from v$enqueue_stat where cum_wait_time>0
order by cum_wait_time desc;
INST_ID EQ TOTAL_REQ# TOTAL_WAIT# SUCC_REQ#
FAILED_REQ# CUM_WAIT_TIME
1 TX 310179 8293 310175
4 76761677
TC 130 25 130 0
526
US 21407 7 21407 0
97
SQ 14093 217 14093 0
78
HW 9835 13 9835 0
40
CF 8901 2 8901 0
29Well, the result says that your enqueue waits were "TX" type.
Followings are genercal cases of TX lock contention:
- Row lock contention - Row update confliction, PK confliction, Bitmap Index confliction, ...
- ITL entry contention
- Index split contention
- Misc
Unfortunately, TX lock contention generally has no relationship with buffer cache size. I don't think your increased enqueue wait was caused by bigger buffer cache. TX lock contention means that multiple transactions are modifying/accessing same data.
You need to capture V$SESSION_WAIT, V$SESSION and V$SQL(as i said) just when the enqueue wait happens. More info will tell you more reasonable explanation.
Similar Messages
-
High Enqueue Waits in Statspack Report
Hi Everybody,
Oracle Version:Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
OS:Solaris 64 bit
Statspack Report is showing High Enqueue Waits
Here is a Snapshot.....
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
XXXXX 434917312 XXXXX 1 9.2.0.7.0 NO INgenius1
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 1064 24-Sep-08 06:00:01 1,333 19.6
End Snap: 1065 24-Sep-08 07:00:01 1,344 19.7
Elapsed: 60.00 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 1,152M Std Block Size: 8K
Shared Pool Size: 752M Log Buffer: 1,536K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 27,771.53 1,196.23
Logical reads: 777.20 33.48
Block changes: 180.58 7.78
Physical reads: 33.25 1.43
Physical writes: 7.51 0.32
User calls: 76.89 3.31
Parses: 23.29 1.00
Hard parses: 0.27 0.01
Sorts: 0.42 0.02
Logons: 0.05 0.00
Executes: 88.69 3.82
Transactions: 23.22
% Blocks changed per Read: 23.23 Recursive Call %: 61.50
Rollback per transaction %: 0.00 Rows per Sort: 738.76
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 96.90 In-memory Sort %: 99.67
Library Hit %: 99.76 Soft Parse %: 98.84
Execute to Parse %: 73.74 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 94.38 % Non-Parse CPU: 94.45
Shared Pool Statistics Begin End
Memory Usage %: 86.56 87.54
% SQL with executions>1: 50.57 63.10
% Memory for SQL w/exec>1: 10.02 14.63
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
enqueue 19,553 57,405 98.39
db file sequential read 44,161 384 .66
CPU time 333 .57
log file parallel write 166,602 82 .14
log file sync 67,683 71 .12
Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 1065
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,553 19,540 57,405 2936 0.2
db file sequential read 44,161 0 384 9 0.5
log file parallel write 166,602 0 82 0 2.0
log file sync 67,683 0 71 1 0.8
db file scattered read 6,676 0 54 8 0.1
db file parallel write 1,135 0 7 6 0.0
direct path read 1,117 0 3 3 0.0
SQL*Net more data to client 3,932 0 2 0 0.0
control file parallel write 1,200 0 2 1 0.0
control file sequential read 1,389 0 1 1 0.0
PX Deq: Execute Reply 112 0 0 4 0.0
direct path write 752 0 0 1 0.0
db file parallel read 9 0 0 42 0.0
Background Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 10
-> ordered by wait time desc, waits desc (idle events last)
Avg Wt Wait
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
TC 25 24 0 4 32.00 0
TX 84,615 84,605 0 3 8.33 0
HW 118 118 0 2 2.00 0
PS 29 25 4 2 1.00 0
Here frm Statspack Report we can see that enqueue type- TX is taking up most of the resources.......
I want to find out which sql statements are causing this high enqueue waits......
Any Help Appreciated....
Regards,
Prosenjit MukherjeeHi All,
Here is the Statspack Report..........
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
XXXXX 434917312 XXXXX 1 9.2.0.7.0 NO XXXXXxxx1
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 1064 24-Sep-08 06:00:01 1,333 19.6
End Snap: 1065 24-Sep-08 07:00:01 1,344 19.7
Elapsed: 60.00 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 1,152M Std Block Size: 8K
Shared Pool Size: 752M Log Buffer: 1,536K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 27,771.53 1,196.23
Logical reads: 777.20 33.48
Block changes: 180.58 7.78
Physical reads: 33.25 1.43
Physical writes: 7.51 0.32
User calls: 76.89 3.31
Parses: 23.29 1.00
Hard parses: 0.27 0.01
Sorts: 0.42 0.02
Logons: 0.05 0.00
Executes: 88.69 3.82
Transactions: 23.22
% Blocks changed per Read: 23.23 Recursive Call %: 61.50
Rollback per transaction %: 0.00 Rows per Sort: 738.76
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 96.90 In-memory Sort %: 99.67
Library Hit %: 99.76 Soft Parse %: 98.84
Execute to Parse %: 73.74 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 94.38 % Non-Parse CPU: 94.45
Shared Pool Statistics Begin End
Memory Usage %: 86.56 87.54
% SQL with executions>1: 50.57 63.10
% Memory for SQL w/exec>1: 10.02 14.63
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
enqueue 19,553 57,405 98.39
db file sequential read 44,161 384 .66
CPU time 333 .57
log file parallel write 166,602 82 .14
log file sync 67,683 71 .12
Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 1065
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,553 19,540 57,405 2936 0.2
db file sequential read 44,161 0 384 9 0.5
log file parallel write 166,602 0 82 0 2.0
log file sync 67,683 0 71 1 0.8
db file scattered read 6,676 0 54 8 0.1
db file parallel write 1,135 0 7 6 0.0
direct path read 1,117 0 3 3 0.0
SQL*Net more data to client 3,932 0 2 0 0.0
control file parallel write 1,200 0 2 1 0.0
control file sequential read 1,389 0 1 1 0.0
PX Deq: Execute Reply 112 0 0 4 0.0
direct path write 752 0 0 1 0.0
db file parallel read 9 0 0 42 0.0
process startup 6 0 0 44 0.0
SQL*Net break/reset to clien 296 0 0 1 0.0
PX Deq: Signal ACK 3 1 0 33 0.0
latch free 17 4 0 3 0.0
LGWR wait for redo copy 286 0 0 0 0.0
PX Deq: Join ACK 3 0 0 2 0.0
PX Deq: Parse Reply 4 0 0 1 0.0
buffer busy waits 12 0 0 0 0.0
PX Deq Credit: need buffer 4 0 0 0 0.0
PX Deq: Table Q Sample 1 0 0 0 0.0
SQL*Net message from client 204,628 0 4,620,111 22578 2.4
virtual circuit status 120 120 3,516 29297 0.0
PX Idle Wait 753 749 1,470 1953 0.0
SQL*Net more data from clien 20,540 0 3 0 0.2
PX Deq: Execution Msg 128 0 2 13 0.0
SQL*Net message to client 204,628 0 0 0 2.4
Background Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 10
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
log file parallel write 166,634 0 82 0 2.0
db file parallel write 1,134 0 8 7 0.0
control file parallel write 1,200 0 2 1 0.0
control file sequential read 1,327 0 1 1 0.0
db file scattered read 44 0 0 10 0.0
db file sequential read 25 0 0 3 0.0
direct path read 23 0 0 3 0.0
rdbms ipc reply 67 0 0 0 0.0
LGWR wait for redo copy 286 0 0 0 0.0
direct path write 23 0 0 0 0.0
buffer busy waits 2 0 0 0 0.0
rdbms ipc message 86,933 3,437 20,199 232 1.0
pmon timer 1,194 1,194 3,504 2935 0.0
smon timer 17 8 3,239 ###### 0.0
Enqueue activity for DB: XXXXX Instance: XXXXX Snaps: 1064 - 1065
-> Enqueue stats gathered prior to 9i should not be compared with 9i data
-> ordered by Wait Time desc, Waits desc
Avg Wt Wait
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
TC 25 24 0 4 32.00 0
TX 84,615 84,605 0 3 8.33 0
HW 118 118 0 2 2.00 0
PS 29 25 4 2 1.00 0
End of Report (this is not the entire report,only posted part of it,getting error page when trying to post the entire report)
Regards,
Prosenjit Mukherjee -
10.2.0.3 high concurrency wait event
I have a new 64bit windows 10g 10.2.0.3 VM server doing nothing but spinning its wheels. I intalled Oracle on it Friday and when I checked it tonight I see it is getting high concurrency wait events. Looks like every 10 minutes concurrency goes up to about 35.0 (?seconds) and then goes back to 0 - up and down every few minutes. Just enough to hit the warning threashold.
I need this machine for a peoplesoft install on Monday morning.
Anyone have any ideas what this could be? I kinda know what concurrency is but not sure if it could be from hardware or software.
Didn't see any patches/bugs but I will continue to look.
Help, KathieWhat are the wait events as described in v$system_wait and v$session_event?
-
High roll wait time, gui time and processing time
Hi all,
We are having a performance issue with the system at the moment. After running ST03N and STAD, we have determined that we are having abnormally high roll-wait, GUI and processing times.
Here are some results from STAD:
CPU time: 2956ms
Processing time: 4417ms
Roll wait time: 1319ms
GUI Time: 2859ms
Can someone direct me towards finding the problem?
If you need more information please let me know. I would post a screenshot if I was able to.
Thanks,
HLHi,
Regarding Performance Of SAP system it will not be possible to Come to conclusion with the one Single data.
GUI time is basically the time spent in front end i.e Your Network load or such time
Coming to CPU time Can you see ST06 what is the CPU Utilization of your system?
Roll Wait time is the time Spent in buffer if you can share us the OS and Memory configuration.
We can Try to tune the Roll memory ( extended memory tuning) so that we can try to reduce the Roll time.
If possible try to find the Top Dialog Response Time Transaction Is it Standard or for Z Programs.
https://cw.sdn.sap.com/cw/docs/DOC-14387
Go through this link you might get clear idea of performance.
Thanks & Regards,
Balaji.S -
High Database Waits of type "other"
I justed added cpu patch 19 to my 11.1.0.7 windows 2008 database. Now I have a large amount of database waiting of type "other". See below.
Investigate the cause for high "unspecified wait event" waits. Refer to Oracle's "Database Reference" for the description of this wait event.
Action Investigate the cause for high "unspecified wait event" waits in Service "SYS$USERS".
Action Investigate the cause for high "unspecified wait event" waits in Module "DBMS_SCHEDULER"
Could stale statistics cause this and if so what is the correct way to gather system statistics. Do I unlock sys schema , gather data dictionary stats, and then lock sys schema?
Other thoughts what might cause this?
Thanks,
KathieHi,
You should run the following.
DBMS_STATS.GATHER_SYSTEM_STATS (
gathering_mode VARCHAR2 DEFAULT 'NOWORKLOAD',
interval INTEGER DEFAULT NULL,
stattab VARCHAR2 DEFAULT NULL,
statid VARCHAR2 DEFAULT NULL,
statown VARCHAR2 DEFAULT NULL);
dbms_stats.gather_system_stats(gathering_mode=>'start');
dbms_stats.gather_system_stats(gathering_mode=>'stop');
System statistics will start gathering afetr you run the command with start option. System stats gathering will stop and may show them locked after you run stop.
NOWORKLOAD: Will capture characteristics of the I/O system. Gathering may take a few minutes and depends on the size of the database. During this period Oracle will estimate the average read seek time and transfer speed for the I/O system. This mode is suitable for the all workloads. Oracle recommends to run GATHER_SYSTEM_STATS ('noworkload') after creation of the database and tablespaces. To fine tune system statistics for the workload use 'START' and 'STOP' or 'INTERVAL' options. If you gather both 'NOWORKLOAD' and workload specific (statistics collected using 'INTERVAL' or 'START' and 'STOP' ), the workload statistics will be used by optimizer. Collected components: cpuspeednw, ioseektim, iotfrspeed.
INTERVAL: Captures system activity during a specified interval. This works in combination with the interval parameter. You should provide an interval value in minutes, after which system statistics are created or updated in the dictionary or stattab. You can use GATHER_SYSTEM_STATS (gathering_mode=>'STOP') to stop gathering earlier than scheduled. Collected components: maxthr, slavethr, cpuspeed, sreadtim, mreadtim, mbrc.
START | STOP: Captures system activity during specified start and stop times and refreshes the dictionary or stattab with statistics for the elapsed period. Interval value is ignored. Collected components: maxthr, slavethr, cpuspeed, sreadtim, mreadtim, mbrc.
interval
Time, in minutes, to gather statistics. This parameter applies only when gathering_mode='INTERVAL'.
Regards -
HI all,
9.0.1
The users are encountering lots of enqueue waits while running an application.The application updates 2 tables in a single pl/sql block.
select * from v$lock;
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
6A4BCD30 6A4BCD40 2 MR 202 0 4 0
7819 0
6A4BCCE4 6A4BCCF4 2 MR 201 0 4 0
7819 0
6A4BCC98 6A4BCCA8 2 MR 12 0 4 0
7819 0
6A4BCC4C 6A4BCC5C 2 MR 11 0 4 0
7819 0
6A4BCC00 6A4BCC10 2 MR 10 0 4 0
7819 0
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
6A4BCBB4 6A4BCBC4 2 MR 9 0 4 0
7819 0
6A4BCB68 6A4BCB78 2 MR 8 0 4 0
7819 0
6A4BCB1C 6A4BCB2C 2 MR 7 0 4 0
7819 0
6A4BCAD0 6A4BCAE0 2 MR 6 0 4 0
7819 0
6A4BCA84 6A4BCA94 2 MR 5 0 4 0
7819 0
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
6A4BCA38 6A4BCA48 2 MR 4 0 4 0
7819 0
6A4BC9EC 6A4BC9FC 2 MR 3 0 4 0
7819 0
6A4BC9A0 6A4BC9B0 2 MR 2 0 4 0
7819 0
6A4BC954 6A4BC964 2 MR 1 0 4 0
7819 0
6A4BC870 6A4BC880 3 RT 1 0 6 0
7821 0
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
6A4BC8BC 6A4BC8CC 4 KT 3317 0 4 0
7817 0
6A4BC7D8 6A4BC7E8 5 TS 11 1 3 0
7037 0
B5CA4AC8 B5CA4BB0 8 TX 524333 1430 6 0
2079 0
6ABBE34C 6ABBE360 8 TM 32282 0 3 0
2079 0
6ABBE244 6ABBE258 8 TM 32296 0 3 0
2079 0
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
6A4BC908 6A4BC918 8 TX 196637 1411 0 6
2079 0
B5CA4AC8 B5CA4BB0 16 TX 196637 1411 6 0
2227 1
6ABBE2C8 6ABBE2DC 16 TM 32282 0 3 0
2227 0
6ABBE13C 6ABBE150 16 TM 32296 0 3 0
2227 0
6A4BC824 6A4BC834 16 TX 131104 1439 0 6
2227 0
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
B5CA4AC8 B5CA4BB0 18 TX 327709 1446 6 0
1864 0
6ABBE664 6ABBE678 18 TM 32282 0 3 0
1864 0
6ABBE0B8 6ABBE0CC 18 TM 32296 0 3 0
1864 0
6A4BCE14 6A4BCE24 18 TX 196637 1411 0 6
1864 0
B5CA4AC8 B5CA4BB0 24 TX 131104 1439 6 0
2920 1
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
B5CA4AC8 B5CA4BB0 27 TX 262167 1433 6 0
2002 0
6ABBE3D0 6ABBE3E4 27 TM 32282 0 3 0
2002 0
6ABBE1C0 6ABBE1D4 27 TM 32296 0 3 0
2002 0
6A4BCD7C 6A4BCD8C 27 TX 196637 1411 0 6
2002 0
B5CA4AC8 B5CA4BB0 29 TX 65573 1409 6 0
1924 0
.................................................................Press Enter Key
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST
CTIME BLOCK
6ABBE5E0 6ABBE5F4 29 TM 32282 0 3 0
1924 0
6ABBE454 6ABBE468 29 TM 32296 0 3 0
1924 0
6A4BCDC8 6A4BCDD8 29 TX 196637 1411 0 6
1924 0
B5CA4AC8 B5CA4BB0 35 CF 0 3 6 0
2 0
6A4BCE60 6A4BCE70 35 TX 589858 1474 1 0
2 0How to resolve the issue?SQL> SELECT DECODE(request,0,'Holder: ','Waiter: ') ||
sid sess, id1, id2, lmode, request, type
FROM V$LOCK
WHERE (id1, id2, type) IN (SELECT id1, id2, type FROM V$LOCK WHERE request > 0) ORDER BY id1, request; 2 3 4 5
.................................................................Press Enter Key
SESS ID1 ID2
LMODE REQUEST TY
Holder: 24 131104 1439
6 0 TX
Waiter: 16 131104 1439
0 6 TX
Holder: 16 196637 1411
6 0 TX
Waiter: 8 196637 1411
0 6 TX
Waiter: 29 196637 1411
0 6 TX
.................................................................Press Enter Key
SESS ID1 ID2
LMODE REQUEST TY
Waiter: 35 196637 1411
0 6 TX
Waiter: 27 196637 1411
0 6 TX
Waiter: 17 196637 1411
0 6 TX
Waiter: 18 196637 1411
0 6 TXThe users says that they are updating separate rows ie every user is updating different rows.So no concurrent cupdation is going on in any row.
Edited by: MYH on Jan 23, 2009 11:02 PM -
Redo log space requests and Enqueue Waits
Hi all,
I am seeing an increase on the Enqueue Waits and Redo Log Space Request from 58, 274 to 192, 1245 in two weeks time respectively.
The DB is a production database and runs on an HP cluster with 4X1G ram and 550mghz cpu.
There are four Redo Log files with 200M (2 members each)which I have increased to 400M over this past weekend.
I have included below the memory structure details:
Redo Log Summary
Total System Global Area 1646094824 bytes
Fixed Size 104936 bytes
Variable Size 408989696 bytes
Database Buffers 1228800000 bytes
Redo Buffers 8200192 bytes
My question is that, who do I stop it from growing further and passing the 1:5000 ratio ?
At the moment the ratio is in the range of 1:186194.
Your input is much appreciated.
Cheers,
Seyoum.Here is some information from Oracle's Peformance Tuning Guide.
The V$SYSSTAT statistic redo log space requests indicates how many times a server process had to wait for space in the online redo log, not for space in the redo log buffer. A significant value for this statistic and the wait events should be used as an indication that checkpoints, DBWR, or archiver activity should be tuned, not LGWR. Increasing the size of log buffer does not help. -
Trying to diagnose high "Network Wait" in my application
Hello,
I developed a simple standalone application that starts 4 threads. Essentially each thread is a database write operation by calling a stateless session bean (SLSB). The SLSB has a method that make use of Toplink Essentials JPA which is responsible of persisting the objects in the database. I am using an application diagnostic tool. This tool is showing high "Network Wait". For example 50% of the time is being spent on "java.net.SocketException" and finally in the following call "java.net.SocketInputStream -> socketRead0".
1. What is the reason for SocketException?
2. Could it because threads are trying to write at the same time?
I would appreciate if you could share with me any thoughts.
Thanks,
Mustafa50% of the time is being spent on "java.net.SocketException" and finally in the following call "java.net.SocketInputStream -> socketRead0". That means that 50% of the time there is nothing to read. You would expect at least that, more like 90% IMO.
1. What is the reason for SocketException?What SocketException? Did you catch one? What was the text? Where was it thrown from?
2. Could it because threads are trying to write at the same time?Could what be because ...? -
For my below crm transaction for a perticular user the response time is ~18000 ms out of which 15130 is roll time. Memory used is 16 MB.
"BSPWDWCC_SLS_HOME SAPMHTTP /sap/bc/bsp/sap/CRM_UI_PORTAL/B"
Roll time
Out 5 ms
In 5 ms
Wait 15,130 ms
Where is causing this roll wait ? What memory is not sufficient ? What to look at ?
In st02
roll buffer max used is 25% ,
page buffer max used is 10%.
EM max used 50%.
No swaps for above.
My user memory settings are as below:
ztta/roll_area 9000000
ztta/roll_first 1
ztta/roll_extension 4000000000
thank you
LaxmiThank you Shyam,Krishna.
I checked another similar instance again today with high wait time. Most of the time was wait time. There were NO RFC RECORD. just http record with most of it being on the server (Remote exec. time ~ 16 534 ms).
Now i assume calling time = gate way time + remote exec time. So as most of it was on the server I can rule out network.
I checked st03 and rfc times were really small. Also we have plenty of wps .. ~ 90. So not sure why still the high wait time. Also stad does not give any RFC record. If this was rfc will it now show up ?
I have given below the details. Am I missing some thing ?
Response time 16,546 ms
Wait for work process 5 ms
Processing time 1,155 ms
Load time 28 ms
Generating time 0 ms
Roll (in+wait) time 15,016 ms
Database request time 341 ms
Enqueue time 0 ms
Roll time Out 5 ms
In 3 ms
Wait 15,013 ms
NO RFC SUB RECORD
HTTP Records
as Server
Number Connections 1
Destinations 1
Calls 1
Time Calling 16,540 ms
Remote execution 16,534 ms
Idle 219 ms
HTTP Server connection record
Connection-Id 4B7CAADEF5DA00A7E10080000A68F456
Communication step Send/Receive
Timestamp 20100218 155216 EST
Host sappc003
Port 8031
Path /sap/bc/bsp/sap/crm_ui_frame/BSPWDApplication.do
Status phrase OK
Status code 200
Protocol HTTP
Method GET -
High PREEMPTIVE_SP_SERVER_DIAGNOSTICS wait time in sql server 2014 server
I am seeing high waittimes for PREEMPTIVE_SP_SERVER_DIAGNOSTICS and the server is runninf with high cpu.
What could have triggered this high waittime, how can this be fixed,
WaitType Wait_S
Resource_S Signal_S
WaitCount Percentage
AvgWait_S AvgRes_S
AvgSig_S
PREEMPTIVE_SP_SERVER_DIAGNOSTICS 1655043.72
1655043.72 0.00
4 39.75
413760.9305 413760.9305
0.0000
PREEMPTIVE_OS_PIPEOPS 935157.21
935157.21 0.00
343445 22.46
2.7229 2.7229
0.0000
CXPACKET 259912.58
250608.73 9303.85
25115286 6.24
0.0103 0.0100
0.0004
PAGEIOLATCH_SH 206350.14
205563.18 786.96
37649420 4.96
0.0055 0.0055
0.0000
SOS_SCHEDULER_YIELD 139263.91
2430.27 136833.63
293382681 3.35
0.0005 0.0000
0.0005Hello,
I remember I read a user complaining about high waits on PREEMPTIVE_SP_SERVER_DIAGNOSTICS when he installed CU5 for SQL
Server 2012 SP1, but in your scenario you have a SQL Server 2014 with CU1 (build 2342). I am treating this as a bug and that is the reason I would like you to consider applying CU5, the latest cumulative update, to see if that solves the issue.
http://support.microsoft.com/kb/3011055/en-us
This is the first time I read about this happening on SQL Server 2014 though.
I would like to suggest you to create a Connect item too, to see if the SQL Server team can suggest you another way to
treat this issue.
https://connect.microsoft.com/sqlServer/
Hope this helps.
Regards,
Alberto Morillo
SQLCoffee.com -
Hi All,
I find the following in statspack report.
Enqueue activity for DB: FCRLIVE Instance: fcrlive Snaps: 8318 -8319
-> Enqueue stats gathered prior to 9i should not be compared with 9i data
-> ordered by Wait Time desc, Waits desc
Avg Wt Wait
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
PS 33,012 19,048 13,964 2,876 .00 0
Kindly denote how do we interprete this and how do we resolve this??
Regards
VijayVijay Salian wrote:
Hi
Below are the wait events.I more concerned about Failed Gets in Enqueue -PS part which shows 15,187.What does that denote??Secondly how do we resolve it??
Event Waits Time (s) Ela Time
SQL*Net message from dblink 8,190 12,641 88.42
db file scattered read 389,290 858 6.00
db file sequential read 125,058 515 3.60
PX Deq Credit: send blkd 50,441 117 .82
PX Deq: Execute Reply 21,396 71 .50Avg Wt Wait
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
PS 35,991 20,804 *15,187* 3,141 .00 0
Regards
vijayCheck your "Instance statistics" for the number of parallel queries that have been downgraded - (search for downgraded) and show us the results.
Looking at your Top 5 you seem to spend all your time waiting on a remote database - so if you have specific queries that are slower than expected it might something to do with the remote database rather than the behaviour of local PQ slaves. Note - a parallel query that goes distributed HAS to go through a serial step to get remote data - which may make problems with parallel slaves insignificant.
Regards
Jonathan Lewis -
High CPU Waits but medium CPU Usage
Hi All,
I'm currently checking one SQL Server after our users complained about poor application performance.
In the Data Collection Report, I can see that most of the waits come from the CPU. However, the average CPU load is just between 30% and 50%. What could be the cause for this?
The queries which generate most of the CPU wait time are SELECTs to a view with around 30 columns (decimals) and 30.000 row and the INSERTS to the tables behind the view. Since the physical ready are 0, I guess the results come all from memory.
How could I reduce the CPU waits? Thanks!
Greetings,
Theodor>Could you explain why you think that showing CPU utilization as waits is a *good* thing?
In performance analysis you care about reducing the elapsed time. Including CPU use as a wait gives you a unified view of the relative importance of CPU and the other waits. Otherwise you can't know whether to focus on reducing, say lock waits,
IO waits or CPU use.
> I don't see anything to fix!
Yes SOS_SCHEDULER_YEILD and Signal Wait Time are both fine. Those indicate global CPU pressure, ie not having an available CPU to do the work that's ready to be done.
Here the problem is CPU Consumed, which is simply queries chewing up CPU. So look at the queries, find the queries that account for the most CPU and analyze them. Looking for instance for unnecessary table scans on tables in memory, inefficient
joins, running moderately expensive queries too frequently.
I really can't follow that. A wait is when you really are losing elapsed time for external reasons. CPU or reads are what your code really needs to do. Very different categories and should not be added, and that's why they
are tracked separately inside of SQL Server.
OP is asking about 9 millisecond transactions. In most systems I work on, that's already very quick. I've worked on maybe one system in the last few years where the average transaction was faster than that because of the way it was architected
(we called it "chatty"), most OLTP systems I've worked on in recent years are in the 1000++ millisecond ranges and we consider that excellent performance given what we ask of it. If that is reported as a "wait", I could make no sense out of such a report.
As you point out, the DMV report is much more useful.
Fortunately we can break out the numbers and separate the waits from the uses, but I think that SQL Server waits report is highly misleading.
Josh -
High "CPU + Wait for CPU" event on pl/sql execute operation
I am executing a pl/sql in 256 parallel sessions, on 11G r2 DB (RAC 2 nodes), on a 42core IBM P7 Machine.
PL/sql function opens a cursor on a huge table with around 20M rows and does further processing.
Work-load is equally divided into 256 parallel sessions. 256 parallel sessions are opened by a middle-ware application and each session processes data based on an identifier (there are 256 distinct identifier values).
PL/sql function is comprised of some SQL operations, on which i am experiencing some cluster waits. But CPU + wait for CPU event on pl/sql execute operation is close to 80% of the total execution time.
Top user events:
Event Event Class % Event Avg Active Sessions
CPU + Wait for CPU CPU 80.88 98.15
gc current block 2-way Cluster 3.33 4.05
gc cr block busy Cluster 2.01 2.44
gc cr block 2-way Cluster 1.97 2.39
db file sequential read User I/O 1.81 2.20
Top SQL command type:
SQL Command Type Distinct SQLIDs % Activity Avg Active Sessions
PL/SQL EXECUTE 3 60.99 74.02
SELECT 66 12.90 15.65
INSERT 24 9.89 12.01
UPDATE 9 6.00 7.28
DELETE 2 1.33 1.61Rest of the SQL queries seem to be very optimum, but waits on pl/sql execute operation are causing very low tps.
Can anybody give me some heads-up about where to and what to look for to resolve?
Please let me know if any extra information is required.AWR report :
Header
DB Name DB Id Instance Inst num Startup Time Release RAC
FCR 1304316316 fcrypp1 1 01-12ÔÂ-12 04:12 11.2.0.2.0 YES
Host Name Platform CPUs Cores Sockets Memory (GB)
z4ci2011 AIX-Based Systems (64-bit) 168 42 320.00
Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 40650 01-12ÔÂ-12 06:40:03 1203 5.8
End Snap: 40669 01-12ÔÂ-12 09:50:01 1122 5.3
Elapsed: 189.96 (mins)
DB Time: 22,251.65 (mins)
Load profile
Per Second Per Transaction Per Exec Per Call
DB Time(s): 117.1 19.5 0.00 3.89
DB CPU(s): 16.1 2.7 0.00 0.53
Redo size: 12,759,186.3 2,126,361.0
Logical reads: 181,875.9 30,310.2
Block changes: 54,515.5 9,085.2
Physical reads: 1,340.3 223.4
Physical writes: 8,788.9 1,464.7
User calls: 30.1 5.0
Parses: 26.5 4.4
Hard parses: 0.4 0.1
W/A MB processed: 8.5 1.4
Logons: 0.1 0.0
Executes: 41,176.0 6,862.1
Rollbacks: 1.9 0.3
Transactions: 6.0
Time model statistics
Statistic Name Time (s) % of DB Time
sql execute elapsed time 1,334,935.55 99.99
PL/SQL execution elapsed time 1,180,376.60 88.41
DB CPU 182,904.44 13.7
repeated bind elapsed time 292.83 0.02
sequence load elapsed time 279.75 0.02
parse time elapsed 87.4 0.01
hard parse elapsed time 22.52 0
failed parse elapsed time 5.12 0
connection management call elapsed time 4.61 0
PL/SQL compilation elapsed time 1.91 0
hard parse (sharing criteria) elapsed time 0.49 0
hard parse (bind mismatch) elapsed time 0.39 0
inbound PL/SQL rpc elapsed time 0.1 0
DB time 1,335,099.30
background elapsed time 33,298.67
background cpu time 11,692.76
Operating System Statistics
Statistic Value End Value
AVG_BUSY_TIME 202,428
AVG_IDLE_TIME 936,397
AVG_IOWAIT_TIME 4,124
AVG_SYS_TIME 84,480
AVG_USER_TIME 117,573
BUSY_TIME 34,074,303
IDLE_TIME 157,378,825
IOWAIT_TIME 755,368
SYS_TIME 14,256,010
USER_TIME 19,818,293
LOAD 21 10
OS_CPU_WAIT_TIME 23,770,800
VM_IN_BYTES 20,496
VM_OUT_BYTES 2,086,940,520
PHYSICAL_MEMORY_BYTES 343,597,383,680
NUM_CPUS 168
NUM_CPU_CORES 42
NUM_LCPUS 168
NUM_VCPUS 42
GLOBAL_RECEIVE_SIZE_MAX 41,943,040
GLOBAL_SEND_SIZE_MAX 41,943,040
TCP_RECEIVE_SIZE_DEFAULT 16,384
TCP_RECEIVE_SIZE_MAX 9,223,372,036,854,775,807
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 9,223,372,036,854,775,807
TCP_SEND_SIZE_MIN 4,096
SQL ordered by CPU Time
CPU Time (s) Executions CPU per Exec (s) %Total Elapsed Time (s) %CPU %IO SQL Id SQL Module SQL Text
180,330.13 127 1,419.92 98.59 1,326,401.03 13.60 1.08 04kt8u64udphu BEGIN :1 := ap_ch_eod_shell_en...
8,025.48 9,868,469 0.00 4.39 10,809.88 74.24 9.21 arnkbsnzhha77 ch_txn_shell_115 SELECT * FROM CH_ACCT_MAST WHE...
6,117.64 9,873,495 0.00 3.34 8,557.64 71.49 7.14 8qcryvj294s79 ch_eod_shell_138 UPDATE CH_ACCT_MAST_PAR SET DA...
4,614.71 3,185,313 0.00 2.52 11,130.77 41.46 11.88 b75wwkxw34x2k ch_eod_shell_228 INSERT INTO CH_TMP_XF_GL_TXNS ...
4,374.29 9,866,217 0.00 2.39 5,876.00 74.44 37.88 g22p493ra2zr5 ch_eod_shell_248 UPDATE CH_ACCT_MAST SET BAL_LA...
3,514.57 14,026,451 0.00 1.92 6,274.60 56.01 29.55 7bwhphfnnuqpr ch_eod_shell_59 INSERT INTO CH_ACCT_INT_BREAKU...
3,253.36 3,185,706 0.00 1.78 3,875.42 83.95 9.20 9dq134q9btxq8 ch_eod_shell_74 INSERT INTO CH_ST_CAP_INPUT_TX...
3,131.64 9,875,603 0.00 1.71 5,338.43 58.66 15.55 6xhwk1b37rh1t ch_txn_shell_143 UPDATE CH_ACCT_ATTRIBUTES SET ...
2,954.15 9,878,718 0.00 1.62 5,692.88 51.89 13.22 b4at7uq2hw6r7 ch_sweepin_shell SELECT TRIM(A.COD_PKG) FROM RP...
2,572.01 9,867,277 0.00 1.41 4,605.88 55.84 12.58 54ud0a8tuwwbc ch_txn_shell_17 SELECT * FROM CH_ACCT_ATTRIBUT...
1,941.29 19,730,455 0.00 1.06 5,580.38 34.79 7.02 dx5kng8qu560t ch_txn_shell_59 UPDATE CH_ACTIONS_DUE SET COD_...
1,846.01 9,875,239 0.00 1.01 4,737.66 38.96 12.55 af7f92f13rmy4 ch_txn_shell_85 INSERT INTO CH_ACTIONS_DUE (CO... -
How to Reduce Clusetering Factor on Table?
I am seeing a very high clustering factor on an SDO geometry table in our 10g RAC DB on our Linux boxes. This slow performance is repeateable on othe r Linux as well as Solaris DBs for the same table. Inserts go in at a rate of 44 milliseconds per insert and we only have about 27000 rows in the table. After viewing a VERY slow insert of about 600 records into this same table, I saw the clustering factor in OEM. The clustering factor is nearly identical to the # rows in the table indicating that useability of the index is fairly low now. I have referenced Metalink Tech Note 223117.1 and, while it affirms what I've seen, I am still trying to determine how to reduce the Clustering Factor. The excerpt on how to do this is below:
"The only method to affect the clustering factor is to sort and then store the rows in the table in the same order as in they appear in the index. Exporting rows and putting them back in the same order that they appeared originally will have no affect. Remember that ordering the rows to suit one index may have detrimental effects on the choice of other indexes."
Sounds great, but how does one actually go about storing the rows in the table in the same order as they appear in the index?
We have tried placing our commits after the last insert as well as after every insert and the results are fairly neglible. We also have a column of type SDE.ST_GEOMETRY in the table and are wondering if this might also be an issue. Thanks in advance for any help.
Matt SauterJoel is right that the clustering factor is going to have absolutely no effect on the speed of inserts. The clustering factor is merely one, purely statistical, factor the optimiser makes use of to determine how to perform a SELECT statement (i.e., do I bother to use this index or not for row retrieval). It's got nothing to do with the efficiency of inserts.
If I were you, I'd be looking at factors such as excessive disk I/O taking place for other reasons, inadequate buffer cache and/or enqueue and locking issues instead.
If you're committing after every insert, for example, then redo will have to be flushed (a commit is about the only foreground wait event -i.e., one that you get to experience in real time- that Oracle has, so a commit after every insert's really not a smart idea). If your redo logs are stored on, say, the worst-performing disk you could buy that's also doing duty as a fileserver's main hard disk, then LGWR will be twiddling its thumbs a lot! You say you've tested this, and that's fine... I'm just saying, it's one theoretical possibility in these sorts of situations. You still want to make sure you're not suffering any log writer-related waits, all the same.
Similarly, if you're performing huge reads on a (perhaps completely separate) table that is causing the buffer cache to be wiped every second or so, then getting access to your table so your inserts can take place could be problematic. Check if you've got any database writer waits, for example: they are usally a good sign of general I/O bottlenecks.
Finally, you're on a RAC... so if the blocks of the table you're writing to are in memory over on another instance, and they have to be shipped to your instance, you could have high enqueue waits whilst that shipment is taking place. Maybe your interconnect is not up to the job? Maybe it's faulty, even, with significant packet loss along the way? Even worse if someone's decided to switch off cache fusion transfer for the datafiles invoved (for then block shipment happens by writing them to disk in one instance and reading from disk in the other). RAC adds a whole new level of complexity to things, so good luck tracking that lot down!!
Also, maybe you're using Freelists and Freelist groups rather than ASSM, so perhaps you're fighting for access to the freelist with whatever else is happening on your database at the time...
You get the idea: this could be a result of activity taking place on the server for reasons completely unconnected with your insert. It could be a feature of Spatial (with which not many people will be familiar, so good luck if so!) It could be a result of the way your RAC is configured. It could be any number of things... but I'd be willing to bet quite a bit that it's got sod-all to do with the clustering factor!
You'll need to monitor the insert using a tool like Insider or Toad so you can see if waits and so on happen, more or less in real time -or start using the built-in tools like Statspack or AWR to analyze your workload after it's completed- to work out what your best fix is likely to be. -
Exremely high (98%) roll wait time
I am running a ECC 6.0 EHP4 server in solaris 5.10 OS, Oracle 10.2.0.4.
My users experience extremely high roll wait time when they execute transactions like PA20/PA30. It took about 20 minutes for it to load.
I ran stad and found out the majority of the portion is wasted at roll wait time. How do we know what caused the wait time?
e.g. user execute TCODE PA20 and wait for the hourglass for about 20 minutes before the record is shown
Other transactions ran smoothly.
Your help is appreciated.After a RFC trace, it showed this RFC statement having very high duration:
RFC Statement
Function module name HTTP_GET
Source IP Address a.b.c.d
Source Server ourSECCServer
Destination IP Address a.b.c.d (same ip as source)
Destination Server ourSECCServer
Client/Server Client
Conversation ID
RFC Trace Rec. Status 4
Sent Bytes 9.476
Received Bytes 3.353
Total Sent Bytes
Total Received Bytes
ABAP program name SAPLSFTP
RFC Time 224881.681
Any ideas?
Maybe you are looking for
-
Auto creation of Purchase order by heuristic
Hello Experts, In my project, Heuristic is creating "Purchase orders" automatically, which should have been created by planners only. Sometimes, it creates it without even creating "Purchase requisitions" and sometimes with PurReq. Has any one faced
-
Hi Guys, I am facing a problem with the use of the matrix. I am not able to select cells from the matrix with a click although I set the SelectMode = ms_Auto. does someone know how to solve this problem ?? Thanks Bop
-
some of my images in Aperture/iPhoto are appearing pixelated. any idea on how to get the issue fixed
-
Fonts are activated despite being off in Font Book
I opened a Word doc in MS Word 2008 and, all of a sudden, started asking me in a dialog box: "Microsoft Word want to use the font... "bla bla" ... that was downloaded from the internet. Allow Microsoft Word to use this font? And then buttons to eithe
-
Hi, I want to write an app which allows someone to describe how to parse an incoming byte stream into a number of strings. The user can describe the length of each string and how it's encoded. What I'm trying to figure out is - depending on the encod