CXPACKET Wait events
Hi,
I am facing wait event CXPACKET in top wait events. we are using SQL Server 2012. the DOP is set to 1.
Please suggest how to resolve this
REgards
Hi,
I am facing wait event CXPACKET in top wait events. we are using SQL Server 2012. the DOP is set to 1.
Any reaosn why DOP is set to 1.Have you tested this scenario.Instaed of first moving for resolving CXPACKET wait which is not a acctual reason it is just a symptom .It happens when various parallel threads are waiting to synchronize after doing the
task.What is other major wait stats you can see.Can you paste output of below query here
--By Jnathan Kehayias
SELECT TOP 10
wait_type ,
max_wait_time_ms wait_time_ms ,
signal_wait_time_ms ,
wait_time_ms - signal_wait_time_ms AS resource_wait_time_ms ,
100.0 * wait_time_ms / SUM(wait_time_ms) OVER ( )
AS percent_total_waits ,
100.0 * signal_wait_time_ms / SUM(signal_wait_time_ms) OVER ( )
AS percent_total_signal_waits ,
100.0 * ( wait_time_ms - signal_wait_time_ms )
/ SUM(wait_time_ms) OVER ( ) AS percent_total_resource_waits
FROM sys.dm_os_wait_stats
WHERE wait_time_ms > 0 -- remove zero wait_time
AND wait_type NOT IN -- filter out additional irrelevant waits
( 'SLEEP_TASK', 'BROKER_TASK_STOP', 'BROKER_TO_FLUSH',
'SQLTRACE_BUFFER_FLUSH','CLR_AUTO_EVENT', 'CLR_MANUAL_EVENT',
'LAZYWRITER_SLEEP', 'SLEEP_SYSTEMTASK', 'SLEEP_BPOOL_FLUSH',
'BROKER_EVENTHANDLER', 'XE_DISPATCHER_WAIT', 'FT_IFTSHC_MUTEX',
'CHECKPOINT_QUEUE', 'FT_IFTS_SCHEDULER_IDLE_WAIT',
'BROKER_TRANSMITTER', 'FT_IFTSHC_MUTEX', 'KSOURCE_WAKEUP',
'LOGMGR_QUEUE', 'ONDEMAND_TASK_QUEUE',
'REQUEST_FOR_DEADLOCK_SEARCH', 'XE_TIMER_EVENT', 'BAD_PAGE_PROCESS',
'DBMIRROR_EVENTS_QUEUE', 'BROKER_RECEIVE_WAITFOR',
'PREEMPTIVE_OS_GETPROCADDRESS', 'PREEMPTIVE_OS_AUTHENTICATIONOPS',
'WAITFOR', 'DISPATCHER_QUEUE_SEMAPHORE', 'XE_DISPATCHER_JOIN',
'RESOURCE_QUEUE' )
ORDER BY wait_time_ms DESC
Please mark this reply as the answer or vote as helpful, as appropriate, to make it useful for other readers
Similar Messages
-
So I have a query that is running with a large number of CXPacket waits. There are no other tasks in the sys.dm_os_waiting_tasks that are blocking this session id with a wait other than CXPacket Wait. This is taking place on a SQL Server 2008
R2 machine. My question is there something else to look at? I've gone ahead and posted the output of the entire sys.dm_os_waiting_tasks. Any thoughts?
waiting_task_address session_id exec_context_id wait_duration_ms wait_type resource_address blocking_task_address blocking_session_id
blocking_exec_context_id resource_description
0x0000000004E88988 NULL NULL 13525 FT_IFTS_SCHEDULER_IDLE_WAIT NULL NULL NULL NULL NULL
0x0000000004E89288 NULL NULL 64128606 DISPATCHER_QUEUE_SEMAPHORE NULL NULL NULL NULL NULL
0x0000000004ED6748 2 0 19502 XE_TIMER_EVENT NULL NULL NULL NULL NULL
0x0000000004450508 3 0 2779501 XE_DISPATCHER_WAIT NULL NULL NULL NULL NULL
0x000000000446A748 4 0 838 LAZYWRITER_SLEEP NULL NULL NULL NULL NULL
0x000000000448E508 5 0 438 REQUEST_FOR_DEADLOCK_SEARCH 0x000000000B9301F8 NULL NULL NULL NULL
0x0000000004E88508 6 0 1304946379 KSOURCE_WAKEUP NULL NULL NULL NULL NULL
0x0000000004484508 7 0 146961 LOGMGR_QUEUE 0x0000000003597248 NULL NULL NULL NULL
0x00000000804942C8 9 0 3366 SQLTRACE_INCREMENTAL_FLUSH_SLEEP NULL NULL NULL NULL NULL
0x000000000448E748 10 0 929 SLEEP_TASK NULL NULL NULL NULL NULL
0x0000000004450988 11 0 26214727 BROKER_EVENTHANDLER NULL NULL NULL NULL NULL
0x0000000004EBC508 12 0 1304959888 ONDEMAND_TASK_QUEUE 0x0000000003575508 NULL NULL NULL NULL
0x0000000004EA2508 13 0 146959 CHECKPOINT_QUEUE 0x0000000003590C70 NULL NULL NULL NULL
0x0000000004EA2748 14 0 1304947282 BROKER_TRANSMITTER NULL NULL NULL NULL NULL
0x0000000004484988 16 0 1304947282 BROKER_TRANSMITTER NULL NULL NULL NULL NULL
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x0000000255EFEBC8 154 19 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x0000000258D26748 154 17 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x0000000258DB0E08 154 18 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x000000025A392748 154 21 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x0000000258DB2E08 154 20 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x000000025C694508 154 23 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x000000025A675708 154 24 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08 154 0 19974433 CXPACKET 0x0000000804ECC1D0 0x00000001D3F902C8 154 22 exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x0000000004EBCE08 154 1 18269246 CXPACKET 0x000000080C705200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6220e00 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004485288 154 2 18269246 CXPACKET 0x000000080CD4F200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6220f00 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004EA3B88 154 3 18269246 CXPACKET 0x000000080CCDD200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6221000 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004ED7948 154 4 18269246 CXPACKET 0x000000080CC8B200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6220f80 WaitType=e_waitPipeGetRow nodeId=33
0x000000000448F708 154 5 18269242 CXPACKET 0x000000080CD41200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6220e80 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004451B88 154 6 18269245 CXPACKET 0x000000080CE31200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6221080 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004E89048 154 7 18269246 CXPACKET 0x000000080D12B200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6221100 WaitType=e_waitPipeGetRow nodeId=33
0x000000000446ABC8 154 8 18269243 CXPACKET 0x000000080CE7D200 0x000000000448FB88 154 13 exchangeEvent
id=Pipe2f6221200 WaitType=e_waitPipeGetRow nodeId=33
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748 154 17 18270546 CXPACKET 0x000000025C763020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08 154 18 19974425 CXPACKET 0x00000008064DD020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8 154 19 18269474 CXPACKET 0x0000000806355020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08 154 20 18270502 CXPACKET 0x0000000805CC1020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322346 CXPACKET 0x0000000806395020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748 154 21 18322347 CXPACKET 0x0000000806395020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8 154 22 18270621 CXPACKET 0x000000025CE67020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508 154 23 18270541 CXPACKET 0x000000025C75D020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x0000000004EBCE08 154 1 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x000000000448F708 154 5 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x0000000004485288 154 2 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x0000000004ED7948 154 4 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x0000000004EA3B88 154 3 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x0000000004451B88 154 6 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x0000000004E89048 154 7 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708 154 24 18269840 CXPACKET 0x000000025C767020 0x000000000446ABC8 154 8 exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17Hello,
First, I would take a look at what your MAXDOP setting is, that's seems a bit high unless you have around 80 processors - stick to multiples of the number of processors in a single numa node and don't go over half of the numa nodes if this is for an OLTP
system. If it's for a DSS or DW system then having a very high maxdop should be fine.
When dealing with CXPACKET the wait means that it's waiting for all of the parallel threads to complete before it can merge the results and move on. If any single thread has to process more than any other (or waits longer for data) the whole process has
to wait. I would investigate to see if there is a large skew in statistics and one thread is processing much more than another. I'd also try creating an extended events session to grab all of the waitinfo for that session.
Sean Gallardy | Blog |
Twitter -
Performance Issue: Wait event "log file sync" and "Execute to Parse %"
In one of our test environments users are complaining about slow response.
In statspack report folowing are the top-5 wait events
Event Waits Time (cs) Wt Time
log file parallel write 1,046 988 37.71
log file sync 775 774 29.54
db file scattered read 4,946 248 9.47
db file parallel write 66 248 9.47
control file parallel write 188 152 5.80
And after runing the same application 4 times, we are geting Execute to Parse % = 0.10. Cursor sharing is forced and query rewrite is enabled
When I view v$sql, following command is parsed frequently
EXECUTIONS PARSE_CALLS
SQL_TEXT
93380 93380
select SEQ_ORDO_PRC.nextval from DUAL
Please suggest what should be the method to troubleshoot this and if I need to check some more information
Regards,
Sudhanshu BhandariWell, of course, you probably can't eliminate this sort of thing entirely: a setup such as yours is inevitably a compromise. What you can do is make sure your log buffer is a good size (say 10MB or so); that your redo logs are large (at least 100MB each, and preferably large enough to hold one hour or so of redo produced at the busiest time for your database without filling up); and finally set ARCHIVE_LAG_TARGET to something like 1800 seconds or more to ensure a regular, routine, predictable log switch.
It won't cure every ill, but that sort of setup often means the redo subsystem ceases to be a regular driver of foreground waits. -
Hi: I'm analyzing this STATSPACK report: it is "volume test" on our UAT server, so most input is from 'bind variables'. Our shared pool is well utilized in oracle. Oracle redo logs is not appropriately configured on this server, as in 'Top 5 wait events' there are 2 for redos.
I need to know what else information can be dig-out from 'foreground wait events' & 'background wait events', and what can assist us to better understanding, in combination of 'Top 5 wait event's, that how the server/test went? it could be overwelming No. of wait events, so appreciate any helpful diagnostic or analysis. Database is oracle 11.2.0.4 upgraded from 11.2.0.3, on IBM AIX power system 64bit, level 6.x
STATSPACK report for
Database DB Id Instance Inst Num Startup Time Release RAC
~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---
700000XXX XXX 1 22-Apr-15 12:12 11.2.0.4.0 NO
Host Name Platform CPUs Cores Sockets Memory (G)
~~~~ ---------------- ---------------------- ----- ----- ------- ------------
dXXXX_XXX AIX-Based Systems (64- 2 1 0 16.0
Snapshot Snap Id Snap Time Sessions Curs/Sess Comment
~~~~~~~~ ---------- ------------------ -------- --------- ------------------
Begin Snap: 5635 22-Apr-15 13:00:02 114 4.6
End Snap: 5636 22-Apr-15 14:00:01 128 8.8
Elapsed: 59.98 (mins) Av Act Sess: 0.6
DB time: 35.98 (mins) DB CPU: 19.43 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 2,064M Std Block Size: 8K
Shared Pool: 3,072M Log Buffer: 13,632K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ ------------------ ----------------- ----------- -----------
DB time(s): 0.6 0.0 0.00 0.00
DB CPU(s): 0.3 0.0 0.00 0.00
Redo size: 458,720.6 8,755.7
Logical reads: 12,874.2 245.7
Block changes: 1,356.4 25.9
Physical reads: 6.6 0.1
Physical writes: 61.8 1.2
User calls: 2,033.7 38.8
Parses: 286.5 5.5
Hard parses: 0.5 0.0
W/A MB processed: 1.7 0.0
Logons: 1.2 0.0
Executes: 801.1 15.3
Rollbacks: 6.1 0.1
Transactions: 52.4
Instance Efficiency Indicators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.98 Optimal W/A Exec %: 100.00
Library Hit %: 99.77 Soft Parse %: 99.82
Execute to Parse %: 64.24 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 53.15 % Non-Parse CPU: 98.03
Shared Pool Statistics Begin End
Memory Usage %: 10.50 12.79
% SQL with executions>1: 69.98 78.37
% Memory for SQL w/exec>1: 70.22 81.96
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
CPU time 847 50.2
enq: TX - row lock contention 4,480 434 97 25.8
log file sync 284,169 185 1 11.0
log file parallel write 299,537 164 1 9.7
log file sequential read 698 16 24 1.0
Host CPU (CPUs: 2 Cores: 1 Sockets: 0)
~~~~~~~~ Load Average
Begin End User System Idle WIO WCPU
1.16 1.84 19.28 14.51 66.21 1.20 82.01
Instance CPU
~~~~~~~~~~~~ % Time (seconds)
Host: Total time (s): 7,193.8
Host: Busy CPU time (s): 2,430.7
% of time Host is Busy: 33.8
Instance: Total CPU time (s): 1,203.1
% of Busy CPU used for Instance: 49.5
Instance: Total Database time (s): 2,426.4
%DB time waiting for CPU (Resource Mgr): 0.0
Memory Statistics Begin End
~~~~~~~~~~~~~~~~~ ------------ ------------
Host Mem (MB): 16,384.0 16,384.0
SGA use (MB): 7,136.0 7,136.0
PGA use (MB): 282.5 361.4
% Host Mem used for SGA+PGA: 45.3 45.8
Foreground Wait Events DB/Inst: XXXXXs Snaps: 5635-5636
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg %Total
%Tim Total Wait wait Waits Call
Event Waits out Time (s) (ms) /txn Time
enq: TX - row lock contentio 4,480 0 434 97 0.0 25.8
log file sync 284,167 0 185 1 1.5 11.0
Disk file operations I/O 8,741 0 4 0 0.0 .2
direct path write 13,247 0 3 0 0.1 .2
db file sequential read 6,058 0 1 0 0.0 .1
buffer busy waits 1,800 0 1 1 0.0 .1
SQL*Net more data to client 29,161 0 1 0 0.2 .1
direct path read 7,696 0 1 0 0.0 .0
db file scattered read 316 0 1 2 0.0 .0
latch: shared pool 144 0 0 2 0.0 .0
CSS initialization 30 0 0 3 0.0 .0
cursor: pin S 10 0 0 9 0.0 .0
row cache lock 41 0 0 2 0.0 .0
latch: row cache objects 19 0 0 3 0.0 .0
log file switch (private str 8 0 0 7 0.0 .0
library cache: mutex X 28 0 0 2 0.0 .0
latch: cache buffers chains 54 0 0 1 0.0 .0
latch free 290 0 0 0 0.0 .0
control file sequential read 1,568 0 0 0 0.0 .0
log file switch (checkpoint 4 0 0 6 0.0 .0
direct path sync 8 0 0 3 0.0 .0
latch: redo allocation 60 0 0 0 0.0 .0
SQL*Net break/reset to clien 34 0 0 1 0.0 .0
latch: enqueue hash chains 45 0 0 0 0.0 .0
latch: cache buffers lru cha 7 0 0 2 0.0 .0
latch: session allocation 5 0 0 1 0.0 .0
latch: object queue header o 6 0 0 1 0.0 .0
ASM file metadata operation 30 0 0 0 0.0 .0
latch: In memory undo latch 15 0 0 0 0.0 .0
latch: undo global data 8 0 0 0 0.0 .0
SQL*Net message from client 6,362,536 0 278,225 44 33.7
jobq slave wait 7,270 100 3,635 500 0.0
SQL*Net more data from clien 7,976 0 15 2 0.0
SQL*Net message to client 6,362,544 0 8 0 33.7
Background Wait Events DB/Inst: XXXXXs Snaps: 5635-5636
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg %Total
%Tim Total Wait wait Waits Call
Event Waits out Time (s) (ms) /txn Time
log file parallel write 299,537 0 164 1 1.6 9.7
log file sequential read 698 0 16 24 0.0 1.0
db file parallel write 9,556 0 13 1 0.1 .8
os thread startup 146 0 10 70 0.0 .6
control file parallel write 2,037 0 2 1 0.0 .1
Log archive I/O 35 0 1 30 0.0 .1
LGWR wait for redo copy 2,447 0 0 0 0.0 .0
db file async I/O submit 9,556 0 0 0 0.1 .0
db file sequential read 145 0 0 2 0.0 .0
Disk file operations I/O 349 0 0 0 0.0 .0
db file scattered read 30 0 0 4 0.0 .0
control file sequential read 5,837 0 0 0 0.0 .0
ADR block file read 19 0 0 4 0.0 .0
ADR block file write 5 0 0 15 0.0 .0
direct path write 14 0 0 2 0.0 .0
direct path read 3 0 0 7 0.0 .0
latch: shared pool 3 0 0 6 0.0 .0
log file single write 56 0 0 0 0.0 .0
latch: redo allocation 53 0 0 0 0.0 .0
latch: active service list 1 0 0 3 0.0 .0
latch free 11 0 0 0 0.0 .0
rdbms ipc message 314,523 5 57,189 182 1.7
Space Manager: slave idle wa 4,086 88 18,996 4649 0.0
DIAG idle wait 7,185 100 7,186 1000 0.0
Streams AQ: waiting for time 2 50 4,909 ###### 0.0
Streams AQ: qmn slave idle w 129 0 3,612 28002 0.0
Streams AQ: qmn coordinator 258 50 3,612 14001 0.0
smon timer 43 2 3,605 83839 0.0
pmon timer 1,199 99 3,596 2999 0.0
SQL*Net message from client 17,019 0 31 2 0.1
SQL*Net message to client 12,762 0 0 0 0.1
class slave wait 28 0 0 0 0.0
thank you very much!Hi: just know it now: it is a large amount of 'concurrent transaction' designed in this "Volume Test" - to simulate large incoming transaction volme, so I guess wait in eq:TX - row is expected.
The fact: (1) redo logs at uat server is known to not well-tune for configurations (2) volume test slow 5%, however data amount in its test is kept the same by each time import production data, by the team. So why it slowed 5% this year?
The wait histogram is pasted below, any one interest to take a look? any ideas?
Wait Event Histogram DB/Inst: XXXX/XXXX Snaps: 5635-5636
-> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
-> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
-> % of Waits - value: .0 indicates value was <.05%, null is truly 0
-> Ordered by Event (idle events last)
Total ----------------- % of Waits ------------------
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 19 26.3 5.3 10.5 57.9
ADR block file write 5 40.0 60.0
ADR file lock 6 100.0
ARCH wait for archivelog l 14 100.0
ASM file metadata operatio 30 100.0
CSS initialization 30 100.0
Disk file operations I/O 9090 97.2 1.4 .6 .4 .2 .1 .1
LGWR wait for redo copy 2447 98.5 .5 .4 .2 .2 .2 .1
Log archive I/O 35 40.0 8.6 25.7 2.9 22.9
SQL*Net break/reset to cli 34 85.3 8.8 5.9
SQL*Net more data to clien 29K 99.9 .0 .0 .0 .0 .0
buffer busy waits 1800 96.8 .7 .7 .6 .3 .4 .5
control file parallel writ 2037 90.7 5.0 2.1 .8 1.0 .3 .1
control file sequential re 7405 100.0 .0
cursor: pin S 10 10.0 90.0
db file async I/O submit 9556 99.9 .0 .0 .0
db file parallel read 1 100.0
db file parallel write 9556 62.0 32.4 1.7 .8 1.5 1.3 .1
db file scattered read 345 72.8 3.8 2.3 11.6 9.0 .6
db file sequential read 6199 97.2 .2 .3 1.6 .7 .0 .0
direct path read 7699 99.1 .4 .2 .1 .1 .0
direct path sync 8 25.0 37.5 12.5 25.0
direct path write 13K 97.8 .9 .5 .4 .3 .1 .0
enq: TX - row lock content 4480 .4 .7 1.3 3.0 6.8 12.3 75.4 .1
latch free 301 98.3 .3 .7 .7
latch: In memory undo latc 15 93.3 6.7
latch: active service list 1 100.0
latch: cache buffers chain 55 94.5 3.6 1.8
latch: cache buffers lru c 9 88.9 11.1
latch: call allocation 6 100.0
latch: checkpoint queue la 3 100.0
latch: enqueue hash chains 45 97.8 2.2
latch: messages 4 100.0
latch: object queue header 7 85.7 14.3
latch: redo allocation 113 97.3 1.8 .9
latch: row cache objects 19 89.5 5.3 5.3
latch: session allocation 5 80.0 20.0
latch: shared pool 147 90.5 1.4 2.7 1.4 .7 1.4 2.0
latch: undo global data 8 100.0
library cache: mutex X 28 89.3 3.6 3.6 3.6
log file parallel write 299K 95.6 2.6 1.0 .4 .3 .2 .0
log file sequential read 698 29.5 .1 4.6 46.8 18.9
log file single write 56 100.0
log file switch (checkpoin 4 25.0 50.0 25.0
log file switch (private s 8 12.5 37.5 50.0
log file sync 284K 93.3 3.7 1.4 .7 .5 .3 .1
os thread startup 146 100.0
row cache lock 41 85.4 9.8 2.4 2.4
DIAG idle wait 7184 100.0
SQL*Net message from clien 6379K 86.6 5.1 2.9 1.3 .7 .3 2.8 .3
SQL*Net message to client 6375K 100.0 .0 .0 .0 .0 .0 .0
Wait Event Histogram DB/Inst: XXXX/xxxx Snaps: 5635-5636
-> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
-> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
-> % of Waits - value: .0 indicates value was <.05%, null is truly 0
-> Ordered by Event (idle events last)
Total ----------------- % of Waits ------------------
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
SQL*Net more data from cli 7976 99.7 .1 .1 .0 .1
Space Manager: slave idle 4086 .1 .2 .0 .0 .3 3.2 96.1
Streams AQ: qmn coordinato 258 49.2 .8 50.0
Streams AQ: qmn slave idle 129 100.0
Streams AQ: waiting for ti 2 50.0 50.0
class slave wait 28 92.9 3.6 3.6
jobq slave wait 7270 .0 100.0
pmon timer 1199 100.0
rdbms ipc message 314K 10.3 7.3 39.7 15.4 10.6 5.3 8.2 3.3
smon timer 43 100.0 -
Workflow error in fork step, process control, wait event
I am using fork step in workflow which has 2 parallel branches. In 1st branch i have a user decision step followed by a task for posting PO document in case of approval. In the 2nd branch of fork step I have a wait step to wait for an event followed by the same task for posting document with a process control step after that in the end to cancel the workitem(workitem generated by user decision step in the 1st branch of fork). I created the event by using a custom BOR object.
After the fork step is triggered, i have both a wait event running and workitem generated. When i raise the wait event from SWUE by entering the event, object key etc it works fine i.e., the workitem in the other branch is set of logically deleted and workflow ends.
But if the wait event is triggered from the program i.e., using FM SWW_WI_CREATE_VIA_EVENT, both get an error message in workflow log(SWIA). The message is: Error when executing the binding between work item 000000XXXXXX and flow item 000000XXXXXX where workitem number is the workitem id of the posting document task and flow item id is the workflow parent idhi,
message is self explanatory.
Activate the event trace SWELS, then do the event with SWUE and within your program (als please use SAP_WAPI function modules).
Now compare the 2 events in SWEL to see what the differences are .
Kind regards, Rob Dielemans -
Library Cache Pin Wait Event (within the context of APEX)
Hello,
Firstly -
Oracle Version: 10.2.0.4.0
Apex Version: 3.0.1.00.08
Okay, my colleague (no really! This isn't one of those "Ahem ... A friend of mine has contracted something nasty +downstairs+..."-type questions) is having problems compiling a package (using TOAD incidentally, but it's the same in SQL Developer).
I've searched the forum and the web for a bit of help on what's maybe happening here and it appears to be related to a concurrency conflict with the package definition - from what I can understand it's a case of the package is in use by another session, therefore another session cannot alter it at the same time (which makes sense)
"What does this have to do with APEX?"... well, he is working on this package using the following methodology:
1. Compile the package body/spec (as necessary - body more often obviously)
2. run an apex page which uses the code in a process, which may or may not result in the error page being displayed
3. Making changes to the package body/spec
repeat steps 1-3 ad nauseum...
He is the only user directly accessing the schema (and the only user accessing the page via APEX too, although I appreciate this isn't quite the same thing).
I was wondering if, due to the architecture of APEX (the use of session pools etc), the state of a package might be being retained in some manner, thus resulting in this library cache pin wait event? If so, is there anything I can do to mitigate against this occurring?
p.s. the only difference I can see between this particular package and any other package in the schema is that this one interacts with blobs (including making references to the wwv_flow_files view) - with blobs being passed as parameters between procedures (thus potentially creating temporary blobs which may or may not being closed).
Any ideas?
p.p.s. there are also no DBMS_SCHEDULER jobs or anything that might potentially be running the code incidentally...
Edited by: Joel_C on 11-Nov-2011 11:58
We got our DBAs to run a bit of code to identify the blocking session:
select
decode(lob.kglobtyp, 0, 'NEXT OBJECT', 1, 'INDEX', 2, 'TABLE', 3, 'CLUSTER',
4, 'VIEW', 5, 'SYNONYM', 6, 'SEQUENCE',
7, 'PROCEDURE', 8, 'FUNCTION', 9, 'PACKAGE',
11, 'PACKAGE BODY', 12, 'TRIGGER',
13, 'TYPE', 14, 'TYPE BODY',
19, 'TABLE PARTITION', 20, 'INDEX PARTITION', 21, 'LOB',
22, 'LIBRARY', 23, 'DIRECTORY', 24, 'QUEUE',
28, 'JAVA SOURCE', 29, 'JAVA CLASS', 30, 'JAVA RESOURCE',
32, 'INDEXTYPE', 33, 'OPERATOR',
34, 'TABLE SUBPARTITION', 35, 'INDEX SUBPARTITION',
40, 'LOB PARTITION', 41, 'LOB SUBPARTITION',
42, 'MATERIALIZED VIEW',
43, 'DIMENSION',
44, 'CONTEXT', 46, 'RULE SET', 47, 'RESOURCE PLAN',
48, 'CONSUMER GROUP',
51, 'SUBSCRIPTION', 52, 'LOCATION',
55, 'XML SCHEMA', 56, 'JAVA DATA',
57, 'SECURITY PROFILE', 59, 'RULE',
62, 'EVALUATION CONTEXT',
'UNDEFINED') object_type,
lob.KGLNAOBJ object_name,
pn.KGLPNMOD lock_mode_held,
pn.KGLPNREQ lock_mode_requested,
ses.sid,
ses.serial#,
ses.username
FROM
x$kglpn pn,
v$session ses,
x$kglob lob,
v$session_wait vsw
WHERE
pn.KGLPNUSE = ses.saddr and
pn.KGLPNHDL = lob.KGLHDADR
and lob.kglhdadr = vsw.p1raw
and vsw.event = 'library cache pin'
order by lock_mode_held descresults as follows (I've changed some object names to protect the ignorant):
OBJECT_TYP OBJECT_NAME LOCK_MODE_HELD LOCK_MODE_REQUESTED SID SERIAL# USERNAME
PACKAGE PKG_FOOBAR 2 0 356 21694 HTMLDB_PUBLIC_U
SER
PACKAGE PKG_FOOBAR 0 3 463 22309 FOOHTMLDB_PUBLIC_USER is the apex user incidentally. The session is marked in the v$session table as "inactive", the last statement being
Begin
Dbms_session.reset_package;
End;Edited by: Joel_C on 11-Nov-2011 14:39bump
No-one?
The problem seems to have 'resolved itself' over the weekend incidentally (although I don't believe anything truly resolves itself in this manner - something must have changed). -
Problem identifying db object for "buffer busy waits" event.
10.2.0.3
AIX 64
SELECT username, a.p1text, a.p1, a.p2text, a.p2, a.p3text, a.p3, event FROM v$session a WHERE
a.status='ACTIVE'
AND a.event = 'buffer busy waits'
Query reports about 40 active sessions with this information:
file# 3746
block# 2
class# 13
select
owner,
segment_name,
segment_type
from
dba_extents
where
file_id = 3746
and
2 between block_id and block_id + blocks -1;
no rows returned
SELECT MAX(a.file#) FROM sys.file$ a
3535
This was only a temporary situation when after couple of minutes(7) wait event "buffer busy waits" dissapeared completely.
Any ideas?
Thank you,
Daniel.http://perfvision.com/papers/06_buffer_cache.ppt
Slide 80-81 points at increasing the size of the initial and next extent for File Header Block buffer busy waits
Side 85 points at high extent allocation for File Header Block buffer busy waits
http://perfvision.com/ftp/hotsos/aas.ppt
Side 55 points at extent allocation too small/too many extents being allocated for File Header Block buffer busy waits
A couple hints from the documentation:
http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/instance_tune.htm
"To determine the possible causes [of buffer busy waits], first query V$SESSION to identify the value of ROW_WAIT_OBJ# when the session waits for buffer busy waits."
"To identify the object and object type contended for, query DBA_OBJECTS using the value for ROW_WAIT_OBJ# that is returned from V$SESSION."
"V$SEGMENT_STATISTICS - This is a user-friendly view of statistic values. In addition to all the columns of V$SEGSTAT, it has information about such things as the segment owner and table space name. It makes the statistics easy to understand, but it is more costly."
You may want to query DBA_TEMP_FILES for the specific FILE_ID identified by the V$SESSION. Taking a look at V$SEGMENT_STATISTICS might also be helpful.
Are you using dictionary managed tablespaces, locally managed tablespaces with manual extent size management, ASSM with manual extent size management, or ASSM with automatic extent size management?
Charles Hooper
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Hi frnds,
As, I'm beginner to performance tuning I dont know
What action do i need to take?
I mean how to read the output which I given below.
this is the output suffering buffer busy waits.
Could anyone please tell me
CLASS TOTAL_WAITS TOTAL_TIME
data block 93303 58711
unused 0 0
system undo header 12 232
undo header 7847 6636
3rd level bmb 0 0
save undo header 0 0
bitmap index block 0 0
file header block 0 0
free list 0 0
undo block 68 207
segment header 422 399
extent map 0 0
2nd level bmb 0 0
system undo block 0 0
sort block 0 0
save undo block 0 0
1st level bmb 1 17
bitmap block 0 0
Thanks, Muhammed Thameem. SHello,
"Buffer busy waits" is contention for a buffer (representing a specific
version of a database block) within the Buffer Cache. So, in essence
it is block contention and thus it is most likely something to do with
the design of the tables and indexes supporting the application. A
built-in bottleneck. On indexes, it could be the age-old problem of
insertions into an index on a column with a monotonically-ascending
data value (i.e. timestamps or sequence numbers) which tends to cause
contention on the highest leaf node of the index. On tables, it might
have to do with many concurrent insertions into a table in a
freelist-managed tablespace where the table has only one freelist. It
could also be due to a home-grown implementation of sequence-number
generators (i.e. small table with one row, one column in which contains
the "last value" of a sequence, etc) which lots of people use to avoid
not being "portable across databases" which they think means not using
Oracle sequences (yadda yadda yadda).
I'd look for any SQL statement in the "SQL sorted by Elapsed Time"
section of the AWR report which exhibits high elapsed time but
relatively low CPU time, indicating a lot of wait time. Of course,
there are something like 800 possible wait events in current releases
of Oracle, of which "buffer busy waits" is only one, so this is just
inference and not a direct causal connection to your problem. But,
once I find such statements I'd check to see if they are
accessing/manipulating tables within the CUBS_DATA tablespace, and then
use "select * from table(dbms_xplan.display_awr('sql-id'))" to
get the execution plan(s), and then look for something ineffective
within the execution plan. You might find the script "sqlhistory.sql" helpful
here as well, to get a "historical perspective" on the execution of the
SQL statements over time, in case the buffer busy waits peaked at some
point in the past
Please refer to:
http://www.pubbs.net/201003/oracle/51925-understanding-awr-buffer-waits.html
Also
http://www.remote-dba.net/oracle_10g_tuning/t_buffer_busy_waits.htm
kind regards
Mohamed -
Wait Time / Timeout Time in waiting events
Hi,
From manual, I came to know that some of waiting events having wait time or time out duration. I have not understood, what happens if it has reached timeout. What happens next ? ( Will you explain in more detail ). It saysl it will renew wait event. ( I have not understood what does mean "RENEW", Or what steps will be taken by Oracle, what happens if again reached timeout ) ?
Thanks for clearing my doubt in detail.
regards
pjpPlease read the FAQ and learn how to enclose your listings in tags so we can read them.
I can't help you at this time because I can not read what you posted. -
Wait Events "log file parallel write" / "log file sync" during CREATE INDEX
Hello guys,
at my current project i am performing some performance tests for oracle data guard. The question is "How does a LGWR SYNC transfer influences the system performance?"
To get some performance values, that i can compare i just built up a normal oracle database in the first step.
Now i am performing different tests like creating "large" indexes, massive parallel inserts/commits, etc. to get the bench mark.
My database is an oracle 10.2.0.4 with multiplexed redo log files on AIX.
I am creating an index on a "normal" table .. i execute "dbms_workload_repository.create_snapshot()" before and after the CREATE INDEX to get an equivalent timeframe for the AWR report.
After the index is built up (round about 9 GB) i perform an awrrpt.sql to get the AWR report.
And now take a look at these values from the AWR
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
log file parallel write 10,019 .0 132 13 33.5
log file sync 293 .7 4 15 1.0
......How can this be possible?
Regarding to the documentation
-> log file sync: http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3120
Wait Time: The wait time includes the writing of the log buffer and the post.-> log file parallel write: http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3104
Wait Time: Time it takes for the I/Os to complete. Even though redo records are written in parallel, the parallel write is not complete until the last I/O is on disk.This was also my understanding .. the "log file sync" wait time should be higher than the "log file parallel write" wait time, because of it includes the I/O and the response time to the user session.
I could accept it, if the values are close to each other (maybe round about 1 second in total) .. but the different between 132 seconds and 4 seconds is too noticeable.
Is the behavior of the log file sync/write different when performing a DDL like CREATE INDEX (maybe async .. like you can influence it with the initialization parameter COMMIT_WRITE??)?
Do you have any idea how these values come about?
Any thoughts/ideas are welcome.
Thanks and RegardsSurachart Opun (HunterX) wrote:
Thank you for Nice Idea.
In this case, How can we reduce "log file parallel write" and "log file sync" waited time?
CREATE INDEX with NOLOGGINGA NOLOGGING can help, can't it?Yes - if you create index nologging then you wouldn't be generating that 10GB of redo log, so the waits would disappear.
Two points on nologging, though:
<ul>
it's "only" an index, so you could always rebuild it in the event of media corruption, but if you had lots of indexes created nologging this might cause an unreasonable delay before the system was usable again - so you should decide on a fallback option, such as taking a new backup of the tablespace as soon as all the nologging operatons had completed.
If the database, or that tablespace, is in +"force logging"+ mode, the nologging will not work.
</ul>
Don't get too alarmed by the waits, though. My guess is that the +"log file sync"+ waits are mostly from other sessions, and since there aren't many of them the other sessions are probably not seeing a performance issue. The +"log file parallel write"+ waits are caused by your create index, but they are happeninng to lgwr in the background which is running concurrently with your session - so your session is not (directly) affected by them, so may not be seeing a performance issue.
The other sessions are seeing relatively high sync times because their log file syncs have to wait for one of the large writes that you have triggered to complete, and then the logwriter includes their (little) writes with your next (large) write.
There may be a performance impact, though, from the pure volume of I/O. Apart from the I/O to write the index you have LGWR writting (N copies) of the redo for the index and ARCH is reading and writing the completed log files caused by the index build. So the 9GB of index could easily be responsible for vastly more I/O than the initial 9GB.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan -
Wait events 'direct path write' and 'direct path read'
Hi,
We have a query which is taking more that 2 min. It's a 9.2.0.7 database. We took the trace/tkprof of the query,and identified that there are so manay 'direct path write' and 'direct path read' wait events in the trace file.
WAIT #3: nam='direct path write' ela= 5 p1=201 p2=70710 p3=15
WAIT #3: nam='direct path read' ela= 170 p1=201 p2=71719 p3=15
In the above, "p1=201" is a file_id, but we could not find any data file, temp file, control file with that id# 201.
Can you please let us know what's "p1=201" here, how to identify the file which is causing the issue.
Thanks
SravanWhat does:
show parameter db_filesreturn? My guess, is that it returns 200.
The direct file read and direct file write events are reads and writes to TEMP tablespace. In those wait events, the file# is reported as db_files+temp file id. So, 201 means temp file #1.
Now, as to your actual performance problem.
Without seeing the SQL and the corresponding execution plan, it's impossible to be sure. However, the most common causes of temp writes are sort operations and group by operations.
If you decide to post your SQL and execution plan, please be sure to make it readable by formatting it. Information on how to do so can be found here.
Hope that helps,
-Mark
Edited by: mbobak on May 1, 2011 1:50 AM -
How to see the wait events info. after excute a select query
Hi
How to see the wait events info. after execute a select query. Are there any parameter to set for this option?
And also wanna see the follwing info. in trace file. For this what are the parameters I have to set right?
SELECT * FROM emp, dept
WHERE emp.deptno = dept.deptno;
call count cpu elapsed disk query current rows
Parse 1 0.16 0.29 3 13 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 0.03 0.26 2 2 4 14
Misses in library cache during parse: 1
Parsing user id: (8) SCOTT Regards
Arpitha$ sqlplus scott/tiger
SQL*Plus: Release 10.2.0.2.0 - Production on Wed Apr 20 15:29:33 2011
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL> SHOW PARAMETER dump
NAME TYPE VALUE
background_core_dump string partial
background_dump_dest string /user/oracle/app/oracle/admin/
orclsb/bdump
core_dump_dest string /user/oracle/app/oracle/admin/
orclsb/cdump
max_dump_file_size string UNLIMITED
shadow_core_dump string partial
user_dump_dest string /user/oracle/app/oracle/admin/
orclsb/udump
SQL> ALTER SESSION SET EVENTS='10046 trace name context forever, level 12';
Session altered.
SQL> SELECT * FROM emp WHERE deptno=20;
EMPNO ENAME JOB MGR HIREDATE SAL COMM
DEPTNO
7369 SMITH CLERK 7902 17-DEC-80 800
20
7566 JONES MANAGER 7839 02-APR-81 2975
20
7788 SCOTT ANALYST 7566 19-APR-87 3000
20
EMPNO ENAME JOB MGR HIREDATE SAL COMM
DEPTNO
7876 ADAMS CLERK 7788 23-MAY-87 1100
20
7902 FORD ANALYST 7566 03-DEC-81 3000
20Now
$ pwd
/user/oracle/app/oracle/admin/orclsb/udump
$ ls -ltr|tail -5
-rw-r----- 1 oracle oinstall 622 Apr 20 11:35 orclsb_ora_949.trc
-rw-r----- 1 oracle oinstall 651 Apr 20 11:35 orclsb_ora_976.trc
-rw-r----- 1 oracle oinstall 1982 Apr 20 11:35 orclsb_ora_977.trc
-rw-r----- 1 oracle oinstall 1443 Apr 20 15:29 orclsb_ora_1251.trc
-rw-r----- 1 oracle oinstall 279719 Apr 20 15:30 orclsb_ora_1255.trc
$ tkprof orclsb_ora_1255.trc orclsb_ora_1255.txt
TKPROF: Release 10.2.0.2.0 - Production on Wed Apr 20 15:32:17 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
$ ls -ltr|tail -5
-rw-r----- 1 oracle oinstall 651 Apr 20 11:35 orclsb_ora_976.trc
-rw-r----- 1 oracle oinstall 1982 Apr 20 11:35 orclsb_ora_977.trc
-rw-r----- 1 oracle oinstall 1443 Apr 20 15:29 orclsb_ora_1251.trc
-rw-r----- 1 oracle oinstall 279719 Apr 20 15:30 orclsb_ora_1255.trc
-rw-r--r-- 1 oracle oinstall 26872 Apr 20 15:32 orclsb_ora_1255.txtThis orclsb_ora_1255.txt contains the required information. -
Hash join ending up in huge wait events
Hi,
We are experiencing huge wait events ( direct path read temp , direct path write temp) on our Materialized View refresh in 10.2.0.4 version of oracle 10g in linux rhel5 environment while monitoring the refresh session from db console. While checking the explain plan of the mv query there is a huge hash_join (due to self join nature of the query) is shown. As advised in some dba forums, i have increased my pga_aggregate_target to a value of 4 gb from 1800 mb. The PGA_HIT % is raised to 60% from 58% ( just 2% improvement). But still my direct path read temp and direct path write temp wait event have not reduced and a huge temp space is taken for hash join.
Since we have some usage limit set by some hidden parameters for a each session on pga_aggregate_target, increase the size did not helped me much. The mv refresh is taking more than 5 hours ( sometimes it exceeds 5 hrs) to completes it refresh where as the same query in window (production) is completed less than two hours. Before a month, the refresh time in both environment was nearly close. But now it has changed and not able to figure it out.
STATISTICS have been collected regularly using dbms_gather_stats in both environment. Both mv refresh are scheduled to run using dbms_scheduler (Manual refresh). SGA_TARGET and other memory parameters are almost same.
Environment : Dataware house
O/s : RHEL 5
Oracle version : 10.2.0.4
Work_policy=auto
Is there any possibility to reduce this wait event and there by reducing the elapsed time? I am also interested to know changing the plan to use other sort will help? I don't know whether the details are sufficient to analyze this issue. If you need more details on this just let me know.
I really appreciate your help and thanks in advance to all.Thans for your comments. Here is the code, explan plan and autotrace trace stat output.
SELECT lasg.employee_number "EMPLOYEE_NUM",
lasg.full_name "FULL_NAME",
lasg.person_id "PERSON_ID",
SUBSTR (lasg.organization, 1, 4) "DEPT",
casg.assign_start_date "EFFECTIVE_START_DATE",
casg.assign_end_date "EFFECTIVE_END_DATE",
hasg.organization "PRIOR_ORG",
casg.organization organization,
hasg.supervisor "PRIOR_SUPERVISOR",
casg.supervisor "SUPERVISOR_NAME",
hasg.location "PRIOR_LOCATION",
casg.location location,
hasg.job_title "PRIOR_TITLE",
casg.job_title job_name,
CASE
WHEN hasg.organization = casg.organization THEN 'No Change'
ELSE 'Change'
END
org_change,
CASE
WHEN hasg.location = casg.location THEN 'No Change'
ELSE 'Change'
END
loc_change,
CASE
WHEN hasg.supervisor = casg.supervisor THEN 'No Change'
ELSE 'Change'
END
sup_change,
CASE
WHEN hasg.job_title = casg.job_title THEN 'No Change'
ELSE 'Change'
END
job_change
FROM panad.data_employ_details lasg,
panad.data_employ_details casg,
panad.data_employ_details hasg
WHERE lasg.person_id = casg.person_id(+)
AND lasg.assign_end_date = (SELECT MAX (lasg2.assign_end_date)
FROM panad.data_employ_details lasg2
WHERE lasg.person_id = lasg2.person_id)
AND casg.person_id = hasg.person_id(+)
AND hasg.assign_start_date =
(SELECT MAX (hasg2.assign_start_date)
FROM panad.data_employ_details hasg2
WHERE hasg2.person_id = lasg.person_id
AND hasg2.assign_end_date < casg.assign_start_date)
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1 | 303 | | 10261 (91)| 00:02:04 |
|* 1 | FILTER | | | | | | |
|* 2 | HASH JOIN | | 1 | 303 | | 10179 (91)| 00:02:03 |
|* 3 | HASH JOIN | | 5 | 1060 | | 10095 (92)| 00:02:02 |
|* 4 | HASH JOIN | | 6786 | 960K| | 10011 (93)| 00:02:01 |
| 5 | VIEW | VW_SQ_1 | 6786 | 225K| | 9927 (94)| 00:02:00 |
| 6 | HASH GROUP BY | | 6786 | 384K| | 9927 (94)| 00:02:00 |
| 7 | MERGE JOIN | | 50M| 2820M| | 1427 (53)| 00:00:18 |
| 8 | SORT JOIN | | 31937 | 998K| 2776K| 367 (2)| 00:00:05 |
| 9 | TABLE ACCESS FULL| DATA_EMPLOY_DETAILS | 31937 | 998K| | 82 (2)| 00:00:01 |
|* 10 | SORT JOIN | | 31937 | 810K| 2520K| 324 (2)| 00:00:04 |
| 11 | TABLE ACCESS FULL| DATA_EMPLOY_DETAILS | 31937 | 810K| | 82 (2)| 00:00:01 |
| 12 | TABLE ACCESS FULL | DATA_EMPLOY_DETAILS | 31937 | 3461K| | 83 (3)| 00:00:01 |
| 13 | TABLE ACCESS FULL | DATA_EMPLOY_DETAILS | 31937 | 2089K| | 83 (3)| 00:00:01 |
| 14 | TABLE ACCESS FULL | DATA_EMPLOY_DETAILS | 31937 | 2838K| | 83 (3)| 00:00:01 |
| 15 | SORT AGGREGATE | | 1 | 13 | | | |
|* 16 | TABLE ACCESS FULL | DATA_EMPLOY_DETAILS | 5 | 65 | | 82 (2)| 00:00:01 |
Predicate Information (identified by operation id):
1 - filter("LASG"."ASSIGN_END_DATE"= (SELECT MAX("LASG2"."ASSIGN_END_DATE") FROM
"PANAD"."DATA_EMPLOY_DETAILS" "LASG2" WHERE "LASG2"."PERSON_ID"=:B1))
2 - access("CASG"."PERSON_ID"="HASG"."PERSON_ID" AND "HASG"."ASSIGN_START_DATE"="VW_COL_1")
3 - access("LASG"."PERSON_ID"="CASG"."PERSON_ID" AND "PERSON_ID"="LASG"."PERSON_ID")
4 - access("ROWID"=ROWID)
10 - access(INTERNAL_FUNCTION("HASG2"."ASSIGN_END_DATE")<INTERNAL_FUNCTION("CASG"."ASSIGN_START_DATE")
filter(INTERNAL_FUNCTION("HASG2"."ASSIGN_END_DATE")<INTERNAL_FUNCTION("CASG"."ASSIGN_START_DATE")
16 - filter("LASG2"."PERSON_ID"=:B1)
37 rows selected.
- autot trace stat output -
5070 rows selected.
Statistics
35203 recursive calls
0 db block gets
3675913 consistent gets
4269882 physical reads
0 redo size
1046781 bytes sent via SQL*Net to client
4107 bytes received via SQL*Net from client
339 SQL*Net roundtrips to/from client
69 sorts (memory)
0 sorts (disk)
5070 rows processed I have tried running this query with paralell but not helped.
I have read the links provided by both of you. Dictionary and fixed table stats are collected as a routine.
From the link given byTaral, Greg Rahn has suggested that it is a bug as below.
Its bug 9041800 and there is a 10.2.0.4 backport available as of 01/29/10.How can i get this bug fixed since there is no explanation of what need to be done? Do i need to contact oracle support for the 10.2.0.4 backport for RHEL5?
Thanks in advance
Edited by: Karthikambalav on Mar 9, 2010 2:43 AM -
Tuning row lock contention wait events
Hello everyone,
Working on 10g/windows
Top 5 events
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 9462339 48
enq: TX - row lock contention 12531 3660728 2921.34 18
control file parallel write 1300731 3088079 23.74 16
log file parallel write 1510503 1264080 8.37 6
log file sync 1072553 968007 9.03 5
Distribution of row lock wait during the last 4 days in the database server
END_INTERVAL_TIME TOTAL_WAITS TIME_WAITED_MICRO AVG_WAIT_MS
2008-04-01 16:00:58 909 2721008230 2993.41
2008-04-01 15:00:27 50 149941140 2998.82
2008-03-31 12:00:42 193 575595397 2982.36
2008-03-29 23:00:13 172 513058700 2982.9
2008-03-29 22:00:37 164 483940046 2950.85
2008-03-27 22:00:35 565 1667120838 2950.66
2008-03-26 18:00:59 348 1042918982 2996.89
My analysis:
It's obvious that the row lock contention wait time is huge, and this direct me to find out SQL stmt, causing this.
all the SQL statement was SELECT ....... FOR UPDATE stmt.
I was also able to find out locked tables.
My tuning idea:
1. I'm thinking to reorganize hot tables as well as their indexes, but by instinct it seems to not give so much value to avoid the huge row lock wait time.
2. I'm also seeing if I can reduce the number of rows per block, by increasing PCTFREE and diminishing PCTUSED, so the contention will spread over many blocks instead of one heavy block.
Question
As SQL stmt related to those locked tables are select ... for update, how could I tune this kind of stmt?
Does someone have other idea to come up with this row lock contention?
Tanks for your effort and helpTaking another look at your suggested function based index, it depends on the data type of the DEV.POS_FOLIO_ID.POS_FOLIO_ID column. If the column is defined as a number, and it is a primary key, there will already be a usable index on that column.
Yesterday, I wrote this: "Once I understood why or how the sessions were trying to insert duplicate primary key values, I would try to determine why the average number of seconds for the wait event is almost 3 seconds (maybe a timeout)."
After fixing the formatting of the top 5 wait events (total duration unknown):
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 94,623.39 48
enq: TX - row lock contention 12,531 36,607.28 2921.34 18
control file parallel write 1,300,731 30,880.79 23.74 16
log file parallel write 1,510,503 12,640.80 8.37 6
log file sync 1,072,553 9,680.07 9.03 512,531 * 3 second time out = 37,593 seconds = 10.44 hours.
What if the reason for the 3 second average wait time is due to a timeout. I performed a little experiment... I changed a row in a test table and then made a pot of coffee.
In session 1:
CREATE TABLE T1 (
C1 NUMBER(10),
C2 NUMBER(10),
PRIMARY KEY (C1));
INSERT INTO T1
SELECT
ROWNUM,
ROWNUM*10
FROM
DUAL
CONNECT BY
LEVEL<=1000000;
COMMIT;I now have a test table with 1,000,000 rows. I start monitoring the changes in the wait events roughly every 60 seconds, and V$SESSION_WAIT and V$LOCK roughly 4 times per second.
Back in session 1:
UPDATE
T1
SET
C1=-C1
WHERE
C1<=100;I have now modified the first 100 rows that were inserted into the table, time to make the pot of coffee.
In session 2, I try to insert a row with a primary key value of -10:
INSERT INTO T1 VALUES (
-10,
10);Session 2 hangs.
If I take the third 60 second snap of the system wide wait events as the zero point, and the 11th snap as the end point. There were 149 waits on ENQ: TX - ROW LOCK CONTENTION, 148 time outs, 446.62 seconds of total time in the wait event, with an average wait time of 2.997450 seconds.
Rolling down to the session level wait events, SID 208 (my session 2) had 149 waits on ENQ: TX - ROW LOCK CONTENTION, for a total time of 446.61 seconds with an average wait time of 2.997383 seconds. All of the 149 waits and the wait time was in this one session that was locked up for the full duration of this time period because session 1 was making a pot of coffee.
Rolling down to V$SESSION_WAIT (sampled roughly 4 times per second): At the start of the third time interval, SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 39 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
At the end of the 11th time interval: , SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 483 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
Rolling down to V$LOCK (sampled roughly 4 times per second): I see that SID 214 (session 1) is blocking SID 208 (session 2). SID 214 has a TX lock in mode 6 with ID1 of 196646 and ID2 of 4754. SID 208 is requesting a TX lock in mode 4 with ID1 of 196646 and ID2 of 4754.
So, it seems that I need a faster coffee pot rather than an additional index on my table. It could be that the above process would have found that the application associated with SID 214 was abandoned or crashed and for some reason the lock was not released for a long period of time, a little less than 10.44 hours in your case.
Charles Hooper
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Enq: TX - row lock contention wait event
Hi,
I would like to find which DML query has not given COMMIT or ROLLBACK after the execution. Because one of the development database have more table locks and developer reported that their session was hanging. I referred AWR report also and more timed waits occurred in the enq: TX - row lock contention. I need to trace which DML query has not commit or rollback.
Please help me to solve the issue.
Database version: 11.2.0.1.0
Foreground Wait Events
Event
Waits
%Time -outs
Total Wait Time (s)
Avg wait (ms)
Waits /txn
% DB time
enq: TX - row lock contention
320
0
72,047
225147
0.20
99.53
log file sync
547
0
14
26
0.35
0.02
library cache lock
13
0
11
843
0.01
0.02
SQL*Net break/reset to client
1,080
0
2
1
0.69
0.00
SQL*Net message to client
659,006
0
1
0
421.63
0.00
direct path sync
3
0
1
299
0.00
0.00
SQL*Net more data from client
5,541
0
1
0
3.55
0.00
db file scattered read
554
0
0
1
0.35
0.00
SQL*Net more data to client
14,975
0
0
0
9.58
0.00
db file sequential read
2,817
0
0
0
1.80
0.00
ADR block file read
4
0
0
43
0.00
0.00
enq: CR - block range reuse ckpt
2
0
0
71
0.00
0.00
asynch descriptor resize
38,073
100
0
0
24.36
0.00
latch: shared pool
61
0
0
1
0.04
0.00
control file sequential read
6,900
0
0
0
4.41
0.00
Disk file operations I/O
550
0
0
0
0.35
0.00
cursor: pin S
1
0
0
8
0.00
0.00
direct path write temp
34
0
0
0
0.02
0.00
library cache: mutex X
5
0
0
1
0.00
0.00
latch: In memory undo latch
2
0
0
1
0.00
0.00
buffer busy waits
14
0
0
0
0.01
0.00
SQL*Net message from client
658,990
0
294,847
447
421.62
jobq slave wait
669
99
333
497
0.43
PL/SQL lock timer
1
100
1
998
0.00Oracle does not and cannot tell you from historical views (e.g. AWR) which DMLs have not COMMITed or ROLLBACKed. A Transaction ends with a COMMIT or ROLLBACK. The transaction could have a million (or more) DML statements with a million (or more) SELECT statements between the first DML and the COMMIT / ROLLBACK.
Even identifying such DMLs in real time is close to impossible. Because the session holding the lock may have issued a dozen or a million subsequent SQL statements while other sessions are waiting for the lock. You can only identify the session that is the lock holder (the BLOCKING_SESSION in V$SESSION).
If you have tracing enabled for all sessions, then you could review the trace file for the BLOCKING_SESSION to identify the DML(s) the session has executed.
Hemant K Chitale
Maybe you are looking for
-
OAF-Checkbox when selected should update column in a table
Hello, I have a requirement to add checkbox to MES Operator page and it should perform the following, 1) When MES Operator page is opened, it should read attribute1(Y/N) from table and accordingly check/uncheck checkbox. 2) When checkbox is checked/u
-
Hi, I am new to Bi Publisher. I have a simple report query and am able to add a bind variable within the where clause to filter data by one of the fields and it works fine. I am not able to use a bind variable to control the order by. Can a parameter
-
ICal to syncronise with iPod in due date order
Hiya! Could you please let me know if it is possible to syncronise iCal to the iPod (5th generation) in due date order and how to do it? Ta Mac OS X (10.4.4)
-
Create dev database from standby database
Hi folks, I need to create a dev database by using RMAN's method of cloning. But, since it is business hours and the production database is being used at a very high rate, I cannot carryout the cloning by connecting the primary database as it may cau
-
Line indicator for text Area??
for my text field , I used :: TextArea tA01=new TextArea(20,30); is there anyway or functions or methods that I can use to locate the current position of my blinking cursor ??? thanks....