Exremely high (98%) roll wait time
I am running a ECC 6.0 EHP4 server in solaris 5.10 OS, Oracle 10.2.0.4.
My users experience extremely high roll wait time when they execute transactions like PA20/PA30. It took about 20 minutes for it to load.
I ran stad and found out the majority of the portion is wasted at roll wait time. How do we know what caused the wait time?
e.g. user execute TCODE PA20 and wait for the hourglass for about 20 minutes before the record is shown
Other transactions ran smoothly.
Your help is appreciated.
After a RFC trace, it showed this RFC statement having very high duration:
RFC Statement
Function module name HTTP_GET
Source IP Address a.b.c.d
Source Server ourSECCServer
Destination IP Address a.b.c.d (same ip as source)
Destination Server ourSECCServer
Client/Server Client
Conversation ID
RFC Trace Rec. Status 4
Sent Bytes 9.476
Received Bytes 3.353
Total Sent Bytes
Total Received Bytes
ABAP program name SAPLSFTP
RFC Time 224881.681
Any ideas?
Similar Messages
-
High roll wait time, gui time and processing time
Hi all,
We are having a performance issue with the system at the moment. After running ST03N and STAD, we have determined that we are having abnormally high roll-wait, GUI and processing times.
Here are some results from STAD:
CPU time: 2956ms
Processing time: 4417ms
Roll wait time: 1319ms
GUI Time: 2859ms
Can someone direct me towards finding the problem?
If you need more information please let me know. I would post a screenshot if I was able to.
Thanks,
HLHi,
Regarding Performance Of SAP system it will not be possible to Come to conclusion with the one Single data.
GUI time is basically the time spent in front end i.e Your Network load or such time
Coming to CPU time Can you see ST06 what is the CPU Utilization of your system?
Roll Wait time is the time Spent in buffer if you can share us the OS and Memory configuration.
We can Try to tune the Roll memory ( extended memory tuning) so that we can try to reduce the Roll time.
If possible try to find the Top Dialog Response Time Transaction Is it Standard or for Z Programs.
https://cw.sdn.sap.com/cw/docs/DOC-14387
Go through this link you might get clear idea of performance.
Thanks & Regards,
Balaji.S -
Network/roll wait time is high
Hi,
One of my dialog instance is showing high response time at SMLG.
When i am checking Dialog Overview at RZ20, network response time of this server is very high.
When i am checking the parts of response time at ST03, roll wait time is always around 70% for this instance.
(4.7,2003 server, SQL 2000)
Kindly suggest me to resolve this.
Regards,
Gayathry.Hai,
Roll wait time means: During processing of some dialog steps, the user context may be rolled out; for example, during RFCs when the client is waiting for a response from the server. This wait time until the dialog step can continue is called the roll wait time.
try to check what RFC causes this, somtimes this may lead to rollfile overflow and in turn some background jobs might fail due to rollfile overflow.
Check SAP Notes:1100926 and 578118.
Regards,
Yoganand.V
Edited by: Yoganand Vedagiri on Mar 12, 2009 1:47 PM -
Hi All,
We are facing a strange problem.
Our system is performing low just because of the share of high Roll Wait Time.
Can anyone tell why it is so high ? and how to reduce it's time.
Now it's taking more than 60% of Total Resp. Time.
Thanks & Regards,
Santosh RajputHi Santosh,
sometimes we have the same problem when inserting records with MM41. In ST03 Top Response Times the roll wait time takes nearly minutes, the whole response time 5 minutes.
The SAP HELP defines roll wait time as follows:
Wait time in the roll area.
If synchronous RFCs are called, the work process performs a roll out and waits for the end of the RFC in the roll area. The RFC server programs can wait for other RFCs to be sent to them in the roll area.
I'm not quite sure whether this explanation meets your or my problem because the problem occurs just two or three times a day. So - what is the process waiting for?
Can anybody help?
Thanks and best regards,
Frank Ihnen -
Relation between RFC time and Roll wait time
Hello
What is the relation between RFC time and Roll wait time?
In a diagram from E2E100 training I took recently the relationship is made out to be:
RFC+CPIC = some processing time + Roll wait time
Something like this
Processing----
Roll wait-------
Roll in
RFC+CPIC----
In the diagram I saw it shows that roll out occurs during processing.
Assumming that the user context rolls out and then rolls in again, wouldn't "Wait" time occur when the request comes back and the dispatcher has to assign the request to a work process?
Is this wait time calculated as "Wait" time or is it calculated as "Roll wait" time?
I was forced to think about this because of a STAD record I have with high Roll wait time, but with low RFC time as well as zero GUI time. If "Wait" time when the request comes back from RFC is included in "Roll wait" then it might explain what i'm seeing.
Kind regards,
PeterHello Santosh,
Thank you for your reply.
I would like to know how time spend in the dispatcher by a request coming back from an RFC is calculated.
When the RFC is made the user context is rolled out of the work process so the work process becomes free. When the RFC comes back it has to be assigned to a new work process. This means that a bit of time would need to be spent in the dispatcher. Would this time be measured as "wait" time or is it part of "Roll wait time?
From the definition you gave of "Roll wait time", "it is the time dedicated by the R/3 system to waiting for the return value of an RFC call plus the CPIC time required to create the connection for that RFC to an outside system", one would think that the time spent in the dispatcher by a returning RFC would be measured as "wait time" the same as when the dialog request first comes in from the front end. I've never seen a diagram that explained this satisfactorily.
Kind regards,
Peter -
Long Roll-Wait time in SAP (ST03N)
Hello !
On my SAP-PRD, just registering long roll-wait times since almost 6 weeks !! Network diagnostics are normal ! Can somebody give me any hints to the issue !
Thanks in adv.
Kumar Mitra from Germany
====================Hai,
Check the below link.....
http://sap.ittoolbox.com/groups/technical-functional/sap-basis/high-rollwait-time-1514656?cv=expanded
http://www.saptechies.com/rfc-statistics-in-transactions-st03st03n-and-stad/
Roll wait time = time spent in roll context waiting for RFCs.
Also you can check tcode ST05 for RFC trace.
Regards,
Yoganand.V
Edited by: Yoganand Vedagiri on Feb 4, 2009 12:05 PM
Edited by: Yoganand Vedagiri on Feb 4, 2009 12:10 PM -
In t-code ST03N, there are following info:
1. Average Wait Time per Dialog Step (ms)
2. Average Roll In Time (ms)
3. Average Roll Wait Time (ms)
Question:
1. When will Roll Wait happen?
2. What is relationship between Wait Time and Roll Wait Time?
3. When dispatcher looking for free dialog wp, the dispatcher wait time occurs. From above two type of wait time, which one is dispatcher wait time?
Thanks.
JamesHi,
Read the OSS note which explains the decomposition of response time in detail...
Regards,
Olivier -
For my below crm transaction for a perticular user the response time is ~18000 ms out of which 15130 is roll time. Memory used is 16 MB.
"BSPWDWCC_SLS_HOME SAPMHTTP /sap/bc/bsp/sap/CRM_UI_PORTAL/B"
Roll time
Out 5 ms
In 5 ms
Wait 15,130 ms
Where is causing this roll wait ? What memory is not sufficient ? What to look at ?
In st02
roll buffer max used is 25% ,
page buffer max used is 10%.
EM max used 50%.
No swaps for above.
My user memory settings are as below:
ztta/roll_area 9000000
ztta/roll_first 1
ztta/roll_extension 4000000000
thank you
LaxmiThank you Shyam,Krishna.
I checked another similar instance again today with high wait time. Most of the time was wait time. There were NO RFC RECORD. just http record with most of it being on the server (Remote exec. time ~ 16 534 ms).
Now i assume calling time = gate way time + remote exec time. So as most of it was on the server I can rule out network.
I checked st03 and rfc times were really small. Also we have plenty of wps .. ~ 90. So not sure why still the high wait time. Also stad does not give any RFC record. If this was rfc will it now show up ?
I have given below the details. Am I missing some thing ?
Response time 16,546 ms
Wait for work process 5 ms
Processing time 1,155 ms
Load time 28 ms
Generating time 0 ms
Roll (in+wait) time 15,016 ms
Database request time 341 ms
Enqueue time 0 ms
Roll time Out 5 ms
In 3 ms
Wait 15,013 ms
NO RFC SUB RECORD
HTTP Records
as Server
Number Connections 1
Destinations 1
Calls 1
Time Calling 16,540 ms
Remote execution 16,534 ms
Idle 219 ms
HTTP Server connection record
Connection-Id 4B7CAADEF5DA00A7E10080000A68F456
Communication step Send/Receive
Timestamp 20100218 155216 EST
Host sappc003
Port 8031
Path /sap/bc/bsp/sap/crm_ui_frame/BSPWDApplication.do
Status phrase OK
Status code 200
Protocol HTTP
Method GET -
Hello
This was my exam question and still not sure about the answer
You notice there is a very high percentage of wait time for contention event in your Rac database that has frequent insert operations.
Which two recommendations may reduce this problem ?
a-)shorter transactions
b-)increasing sequence cache sizes
c-)using reverse key indexes
d-)uniform and large extents sizes
e-)automatic segment space managemnt
f-)smaller extent sizes Any suggestion ?Again, what exam are we talking about? If you're talking about one of the Oracle certification exams, be aware that you're not allowed to discuss post exam questions on public forums or to discuss them with other candidates. That's a violation of your certification agreement.
Assuming we are not talking about an Oracle certification exam, I'd point out to the instructor that any or all the answers might be correct depending on the wait event.
Justin -
The Average Wait Time of SQL instance "CONFIGMGRSEC" on computer " SEC_SITE_SERVER " is too high
I have a SCCM 2012 SP1 CU4 environment with SCOM monitoring installed.
I also do have 4 secondary sites installed below my primary. The secondaries are using SQL 2012 Express version default deployed using the secondary site installation.
My SCOM monitoring is generating tickets with the following message:
The Average Wait Time of SQL instance "CONFIGMGRSEC" on computer "<SEC_SITE_SERVER>" is too high
How can i solve this ? Or do I need to ignore this ?Never ignore messages, but tune them.
In this specific case you might want to take a look at this:
http://social.technet.microsoft.com/Forums/en-US/ffeefe0d-0ef7-49a3-862e-9be27989dc5d/scom2012-alert-sql-2008-db-average-wait-time-recompilationis-too-high?forum=operationsmanagergeneral
My Blog: http://www.petervanderwoude.nl/
Follow me on twitter: pvanderwoude -
SCOM2012 Alert: SQL 2008 DB Average Wait Time & Recompilationis too high
WE have SCOM 2012sp1 CU3 updated.
I receive the below critical alerts from SQL servers continuously, please help me to resolve this issue.
SQL 2008 DB Average Wait Time is too high
SQL DB 2008 SQL Recompilation is too highI don't know about anyone else but overriding those monitors and rules didn’t work for me, I had to override<o:p></o:p>
SQL Re-Compilation monitor for SQL 2012 DB Engine<o:p></o:p>
SQL Re-Compilation monitor for SQL 2008 DB Engine<o:p></o:p>
Average Wait Time monitor for SQL 2012 DB<o:p></o:p>
Average Wait Time monitor for SQL 2008 DB<o:p></o:p>
Now I am wondering if other monitors are valid as well in particular I have multiple alerts for<o:p></o:p>
Buffer Cache Hit Ratio monitor for SQL 2008 DB Engine is too low<o:p></o:p>
Page Life Expectancy (s) for 2008 DB Engine is too low<o:p></o:p>
is anyone else seeing these issues as well? -
Deadlocks and very high wait times
We are seeing a very high number of deadlocks in the system. The deadlocks trace all show a 'enq: TX - row lock contention' with wait times of around 2929700+ seconds ex:
last wait for 'enq: TX - row lock contention' blocking sess=0x70000006d85e1b8 seq=55793 wait_time=2929704 seconds since wait started=4
name|mode=54580006, usn<<16 | slot=1d0010, sequence=705f
Dumping Session Wait History
for 'enq: TX - row lock contention' count=1 wait_time=2929704
name|mode=54580006, usn<<16 | slot=1d0010, sequence=705f
for 'latch: enqueue hash chains' count=1 wait_time=1649
address=70000006dbb4a20, number=13, tries=0
for 'enq: TX - row lock contention' count=1 wait_time=2929708
name|mode=54580006, usn<<16 | slot=1d0010, sequence=705f
for 'SQL*Net message from client' count=1 wait_time=101740
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to client' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
for 'direct path write temp' count=1 wait_time=921
file number=fb, first dba=6521b, block cnt=2
for 'SQL*Net more data from client' count=1 wait_time=3
driver id=54435000, #bytes=10, =0
for 'SQL*Net more data from client' count=1 wait_time=5
driver id=54435000, #bytes=1e, =0
for 'SQL*Net more data from client' count=1 wait_time=10
driver id=54435000, #bytes=2c, =0
for 'SQL*Net more data from client' count=1 wait_time=5
driver id=54435000, #bytes=3a, =0
Any ideas on how to resolve this ?
Thanks
SuryaSorry for the typo, that Ora-0060 error we are seeing. Here is the deadlock graph:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining Scoring Engine
and Real Application Testing options
ORACLE_HOME = /orasw/product/10.2.0.4.0
System name: AIX
Node name: spda5001
Release: 3
Version: 5
Machine: 00074D5AD400
Instance name: IAMS01P1
Redo thread mounted by this instance: 1
Oracle process number: 21
Unix process pid: 2306444, image: oracle@spda5001
*** 2011-12-24 05:05:39.885
*** SERVICE NAME:(IAMS01P) 2011-12-24 05:05:39.884
*** SESSION ID:(443.2130) 2011-12-24 05:05:39.884
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-00080020-000c3957 21 443 X 58 391 X
TX-001d0010-0000705f 58 391 X 21 443 X
session 443: DID 0001-0015-0000002E session 391: DID 0001-003A-00000081
session 391: DID 0001-003A-00000081 session 443: DID 0001-0015-0000002E
Rows waited on:
Session 391: obj - rowid = 0001098B - AAATtpAAGAAADROAAD
(dictionary objn - 67979, file - 6, block - 13390, slot - 3)
Session 443: obj - rowid = 00010B25 - AAARRwAAGAAAAdgAAN
(dictionary objn - 68389, file - 6, block - 1888, slot - 13)
Information on the OTHER waiting sessions:
Session 391:
pid=58 serial=16572 audsid=52790041 user: 93/IAMS_USR
O/S info: user: , term: , ospid: 1234, machine: mac3023
program:
Current SQL Statement:
update spt_identity set created=:1, modified=:2, owner=:3, assigned_scope=:4, assigned_scope_path=:5, extended1=:6, extended2=:7, extended3=:8, extended4=:9, extended5=:10, extended6=:11, extended7=:12, extended8=:13, extended9=:14, extended10=:15, extended11=:16, extended12=:17, extended13=:18, extended14=:19, extended15=:20, extended16=:21, extended17=:22, extended18=:23, extended19=:24, extended20=:25, name=:26, description=:27, protected=:28, iiqlock=:29, attributes=:30, manager=:31, display_name=:32, firstname=:33, lastname=:34, email=:35, manager_status=:36, inactive=:37, last_login=:38, last_refresh=:39, password=:40, password_expiration=:41, password_history=:42, bundle_summary=:43, assigned_role_summary=:44, correlated=:45, auth_question_lock_start=:46, failed_auth_question_attempts=:47, controls_assigned_scope=:48, certifications=:49, activity_config=:50, preferences=:51, history=:52, scorecard=:53, uipreferences=:54, attribute_meta_data=:55, workgroup=:56 where id=:57
End of information on OTHER waiting sessions.
Current SQL statement for this session:
update spt_workflow_case set created=:1, modified=:2, owner=:3, assigned_scope=:4, assigned_scope_path=:5, stack=:6, attributes=:7, launcher=:8, host=:9, launched=:10, completed=:11, progress=:12, percent_complete=:13, type=:14, messages=:15, name=:16, description=:17, complete=:18, target_class=:19, target_id=:20, target_name=:21, workflow=:22 where id=:23 -
Hello,
Our DB is having very high redo log space wait time :
redo log space requests 867527
redo log space wait time 67752674
LOG_BUFFER is 14 MB and having 6 redo logs groups and the size of redo log file is 500MB for each log file.
Also, the amount of redo generated per hour :
START_DATE START NUM_LOGS MBYTES DBNAME
2008-07-03 10:00 2 1000 TKL
2008-07-03 11:00 4 2000 TKL
2008-07-03 12:00 3 1500 TKL
Does increasing the size of LOG_BUFFER will help to reduce the redo log space wait ?
Thanks in advance ,
Regards,
AmanLooking quickly over the AWR report provided the following information could be helpful:
1. You are currently targeting approx. 6GB of memory with this single instance and the report tells that physical memory is 8GB. According to the advisories it looks like you could decrease your memory allocation without tampering your performance.
In particular the large_pool_size setting seems to be quite high although you're using shared servers.
Since you're using 10.2.0.4 it might be worth to think about using the single SGA_TARGET parameter instead of the specifying all the single parameters. This allows Oracle to size the shared pool components within the given target dynamically.
2. You are currently using a couple of underscore parameters. In particular the "_optimizer_max_permutations" parameter is set to 200 which might reduce significantly the number of execution plans permutations Oracle is investigating while optimizing the statement and could lead to suboptimal plans. It could be worth to check why this has been set.
In addition you are using a non-default setting of "_shared_pool_reserved_pct" which might no longer be necessary if you are using the SGA_TARGET parameter as mentioned above.
3. You are using non-default settings for the "optimizer_index_caching" and "optimizer_index_cost_adj" parameters which favor index-access paths / nested loops. Since the "db file sequntial read" is the top wait event it might be worth to check if the database is doing too excessive index access. Also most of the rows have been fetched by rowid (table fetch by rowid) which could also be an indicator for excessive index access/nested loop usage.
4. You database has been working quite a lot during the 30min. snapshot interval: It processed 123.000.000 logical blocks, which means almost 0.5GB per second. Check the top SQLs, there are a few that are responsible for most of the blocks processed. E.g. there is a anonymous PL/SQL block that has been executed almost 17.000 times during the interval representing 75% of the blocks processed. The statements executed as part of these procedures might be worth to check if they could be tuned to require less logical I/Os. This could be related to the non-default optimizer parameters mentioned above.
5. You are still using the compatible = 9.2.0 setting which means this database could still be opened by a 9i instance. If this is no longer required, you might lift this to the default value of 10g. This will also convert the REDO format to 10g I think which could lead to less amount of redo generated. But be aware of the fact that this is a one-way operation, you can only go back to 9i then via a restore once the compatible has been set to 10.x.
6. Your undo retention is set quite high (> 6000 secs), although your longest query in the AWR period was 151 seconds. It might be worth to check if this setting is reasonable, as you might have quite a large undo tablespace at present. Oracle 10g ignores the setting if it isn't able to honor the setting given the current Undo tablespace size.
7. "parallel_max_servers" has been set to 0, so no parallel operations can take place. This might be intentional but it's something to keep in mind.
Regards,
Randolf
Oracle related stuff:
http://oracle-randolf.blogspot.com/
SQLTools++ for Oracle:
http://www.sqltools-plusplus.org:7676/
http://sourceforge.net/projects/sqlt-pp/ -
High Buffer Busy Wait due to Concurrent INSERTS
Hi All,
One of my OLTP database is running on 11.1.0.7 (11.1.0.7.0 - 64bit Production) with RHEL 5.4.
On frequent basis, i am observing 'BUFFER BUSY WAITS' and last time i tried to capture some dictionary information to dig the waits.
1. Session Watis :
Oracle Sec Hash
Sid,Serial User OS User Svr-Pgm Wait Event State-Seq Wt Module Cmnd Value P1 P2 P3
633,40830 OLTP_USE fateadm 21646-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863905 1
647, 1761 OLTP_USE fateadm 22715-orac buffer busy wai Wtng-3837 0 ORDERS ISRT 3932487748 384 1863905 1
872, 5001 OLTP_USE fateadm 21836-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863905 1
702, 1353 OLTP_USE fateadm 21984-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863905 1
337,10307 OLTP_USE fateadm 21173-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863905 1
751,43016 OLTP_USE fateadm 21619-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863905 1
820,17959 OLTP_USE fateadm 21648-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863905 1
287,63359 OLTP_USE fateadm 27053-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863905 1
629, 1653 OLTP_USE fateadm 22468-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863905 1
788,14160 OLTP_USE fateadm 22421-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863905 1
615, 4580 OLTP_USE fateadm 21185-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863905 1
525,46068 OLTP_USE fateadm 27043-orac buffer busy wai Wtng-9034 1 ORDERS ISRT 3932487748 384 1863905 1
919,23243 OLTP_USE fateadm 21428-orac buffer busy wai Wtng-6340 1 ORDERS ISRT 3932487748 384 1863906 1
610,34557 OLTP_USE fateadm 21679-orac buffer busy wai Wtng-6422 1 ORDERS ISRT 3932487748 384 1863906 1
803, 1583 OLTP_USE fateadm 21580-orac buffer busy wai Wtng-6656 1 ORDERS ISRT 3932487748 384 1863906 1
781, 1523 OLTP_USE fateadm 21781-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863906 1
369,11005 OLTP_USE fateadm 21718-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863906 1
823,35800 OLTP_USE fateadm 21148-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863906 1
817, 1537 OLTP_USE fateadm 22505-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863906 1
579,54959 OLTP_USE fateadm 22517-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863906 1
591,33597 OLTP_USE fateadm 27027-orac buffer busy wai Wtng-9999 1 ORDERS ISRT 3932487748 384 1863906 1
481, 3031 OLTP_USE fateadm 21191-orac buffer busy wai Wtng-3502 1 ORDERS ISRT 3932487748 384 1863906 1
473,24985 OLTP_USE fateadm 22629-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863906 1
868, 3984 OLTP_USE fateadm 27191-orac buffer busy wai Wtng-9999 0 ORDERS ISRT 3932487748 384 1863906 1
select owner,segment_name,segment_type from dba_extents where file_id = 384 and 1863905 between block_id and block_id + blocks -1;
OWNER SEGMENT_NAME SEGMENT_TYPE
ORDER ORDER_DETAILS TABLE
select TABLE_NAME,PARTITIONED,ini_trans ,degree,compression,FREELISTS from dba_TABLES WHERE TABLE_NAME='ORDER_DETAILS';
TABLE_NAME PAR INI_TRANS DEGREE COMPRESS FREELISTS
ORDER_DETAILS NO 1 1 ENABLED 1
Tablespace is not ASSM managed !
select
object_name,
statistic_name,
value
from
V$SEGMENT_STATISTICS
where
object_name = 'ORDER_DETAILS';
OBJECT_NAME STATISTIC_NAME VALUE
ORDER_DETAILS logical reads 487741104
ORDER_DETAILS buffer busy waits 4715174
ORDER_DETAILS db block changes 200858896
ORDER_DETAILS physical reads 143642724
ORDER_DETAILS physical writes 20581330
ORDER_DETAILS physical reads direct 55239903
ORDER_DETAILS physical writes direct 19500551
ORDER_DETAILS space allocated 1.6603E+11
ORDER_DETAILS segment scans 9727
ORDER_DETAILS table is ~ 153 GB non-partitioned table.
It seems its not a READ BY OTHER SESSIONS wait but BUFFER BUSY due to write-wirte contention inside same block. I have never observed Cache Buffer Chain/ ITL-Wait/ High wait time on dbfile sequential/scattered reads.Table contains one PK (composite index on 3 columns) which seems to be highly fragmented.This non partitioned global Index has 3182037735 rows and blevel is 4.
BHAVIK_DBA.FATE1NA>select index_name,status,num_rows,blevel,pct_free,ini_trans,clustering_factor from dba_indexes where index_name='IDX_ORDERS';
INDEX_NAME STATUS NUM_ROWS BLEVEL PCT_FREE INI_TRANS CLUSTERING_FACTOR
IDX_ORDERS VALID 3182037735 4 2 2 2529462377
1 row selected.
One of the index column value is being populated by sequence. (Monotonically increasing value)
SEGMENT_NAME MB
IDX_ORDERS 170590.438
Index size is greater than table size !Tuning goal here is to reduce buffer busy waits and thus commit latencies.
I think, i need to increase FREELISTS and PCT_FREE to address this issue, but not much confident whether it is going to solve the issue or not?
Can i ask for any help here ?Hi Johnathan;
Many thanks for your detailed write-up. I was expecting you !
Your post here gave lot of information and wisdom that made me think last couple of hrs that is the reason for the delay in reply.
I did visited your index explosion posts couple of times and that scenario only gave me insight that concurrent DML (INSERT) is cause of index fragmentation in my case.
Let me also pick the opportunity to ask you to shed more light on some of the information you have highlighted.
if you can work out the number of concurrent inserts that are really likely to happen at any one instant then a value of freelists that in the range of
concurrency/4 to concurrency/2 is probably appropriate.May i ask you how did you derive this formula ? I dont want to miss learning opportunity here !
Note - with multiple freelists, you may find that you now get buffer busy waits on the segment header block.I did not quite get this point ? Can you shed more light please? What piece in segment header block is going to result contention(BBW on SEGMENT HEADER) on all concurrent inserts ?
The solution to this is to increase the number of freelist groups (making sure that
freelists and freelist groups have no common factors).My prod db NON-RAC environment. Can i use FREELIST GROUPS here ? My little knowledge did not get, What "common factors" you are referring here ?
The reads could be related to leaf block splits, but there are several possible scenarios that could lead to that pattern of activity - so the next step is to find out which blocks are being
read. Capture a sample of the waits, then query dba_extents for the extent_id, file_id, and block_id (don't run that awful query with the "block_id + blocks" predicate) and cross-check the
list of blocks to see if they are typically the first couple of blocks of an extent or randomly scattered throughout extents. If the former the problem is probably related to ASSM, if the
latter it may be related to failed probes on index leaf block reuse (i.e. after large scale deletes).I have 10046 trace file with me (giving you some sample below) that can give some information. However, since the issue was critical, i killed the insert process and rebuilt both the indexes. Since, index is rebuilt, i am not able to find any information in dba_extents.
select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=42 and block_id=1109331;
no rows selected
select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=42 and block_id=1109395 ;
no rows selected
select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=42 and block_id=1109459;
no rows selected
select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=10 and block_id=1107475;
no rows selected
select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=10 and block_id=1107539;
no rows selected
select object_name,object_Type from dba_objects where object_id=17599;
no rows selected
WAIT #4: nam='db file sequential read' ela= 49 file#=42 block#=1109331 blocks=1 obj#=17599 tim=1245687162307379
WAIT #4: nam='db file sequential read' ela= 59 file#=42 block#=1109395 blocks=1 obj#=17599 tim=1245687162307462
WAIT #4: nam='db file sequential read' ela= 51 file#=42 block#=1109459 blocks=1 obj#=17599 tim=1245687162307538
WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107475 blocks=1 obj#=17599 tim=1245687162307612
WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107539 blocks=1 obj#=17599 tim=1245687162307684
WAIT #4: nam='db file sequential read' ela= 198 file#=10 block#=1107603 blocks=1 obj#=17599 tim=1245687162307905
WAIT #4: nam='db file sequential read' ela= 88 file#=10 block#=1107667 blocks=1 obj#=17599 tim=1245687162308016
WAIT #4: nam='db file sequential read' ela= 51 file#=10 block#=1107731 blocks=1 obj#=17599 tim=1245687162308092
WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107795 blocks=1 obj#=17599 tim=1245687162308166
WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107859 blocks=1 obj#=17599 tim=1245687162308240
WAIT #4: nam='db file sequential read' ela= 52 file#=10 block#=1107923 blocks=1 obj#=17599 tim=1245687162308314
WAIT #4: nam='db file sequential read' ela= 57 file#=42 block#=1109012 blocks=1 obj#=17599 tim=1245687162308395
WAIT #4: nam='db file sequential read' ela= 52 file#=42 block#=1109076 blocks=1 obj#=17599 tim=1245687162308470
WAIT #4: nam='db file sequential read' ela= 98 file#=42 block#=1109140 blocks=1 obj#=17599 tim=1245687162308594
WAIT #4: nam='db file sequential read' ela= 67 file#=42 block#=1109204 blocks=1 obj#=17599 tim=1245687162308686
WAIT #4: nam='db file sequential read' ela= 53 file#=42 block#=1109268 blocks=1 obj#=17599 tim=1245687162308762
WAIT #4: nam='db file sequential read' ela= 54 file#=42 block#=1109332 blocks=1 obj#=17599 tim=1245687162308841
WAIT #4: nam='db file sequential read' ela= 55 file#=42 block#=1109396 blocks=1 obj#=17599 tim=1245687162308920
WAIT #4: nam='db file sequential read' ela= 54 file#=42 block#=1109460 blocks=1 obj#=17599 tim=1245687162308999
WAIT #4: nam='db file sequential read' ela= 52 file#=10 block#=1107476 blocks=1 obj#=17599 tim=1245687162309074
WAIT #4: nam='db file sequential read' ela= 89 file#=10 block#=1107540 blocks=1 obj#=17599 tim=1245687162309187
WAIT #4: nam='db file sequential read' ela= 407 file#=10 block#=1107604 blocks=1 obj#=17599 tim=1245687162309618TKPROF for above trace
INSERT into
order_rev
(aggregated_revenue_id,
legal_entity_id,
gl_product_group,
revenue_category,
warehouse_id,
tax_region,
gl_product_subgroup,
total_shipments,
total_units_shipped,
aggregated_revenue_amount,
aggregated_tax_amount,
base_currency_code,
exchange_rate,
accounting_date,
inventory_owner_type_id,
fin_commission_structure_id,
seller_of_record_vendor_id,
organizational_unit_id,
merchant_id,
last_updated_date,
revenue_owner_type_id,
sales_channel,
location)
VALUES
(order_rev.nextval,:p1,:p2,:p3,:p4,:p5,:p6,:p7,:p8,:p9,:p10,:p11,:p12,to_date(:p13, 'dd-MON-yyyy'),:p14,:p15,:p16,:p17,:p18,sysdate,:p19,:p20,:p21)
call count cpu elapsed disk query current rows
Parse 0 0.00 0.00 0 0 0 0
Execute 613 5.50 40.32 96672 247585 306916 613
Fetch 0 0.00 0.00 0 0 0 0
total 613 5.50 40.32 96672 247585 306916 613
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 446
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 164224 0.04 62.33
SQL*Net message to client 613 0.00 0.00
SQL*Net message from client 613 0.03 0.90
latch: cache buffers chains 8 0.00 0.00
latch: object queue header operation 2 0.00 0.00Is there any other way to find out culprit amongst the two you have listed (ASSM / failed probes on index leaf block reuse ) ? -
I was always under the impression that Wait Timers (e.g. Wait and Wait Ms Multiple) always executed last within a while loop. As I'm reading up on timing of real-time loops I'm finding that NI suggested using a stacked sequence structure to force timing to run last. Has my assumption of wait timer execution been wrong all this time or do PC and RT systems differ in how loops handle timing?
Thanks,
CraigSo, ok, coming back on this.
Attached you find a small example to show my remarks in the previous post.
How to use it:
The main focus is in the first iterations once "Wait now!" is pressed. So it is usually not of interest to run the "waiting" for more than 5-10 iterations.
In order to see an "understandable" behavior, i suggest to wait at least 100ms. I configured "Time To Wait" to be at least 10 (coercing).
The second focus is "Time Value".
For "Wait (ms)", the "Time Value" will have any number, but increase each iteration by the amount of "Time To Wait".
For "Wait Until Next Multiple (ms)", "Time Value" will be initialized to a value, which is a integer multiple of the "Time To Wait" and will increase each iteration to the next multiple.
My example does not contain parallel code, so it does not show the parallelism of the waiting functions!
Nevertheless, understanding the shown behavior is the key. Because if the code running in parallel to the waiting requires less execution time than the wait function will wait, the behavior will be exactly as shown.
If the parallel code requires more execution time than the waiting, "Wait (ms)" is already finished and therefore the loop will immediatly continue with the next iteration (high CPU load!).
If the parallel code requires more execution time than the waiting, "Wait Until Nexty Multiple (ms)" will wait that long that the "Time Value" is again in line to an integer multiple of the "Time To Wait" (so this one keeps on running in parallel until waiting time is calculated and over). Hence, "Wait Until Next Multiple" will still introduce a waiting time (reduced CPU load), but you will "miss complete iteration slots".
As you can see, using a single "Wait Until Next Multiple" outside the loop for initialization can make sense if the first iteration should also have a "close to normal" execution time.
Please note that the Timed Loop does something similar once started. As it does it during the iterations, the first 1-3 iteration(s) are called "warmup iterations". Most often, the first single iteration is sufficient for this though....
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.
Attachments:
TimingVI_Behavior_2011.vi 29 KB
Maybe you are looking for
-
Problem while starting the managed server
Hi We have following configuration :- We have 2 physical sesrvers :- Server1 and Server2 in Sun Cluster running Solaris 10. We also have Oracle DB server 9.2.0.8 in RAC mode, one instance each on Server1 & Server2. Server 1 :- cluster1 :- 1 Managed S
-
How to Deploy war file in oracle apps 12
Hi, Please advice the steps to deploy war file on the Oracle apps release 12. Thanks..
-
Pop up with long formatted text
Hai all, I need to create a popup with a long formatted text, i tried many FM's but cannot find one which does the job. the text i need to display = Disclaimer (verplicht) (This is the title) Door akkoord te geven op deze disclaimer, verklaar je uits
-
Editing ABAP Program with out using ABAP Editor
Dear Gurus, I know that ABAP Programs like SAPMSSY1, etc. are stored in Table D010S or REPOSRC as per the version of the SAP Released. my problem is that whem i open the table REPOSRC (Having around 34 colums), no doubt thati found the program list b
-
Accessing Flash variables in MCs (not global)
Hello, Just to make sure : you can't access variables that are in movieclips, can you ? I suppose I need to write get/set function whithin Flash to alter those variable ? Thanks PJ