Performance issue of Concurrency in TimesTen
Hi,
Our project is to test the performance of TimesTen. But we found performance issue when multi processes run concurrently.
[Product Version]
OS: HP-UX B.11.11 U 4CPU
Oracle DB: 9.2.0.4 ??
TimesTen: 7.0.2
[Deployment Description]
Oracle DB and TimesTen is respectively deployed in 2 HP-Unix servers, our application is deployed in the same machine with TimesTen, and our application is connected with TT in CS mode rather than direct connect.
[Scenario Description]
Our test program is quite simple, just use Transparent load to load data from oracle DB into TT cache group.
the transparent load method we used is "select every field from table where id=:b"
For example, there are 100k records in oracle DB. We unload the tt cache group and run the test program, in which every record is loaded into TT through "select" statement.
[Performance Issue]
If we just start one test process to load 10k records into TT from DB, it will cost 19s, If we add the test processes up to 4, and every process load different 10k records into TT from DB, it will cost 23s, Howver, if the test processes is up to 10, and every process load different 10k records into TT from DB, it will cost 55s I think the CPU is not the bottleneck, since in the three scenarios, every CPU used is no more than 40%. and the bandwidth is also sufficient, since we can load 100K records into TT from DB by full load in 8s.
So, what cause the performance issue in high concurrency? Why the performance drops deeply when the number of concurrent processes is more than the number of CPU?
Hi,
What does the cache group definition look like please? How many CPUs are there in the Oracle server machine? What are the DSN settings for the datastore?
If you are loading each record just once into TimesTen from Oracle using TransparentLoad (i.e. during the test run you SELECT each record just once) then you are actually primarily testing the performance of the Oracle DB rather than TimesTen since each SELECT will result in a SELECT in TimesTen, then a SELECT in Oracle (multiple SELECTS if the cache groups has multiple tables in it) followed by INSERT(s) into TimesTen. As long as your DSN settings are correct then the bottleneck here will almost certainly be the SELECTs in Oracle. I would take a look at the performance of the Oracle DB while the highly concurrent test is running.
Chris
Similar Messages
-
Performance Issue with Concurrent Program
Hi Gurus,
I have a loader program which updates some information in the OM sales orders, this is been done using oe_order_pub.process_order and I do not see any performance issue in the package.
Using this program i had tried to update some huge number of orders uand it gets completed in few minutes, but at times this program runs for more than 6-7 hours even with
Pls could anyone advise what could be the issue with this?
Thanks & Regards,
GenooI have a loader program which updates some information in the OM sales orders, this is been done using oe_order_pub.process_order and I do not see any performance issue in the package.
Using this program i had tried to update some huge number of orders uand it gets completed in few minutes, but at times this program runs for more than 6-7 hours even with
Pls could anyone advise what could be the issue with this?Do you have the statistics collected up to date?
Can you find any errors in the database log file?
Any locks in the database?
Any invalid objects?
Please enable trace and generate the TKPROF file once the program is completed.
Thanks,
Hussein -
Asking help for performance issues about concurrent package
One of my friends is developing a service based on resin. They use thread pool of current package in jdk1.5. The service will create a lot of threads in the thread pool. And most of the threads are waiting. What they can not make sure is how the large amount of waiting threads will affect the performance. These threads surely will occupy a lot of memory. But how will they affect the cpu?
Some documents on the Internet say that the large amount of waiting threads will largely increase the thread switching overhead. And some others say no because the scheduler will not be affected by waiting threads. I'm not sure which one is true. Would anyone like to give me some tips? It's better if you can point out any our documents about it.
Thanks!No, it just depends on Data Structures 101.
You would have a list of ready threads, from which you would allocate one to the processor on some priority and fairness scheme, and another list of non-ready threads which you would only promote to the ready list when something happened to them that made them ready.
And among the ready threads you would most likely use a priority queue, so that operations on it were O(log(N)). And if the ready list also included the unready list for some strange reason, operations on it would still be O(log(N)), i.e. less than linear in the total number of threads.
And if for some strange reason it was implemented in a less efficient way than that, I would complain vociferously to the vendor. Scheduling has been going on for fifty years after all. -
Performance issue with BW Reports for concurrent users
All:
I notice that one BW report that runs in few seconds will take upto 3.5 minutes when multiple users run it at the same time.
We are in BW 3.5 and we run reports for portal.
Does anyone have any suggestion about how to handle performance issue with concurrent users?
Thanks,
PranabHi,
use the OLAP-cache and/or aggregates and consider using load balancing to distribute the users to several servers.
Kind regards
/martin -
Concurrent program performance Issue
Hi,
We are currently experiencing performance issue in one of the concurrent program related
to the HR module. The concurrent request is currently completing in 3 hrs time.
We have obtained a trace for the concurrent program.
Please help me analyze the cause of the performance issue from the trace file.
Trace file below:
BEGIN SLC_PYINF_USMONACCROH_PKG.SLC_421_HANDLE_OUTBOUND(:errbuf,:rc,:A0,:A1,
:A2,:A3,:A4,:A5,:A6,:A7,:A8,:A9,:A10,:A11); END;
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 76.08 9602.16 700828 1330818 663813 1
Fetch 0 0.00 0.00 0 0 0 0
total 2 76.08 9602.16 700828 1330818 663813 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
SQL*Net message to client 3 0.00 0.00
SQL*Net message from client 3 0.00 0.00
PL/SQL lock timer 969 9.83 9485.16
UPDATE HRAPPS.SLC_PYINF_USMONACCRO_STG SET PROCESS_STATUS = 2
WHERE
CONC_REQUEST_ID = :B2 AND SET_SEQUENCE_NUM = :B1 AND PROCESS_STATUS = 1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 24.83 45.67 145127 695479 602714 560730
Fetch 0 0.00 0.00 0 0 0 0
total 2 24.83 45.67 145127 695479 602714 560730
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
0 UPDATE SLC_PYINF_USMONACCRO_STG (cr=684898 pr=134556 pw=0 time=44759708 us)
1135266 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=694708 pr=124937 pw=0 time=6874212 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 15622 1.43 13.94
db file sequential read 25578 0.52 14.30
latch: cache buffers lru chain 3 0.00 0.00
DELETE FROM SLC_PYINF_USMONACCRO_ARC
WHERE
EXTRACT_DATE<TRUNC(SYSDATE)-60
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 7.41 15.05 87598 87668 0 0
Fetch 0 0.00 0.00 0 0 0 0
total 2 7.41 15.06 87598 87668 0 0
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
0 DELETE SLC_PYINF_USMONACCRO_ARC (cr=87668 pr=87598 pw=0 time=15053606 us)
0 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_ARC (cr=87668 pr=87598 pw=0 time=15053595 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 3 0.00 0.00
db file scattered read 11025 0.61 13.21
SELECT COUNT(*)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B1
call count cpu elapsed disk query current rows
Parse 2 0.00 0.00 0 0 0 0
Execute 2 0.00 0.00 0 0 0 0
Fetch 2 10.14 10.23 116633 123540 0 2
total 6 10.14 10.23 116633 123540 0 2
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=61770 pr=58317 pw=0 time=5290475 us)
560730 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=61770 pr=58317 pw=0 time=1689204 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 15646 0.27 6.24
db file sequential read 625 0.00 0.01
SELECT COUNT(*)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B1 AND
PROCESS_STATUS = 2
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 5.20 8.32 51482 69842 0 1
total 3 5.20 8.32 51482 69842 0 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=69842 pr=51482 pw=0 time=8323369 us)
560730 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=69842 pr=51482 pw=0 time=2811304 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 6514 0.30 6.09
db file sequential read 114 0.00 0.02
SELECT MAX(SET_SEQUENCE_NUM)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 5.34 6.63 58318 61770 0 1
total 3 5.34 6.63 58318 61770 0 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=61770 pr=58318 pw=0 time=6639527 us)
560730 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=61770 pr=58318 pw=0 time=2250410 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 7820 0.30 4.46
db file sequential read 313 0.00 0.05
SELECT COUNT(*)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B2 AND
SET_SEQUENCE_NUM = :B1 AND PROCESS_STATUS = 1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 4.99 4.88 58315 61770 0 1
total 3 4.99 4.88 58315 61770 0 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=61770 pr=58315 pw=0 time=4887337 us)
560730 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=61770 pr=58315 pw=0 time=1688451 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 7824 0.00 3.02
db file sequential read 313 0.00 0.00
SELECT COUNT(*)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B1 AND
PROCESS_STATUS = 1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 4.98 4.87 58318 61770 0 1
total 3 4.98 4.87 58318 61770 0 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=61770 pr=58318 pw=0 time=4872548 us)
560730 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=61770 pr=58318 pw=0 time=1688407 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 7821 0.00 2.98
db file sequential read 312 0.00 0.00
SELECT COUNT(*)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B1 AND
PROCESS_STATUS = -1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 4.45 4.36 58317 61770 0 1
total 3 4.45 4.36 58317 61770 0 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=61770 pr=58317 pw=0 time=4369473 us)
0 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=61770 pr=58317 pw=0 time=4369425 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 7823 0.00 2.98
db file sequential read 312 0.00 0.00
SELECT COUNT(*)
FROM
HRAPPS.SLC_PYINF_USMONACCRO_STG WHERE CONC_REQUEST_ID = :B1 AND
PROCESS_STATUS < 0
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 4.14 4.24 51481 61770 0 1
total 3 4.14 4.24 51481 61770 0 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1 SORT AGGREGATE (cr=61770 pr=51481 pw=0 time=4243020 us)
0 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_STG (cr=61770 pr=51481 pw=0 time=4242968 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 6537 0.06 2.90
db file sequential read 104 0.00 0.00
DELETE FROM SLC_PYINF_USMONACCRO_GLI_ARC
WHERE
EXTRACT_DATE<TRUNC(SYSDATE)-60
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.63 2.52 7681 7689 0 0
Fetch 0 0.00 0.00 0 0 0 0
total 2 0.63 2.52 7681 7689 0 0
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
0 DELETE SLC_PYINF_USMONACCRO_GLI_ARC (cr=7689 pr=7681 pw=0 time=2521592 us)
0 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_GLI_ARC (cr=7689 pr=7681 pw=0 time=2521583 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 1 0.00 0.00
db file scattered read 976 1.00 2.36
UPDATE HRAPPS.SLC_PYINF_USMONACCRO_GLI_STG SET PROCESS_STATUS = 2
WHERE
CONC_REQUEST_ID = :B1 AND PROCESS_STATUS = 1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 1.89 2.25 5863 16125 60963 52309
Fetch 0 0.00 0.00 0 0 0 0
total 2 1.89 2.25 5863 16125 60963 52309
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
0 UPDATE SLC_PYINF_USMONACCRO_GLI_STG (cr=11787 pr=1273 pw=0 time=1332023 us)
122679 TABLE ACCESS FULL SLC_PYINF_USMONACCRO_GLI_STG (cr=16291 pr=5859 pw=0 time=48501241 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file scattered read 745 0.01 0.76
db file parallel read 1 0.00 0.00
db file sequential read 5 0.00 0.00
SELECT B.ATTRIBUTE1 ,B.ATTRIBUTE2 ,B.ATTRIBUTE3 ,T.FLEX_VALUE_MEANING ,
T.DESCRIPTION
FROM
FND_FLEX_VALUES_TL T ,FND_FLEX_VALUES B WHERE B.FLEX_VALUE_ID =
T.FLEX_VALUE_ID AND T.LANGUAGE = USERENV ('LANG') AND TRIM(UPPER
(B.FLEX_VALUE)) = TRIM(UPPER (:B1 )) AND B.ENABLED_FLAG = 'Y' AND UPPER
(B.VALUE_CATEGORY) = UPPER ('SLCHR_INTERFACE_CLEANUP')
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 2 0.00 0.00 0 0 0 0
Fetch 2 0.25 0.86 1640 3286 0 2
total 5 0.25 0.86 1640 3286 0 2
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
2 NESTED LOOPS (cr=3286 pr=1640 pw=0 time=866461 us)
2 TABLE ACCESS FULL FND_FLEX_VALUES (cr=3280 pr=1637 pw=0 time=848331 us)
2 TABLE ACCESS BY INDEX ROWID FND_FLEX_VALUES_TL (cr=6 pr=3 pw=0 time=18101 us)
2 INDEX UNIQUE SCAN FND_FLEX_VALUES_TL_U1 (cr=4 pr=2 pw=0 time=9705 us)(object id 849241)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 4 0.00 0.02
db file scattered read 208 0.30 0.71
SELECT PHASE_CODE, STATUS_CODE, COMPLETION_TEXT, PHASE.LOOKUP_CODE,
STATUS.LOOKUP_CODE, PHASE.MEANING, STATUS.MEANING
FROM
FND_CONCURRENT_REQUESTS R, FND_CONCURRENT_PROGRAMS P, FND_LOOKUPS PHASE,
FND_LOOKUPS STATUS WHERE PHASE.LOOKUP_TYPE = :B3 AND PHASE.LOOKUP_CODE =
DECODE(STATUS.LOOKUP_CODE, 'H', 'I', 'S', 'I', 'U', 'I', 'M', 'I',
R.PHASE_CODE) AND STATUS.LOOKUP_TYPE = :B2 AND STATUS.LOOKUP_CODE =
DECODE(R.PHASE_CODE, 'P', DECODE(R.HOLD_FLAG, 'Y', 'H',
DECODE(P.ENABLED_FLAG, 'N', 'U', DECODE(SIGN(R.REQUESTED_START_DATE -
SYSDATE),1,'P', R.STATUS_CODE))), 'R', DECODE(R.HOLD_FLAG, 'Y', 'S',
DECODE(R.STATUS_CODE, 'Q', 'B', 'I', 'B', R.STATUS_CODE)), R.STATUS_CODE)
AND (R.CONCURRENT_PROGRAM_ID = P.CONCURRENT_PROGRAM_ID AND
R.PROGRAM_APPLICATION_ID= P.APPLICATION_ID ) AND REQUEST_ID = :B1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 971 0.25 0.16 0 0 0 0
Fetch 971 0.53 0.65 0 13605 0 971
total 1943 0.78 0.81 0 13605 0 971
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
971 TABLE ACCESS BY INDEX ROWID FND_LOOKUP_VALUES (cr=17489 pr=0 pw=0 time=877481 us)
2913 NESTED LOOPS (cr=16518 pr=0 pw=0 time=1643550 us)
971 NESTED LOOPS (cr=11663 pr=0 pw=0 time=658551 us)
971 NESTED LOOPS (cr=5837 pr=0 pw=0 time=95374 us)
971 TABLE ACCESS BY INDEX ROWID FND_CONCURRENT_REQUESTS (cr=2924 pr=0 pw=0 time=63054 us)
971 INDEX UNIQUE SCAN FND_CONCURRENT_REQUESTS_U1 (cr=1953 pr=0 pw=0 time=43874 us)(object id 240792)
971 TABLE ACCESS BY INDEX ROWID FND_CONCURRENT_PROGRAMS (cr=2913 pr=0 pw=0 time=28198 us)
971 INDEX UNIQUE SCAN FND_CONCURRENT_PROGRAMS_U1 (cr=1942 pr=0 pw=0 time=17956 us)(object id 849182)
971 TABLE ACCESS BY INDEX ROWID FND_LOOKUP_VALUES (cr=5826 pr=0 pw=0 time=558105 us)
971 INDEX RANGE SCAN FND_LOOKUP_VALUES_U1 (cr=4855 pr=0 pw=0 time=539171 us)(object id 906518)
971 INDEX RANGE SCAN FND_LOOKUP_VALUES_U1 (cr=4855 pr=0 pw=0 time=172115 us)(object id 906518)
SELECT MAX(LT.SECURITY_GROUP_ID)
FROM
FND_LOOKUP_TYPES LT WHERE LT.VIEW_APPLICATION_ID = :B2 AND LT.LOOKUP_TYPE =
:B1 AND LT.SECURITY_GROUP_ID IN (0,
TO_NUMBER(DECODE(SUBSTRB(USERENV('CLIENT_INFO'),55,1), ' ', '0', NULL, '0',
SUBSTRB(USERENV('CLIENT_INFO'),55,10))))
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1945 0.11 0.11 0 0 0 0
Fetch 1945 0.18 0.10 0 3890 0 1945
total 3891 0.29 0.21 0 3890 0 1945
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Rows Row Source Operation
1945 SORT AGGREGATE (cr=3890 pr=0 pw=0 time=142954 us)
1945 FIRST ROW (cr=3890 pr=0 pw=0 time=96520 us)
1945 INDEX RANGE SCAN (MIN/MAX) FND_LOOKUP_TYPES_U1 (cr=3890 pr=0 pw=0 time=89938 us)(object id 906517)
INSERT INTO HRAPPS.SLC_HRINF_INT_SUMMARY (INT_SUMMARY_ID,
INT_SUMMARY_CREATE_DATE ,INT_SUMMARY_LAST_UPDATE_DATE, INTERFACE_NAME ,
HANDLER_CONC_REQUEST_ID, INT_CONC_REQUEST_ID ,SET_SEQUENCE_NUMBER,
SET_RECORD_COUNT, INT_FROM_DATE ,INT_TO_DATE, INT_STATUS_1_STATE,
INT_STATUS_1_MESSAGE ,INT_STATUS_1_STARTED, INT_STATUS_1_COMPLETED ,
INT_STATUS_1_SUCCESS_COUNT, INT_STATUS_1_ERROR_COUNT ,INT_STATUS_2_STATE,
INT_STATUS_2_MESSAGE ,INT_STATUS_2_STARTED, INT_STATUS_2_COMPLETED ,
INT_STATUS_2_SUCCESS_COUNT, INT_STATUS_2_ERROR_COUNT ,INT_STATUS_3_STATE,
INT_STATUS_3_MESSAGE ,INT_STATUS_3_STARTED, INT_STATUS_3_COMPLETED ,
INT_STATUS_3_SUCCESS_COUNT, INT_STATUS_3_ERROR_COUNT ,INT_STATUS_4_STATE,
INT_STATUS_4_MESSAGE ,INT_STATUS_4_STARTED, INT_STATUS_4_COMPLETED ,
INT_STATUS_4_SUCCESS_COUNT, INT_STATUS_4_ERROR_COUNT ,INT_STATUS_5_STATE,
INT_STATUS_5_MESSAGE ,INT_STATUS_5_STARTED, INT_STATUS_5_COMPLETED ,
INT_STATUS_5_SUCCESS_COUNT, INT_STATUS_5_ERROR_COUNT )
VALUES
(:B7 , :B6 , :B6 , :B5 , :B4 , NULL , NULL, NULL, :B3 , :B2 , :B1 , NULL ,
NULL, NULL , NULL, NULL , :B1 , NULL , NULL, NULL , NULL, NULL , :B1 , NULL
, NULL, NULL , NULL, NULL , :B1 , NULL , NULL, NULL , NULL, NULL , :B1 ,
NULL , NULL, NULL , NULL, NULL )
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.01 0.12 12 1 12 1
Fetch 0 0.00 0.00 0 0 0 0
total 2 0.01 0.12 12 1 12 1
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 70 (recursive depth: 1)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 12 0.02 0.12
Thanks & Regards,
RupHi;
Please check our previous topic
Concurrent manager real time tune
Oracle apps database
tune concurrent manager
Oracle apps database
Concurrent Manager very slow
Concurrent Manager very slow........
Regard
Helios -
Performance issue in DB need help with analysing this ADDM report
Hi,
My environment:
Os: RHEL5U3 / 11.1.0.7 64 bit / R12.1.1 64 bit
Issue:
Few days are am facing serious of performance problem in our Production instance. Normally the issue will occur 5 to 10 minutes occasionally per day. At the time of issue we not able to access the EBS application its taking time to load. But backend all the oracle, listener and apps services are up and running. No locks at table and session level. Cpu and memory usage is normal.
We have monitored using "Enterprise Manager" for this issue and we found the wait session present more in Active session tab. At this time EBS application is not able access its loading too time. After some time the in Active session tab the wait session came normal and when we try to access the EBS application its working fine.
We try to find the cause of the issue by running addm report. But am not able to understand what its says. Kindly suggests me
ADDM Report for Task 'TASK_42656'
Analysis Period
AWR snapshot range from 14754 to 14755.
Time period starts at 17-APR-12 11.00.22 AM
Time period ends at 17-APR-12 12.00.33 PM
Analysis Target
Database 'PRD' with DB ID 1789440879.
Database version 11.1.0.7.0.
ADDM performed an analysis of instance PRD, numbered 1 and hosted at
advgrpdb.advgroup.ae.
Activity During the Analysis Period
Total database time was 18674 seconds.
The average number of active sessions was 5.17.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 Top SQL by DB Time 3.43 | 66.33 5
2 Buffer Busy 2.52 | 48.81 5
3 Buffer Busy 1.39 | 26.81 2
4 Log File Switches .91 | 17.56 1
5 Buffer Busy .56 | 10.87 2
6 Undersized SGA .38 | 7.37 1
7 Commits and Rollbacks .28 | 5.42 1
8 Undo I/O .18 | 3.53 0
9 CPU Usage .13 | 2.57 1
10 Top SQL By I/O .11 | 2.21 1
Findings and Recommendations
Finding 1: Top SQL by DB Time
Impact is 3.43 active sessions, 66.33% of total activity.
SQL statements consuming significant database time were found.
Recommendation 1: SQL Tuning
Estimated benefit is 1.59 active sessions, 30.8% of total activity.
Action
Investigate the SQL statement with SQL_ID "a49xsqhv0h31b" for possible
performance improvements.
Related Object
SQL statement with SQL_ID a49xsqhv0h31b.
SELECT R.Conc_Login_Id, R.Request_Id, R.Phase_Code, R.Status_Code,
P.Application_ID, P.Concurrent_Program_ID, P.Concurrent_Program_Name,
R.Enable_Trace, R.Restart, DECODE(R.Increment_Dates, 'Y', 'Y', 'N'),
R.NLS_Compliant, R.OUTPUT_FILE_TYPE, E.Executable_Name,
E.Execution_File_Name, A2.Basepath, DECODE(R.Stale, 'Y', 'C',
P.Execution_Method_Code), P.Print_Flag, P.Execution_Options,
DECODE(P.Srs_Flag, 'Y', 'Y', 'Q', 'Y', 'N'), P.Argument_Method_Code,
R.Print_Style, R.Argument_Input_Method_Code, R.Queue_Method_Code,
R.Responsibility_ID, R.Responsibility_Application_ID, R.Requested_By,
R.Number_Of_Copies, R.Save_Output_Flag, R.Printer, R.Print_Group,
R.Priority, U.User_Name, O.Oracle_Username,
O.Encrypted_Oracle_Password, R.Cd_Id, A.Basepath,
A.Application_Short_Name, TO_CHAR(R.Requested_Start_Date,'YYYY/MM/DD
HH24:MI:SS'), R.Nls_Language, R.Nls_Territory,
R.Nls_Numeric_Characters, DECODE(R.Parent_Request_ID, NULL, 0,
R.Parent_Request_ID), R.Priority_Request_ID, R.Single_Thread_Flag,
R.Has_Sub_Request, R.Is_Sub_Request, R.Req_Information,
R.Description, R.Resubmit_Time, TO_CHAR(R.Resubmit_Interval),
R.Resubmit_Interval_Type_Code, R.Resubmit_Interval_Unit_Code,
TO_CHAR(R.Resubmit_End_Date,'YYYY/MM/DD HH24:MI:SS'),
Decode(E.Execution_File_Name, NULL, 'N', Decode(E.Subroutine_Name,
NULL, Decode(E.Execution_Method_Code, 'I', 'Y', 'J', 'Y', 'N'),
'Y')), R.Argument1, R.Argument2, R.Argument3, R.Argument4,
R.Argument5, R.Argument6, R.Argument7, R.Argument8, R.Argument9,
R.Argument10, R.Argument11, R.Argument12, R.Argument13, R.Argument14,
R.Argument15, R.Argument16, R.Argument17, R.Argument18, R.Argument19,
R.Argument20, R.Argument21, R.Argument22, R.Argument23, R.Argument24,
R.Argument25, X.Argument26, X.Argument27, X.Argument28, X.Argument29,
X.Argument30, X.Argument31, X.Argument32, X.Argument33, X.Argument34,
X.Argument35, X.Argument36, X.Argument37, X.Argument38, X.Argument39,
X.Argument40, X.Argument41, X.Argument42, X.Argument43, X.Argument44,
X.Argument45, X.Argument46, X.Argument47, X.Argument48, X.Argument49,
X.Argument50, X.Argument51, X.Argument52, X.Argument53, X.Argument54,
X.Argument55, X.Argument56, X.Argument57, X.Argument58, X.Argument59,
X.Argument60, X.Argument61, X.Argument62, X.Argument63, X.Argument64,
X.Argument65, X.Argument66, X.Argument67, X.Argument68, X.Argument69,
X.Argument70, X.Argument71, X.Argument72, X.Argument73, X.Argument74,
X.Argument75, X.Argument76, X.Argument77, X.Argument78, X.Argument79,
X.Argument80, X.Argument81, X.Argument82, X.Argument83, X.Argument84,
X.Argument85, X.Argument86, X.Argument87, X.Argument88, X.Argument89,
X.Argument90, X.Argument91, X.Argument92, X.Argument93, X.Argument94,
X.Argument95, X.Argument96, X.Argument97, X.Argument98, X.Argument99,
X.Argument100, R.number_of_arguments, C.CD_Name,
NVL(R.Security_Group_ID, 0), NVL(R.org_id, 0) FROM
fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application
A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C,
fnd_concurrent_queues Q, fnd_application A2, fnd_executables E,
fnd_conc_request_arguments X WHERE R.Status_code = 'I' And
((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or
(R.OPS_INSTANCE =
decode(:dcp_on,1,FND_CONC_GLOBAL.OPS_INST_NUM,R.OPS_INSTANCE))) And
R.Request_ID = X.Request_ID(+) And R.Program_Application_Id =
P.Application_Id(+) And R.Concurrent_Program_Id =
P.Concurrent_Program_Id(+) And R.Program_Application_Id =
A.Application_Id(+) And P.Executable_Application_Id =
E.Application_Id(+) And P.Executable_Id =
E.Executable_Id(+) And P.Executable_Application_Id =
A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id
= C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id =
:q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag
is NULL OR P.Enabled_Flag = 'Y') And R.Hold_Flag = 'N' And
R.Requested_Start_Date <= Sysdate And ( R.Enforce_Seriality_Flag =
'N' OR ( C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag =
'N' OR Not Exists (Select Null From Fnd_Concurrent_Requests Sr
Where Sr.Status_Code In ('R', 'T') And Sr.Enforce_Seriality_Flag =
'Y' And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <=
Q.Max_Processes And R.Rowid = :reqname And
((P.Execution_Method_Code != 'S' OR
(R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN
((0,98),(0,100),(0,31721),(0,31722),(0,31757))) AND
((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN
((510,40112),(510,40113),(510,41497),(510,41498),(530,41859),(530,418
60),(535,41492),(535,41493),(535,41494)))) FOR UPDATE OF
R.status_code NoWait
Rationale
SQL statement with SQL_ID "a49xsqhv0h31b" was executed 4686 times and
had an average elapsed time of 1.2 seconds.
Rationale
Waiting for event "buffer busy waits" in wait class "Concurrency"
accounted for 85% of the database time spent in processing the SQL
statement with SQL_ID "a49xsqhv0h31b".
Rationale
Waiting for event "log file switch (checkpoint incomplete)" in wait
class "Configuration" accounted for 9% of the database time spent in
processing the SQL statement with SQL_ID "a49xsqhv0h31b".
Recommendation 3: SQL Tuning
Estimated benefit is .56 active sessions, 10.91% of total activity.
Action
Investigate the SQL statement with SQL_ID "5d7957yktf3nn" for possible
performance improvements.
Related Object
SQL statement with SQL_ID 5d7957yktf3nn.
UPDATE ICX_SESSIONS SET TIME_OUT = :B2 WHERE SESSION_ID = :B1
Rationale
SQL statement with SQL_ID "5d7957yktf3nn" was executed 266 times and had
an average elapsed time of 7.6 seconds.
Rationale
Waiting for event "buffer busy waits" in wait class "Concurrency"
accounted for 86% of the database time spent in processing the SQL
statement with SQL_ID "5d7957yktf3nn".
Rationale
Waiting for event "log file switch (checkpoint incomplete)" in wait
class "Configuration" accounted for 7% of the database time spent in
processing the SQL statement with SQL_ID "5d7957yktf3nn".
Finding 2: Buffer Busy
Impact is 2.52 active sessions, 48.81% of total activity.
Read and write contention on database blocks was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is 1.42 active sessions, 27.44% of total activity.
Action
Trace the cause of object contention due to SELECT statements in the
application using the information provided.
Related Object
Database object with ID 34562.
Rationale
The SELECT statement with SQL_ID "a49xsqhv0h31b" was significantly
affected by "buffer busy" waits.
Related Object
SQL statement with SQL_ID a49xsqhv0h31b.
SELECT R.Conc_Login_Id, R.Request_Id, R.Phase_Code, R.Status_Code,
P.Application_ID, P.Concurrent_Program_ID, P.Concurrent_Program_Name,
R.Enable_Trace, R.Restart, DECODE(R.Increment_Dates, 'Y', 'Y', 'N'),
R.NLS_Compliant, R.OUTPUT_FILE_TYPE, E.Executable_Name,
E.Execution_File_Name, A2.Basepath, DECODE(R.Stale, 'Y', 'C',
P.Execution_Method_Code), P.Print_Flag, P.Execution_Options,
DECODE(P.Srs_Flag, 'Y', 'Y', 'Q', 'Y', 'N'), P.Argument_Method_Code,
R.Print_Style, R.Argument_Input_Method_Code, R.Queue_Method_Code,
R.Responsibility_ID, R.Responsibility_Application_ID, R.Requested_By,
R.Number_Of_Copies, R.Save_Output_Flag, R.Printer, R.Print_Group,
R.Priority, U.User_Name, O.Oracle_Username,
O.Encrypted_Oracle_Password, R.Cd_Id, A.Basepath,
A.Application_Short_Name, TO_CHAR(R.Requested_Start_Date,'YYYY/MM/DD
HH24:MI:SS'), R.Nls_Language, R.Nls_Territory,
R.Nls_Numeric_Characters, DECODE(R.Parent_Request_ID, NULL, 0,
R.Parent_Request_ID), R.Priority_Request_ID, R.Single_Thread_Flag,
R.Has_Sub_Request, R.Is_Sub_Request, R.Req_Information,
R.Description, R.Resubmit_Time, TO_CHAR(R.Resubmit_Interval),
R.Resubmit_Interval_Type_Code, R.Resubmit_Interval_Unit_Code,
TO_CHAR(R.Resubmit_End_Date,'YYYY/MM/DD HH24:MI:SS'),
Decode(E.Execution_File_Name, NULL, 'N', Decode(E.Subroutine_Name,
NULL, Decode(E.Execution_Method_Code, 'I', 'Y', 'J', 'Y', 'N'),
'Y')), R.Argument1, R.Argument2, R.Argument3, R.Argument4,
R.Argument5, R.Argument6, R.Argument7, R.Argument8, R.Argument9,
R.Argument10, R.Argument11, R.Argument12, R.Argument13, R.Argument14,
R.Argument15, R.Argument16, R.Argument17, R.Argument18, R.Argument19,
R.Argument20, R.Argument21, R.Argument22, R.Argument23, R.Argument24,
R.Argument25, X.Argument26, X.Argument27, X.Argument28, X.Argument29,
X.Argument30, X.Argument31, X.Argument32, X.Argument33, X.Argument34,
X.Argument35, X.Argument36, X.Argument37, X.Argument38, X.Argument39,
X.Argument40, X.Argument41, X.Argument42, X.Argument43, X.Argument44,
X.Argument45, X.Argument46, X.Argument47, X.Argument48, X.Argument49,
X.Argument50, X.Argument51, X.Argument52, X.Argument53, X.Argument54,
X.Argument55, X.Argument56, X.Argument57, X.Argument58, X.Argument59,
X.Argument60, X.Argument61, X.Argument62, X.Argument63, X.Argument64,
X.Argument65, X.Argument66, X.Argument67, X.Argument68, X.Argument69,
X.Argument70, X.Argument71, X.Argument72, X.Argument73, X.Argument74,
X.Argument75, X.Argument76, X.Argument77, X.Argument78, X.Argument79,
X.Argument80, X.Argument81, X.Argument82, X.Argument83, X.Argument84,
X.Argument85, X.Argument86, X.Argument87, X.Argument88, X.Argument89,
X.Argument90, X.Argument91, X.Argument92, X.Argument93, X.Argument94,
X.Argument95, X.Argument96, X.Argument97, X.Argument98, X.Argument99,
X.Argument100, R.number_of_arguments, C.CD_Name,
NVL(R.Security_Group_ID, 0), NVL(R.org_id, 0) FROM
fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application
A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C,
fnd_concurrent_queues Q, fnd_application A2, fnd_executables E,
fnd_conc_request_arguments X WHERE R.Status_code = 'I' And
((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or
(R.OPS_INSTANCE =
decode(:dcp_on,1,FND_CONC_GLOBAL.OPS_INST_NUM,R.OPS_INSTANCE))) And
R.Request_ID = X.Request_ID(+) And R.Program_Application_Id =
P.Application_Id(+) And R.Concurrent_Program_Id =
P.Concurrent_Program_Id(+) And R.Program_Application_Id =
A.Application_Id(+) And P.Executable_Application_Id =
E.Application_Id(+) And P.Executable_Id =
E.Executable_Id(+) And P.Executable_Application_Id =
A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id
= C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id =
:q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag
is NULL OR P.Enabled_Flag = 'Y') And R.Hold_Flag = 'N' And
R.Requested_Start_Date <= Sysdate And ( R.Enforce_Seriality_Flag =
'N' OR ( C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag =
'N' OR Not Exists (Select Null From Fnd_Concurrent_Requests Sr
Where Sr.Status_Code In ('R', 'T') And Sr.Enforce_Seriality_Flag =
'Y' And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <=
Q.Max_Processes And R.Rowid = :reqname And
((P.Execution_Method_Code != 'S' OR
(R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN
((0,98),(0,100),(0,31721),(0,31722),(0,31757))) AND
((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN
((510,40112),(510,40113),(510,41497),(510,41498),(530,41859),(530,418
60),(535,41492),(535,41493),(535,41494)))) FOR UPDATE OF
R.status_code NoWait
UPDATE ICX_SESSIONS SET LAST_CONNECT = SYSDATE WHERE SESSION_ID = :B1
Recommendation 1: Schema Changes
Estimated benefit is .03 active sessions, .62% of total activity.
Action
Consider rebuilding the TABLE "APPLSYS.FND_LOGIN_RESP_FORMS" with object
ID 34651 using a higher value for PCTFREE.
Related Object
Database object with ID 34651.
Rationale
The UPDATE statement with SQL_ID "cqc5crhxxt36t" was significantly
affected by "buffer busy" waits.
Related Object
SQL statement with SQL_ID cqc5crhxxt36t.
UPDATE FND_LOGIN_RESP_FORMS FLRF SET END_TIME = SYSDATE WHERE
FLRF.LOGIN_ID = :B2 AND FLRF.LOGIN_RESP_ID = :B1 AND FLRF.END_TIME IS
NULL AND (FLRF.FORM_ID, FLRF.FORM_APPL_ID) = (SELECT F.FORM_ID,
F.APPLICATION_ID FROM FND_FORM F, FND_APPLICATION A WHERE F.FORM_NAME
= :B4 AND F.APPLICATION_ID = A.APPLICATION_ID AND
A.APPLICATION_SHORT_NAME = :B3 )
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is 2.53 active sessions, 48.87% of total activity.
Finding 4: Log File Switches
Impact is .91 active sessions, 17.56% of total activity.
Log file switch operations were consuming significant database time while
waiting for checkpoint completion.
This problem can be caused by use of hot backup mode on tablespaces. DML to
tablespaces in hot backup mode causes generation of additional redo.
Recommendation 1: Database Configuration
Estimated benefit is .91 active sessions, 17.56% of total activity.
Action
Verify whether incremental shipping was used for standby databases.
Symptoms That Led to the Finding:
Wait class "Configuration" was consuming significant database time.
Impact is .91 active sessions, 17.63% of total activity.
Finding 5: Buffer Busy
Impact is .56 active sessions, 10.87% of total activity.
A hot data block with concurrent read and write activity was found. The block
belongs to segment "ICX.ICX_SESSIONS" and is block 243489 in file 36.
Recommendation 1: Application Analysis
Estimated benefit is .56 active sessions, 10.87% of total activity.
Action
Investigate application logic to find the cause of high concurrent read
and write activity to the data present in this block.
Related Object
Database block with object number 37562, file number 36 and block
number 243489.
Rationale
The SQL statement with SQL_ID "5d7957yktf3nn" spent significant time on
"buffer busy" waits for the hot block.
Related Object
SQL statement with SQL_ID 5d7957yktf3nn.
UPDATE ICX_SESSIONS SET TIME_OUT = :B2 WHERE SESSION_ID = :B1
Rationale
The SQL statement with SQL_ID "326up1aym56dd" spent significant time on
"buffer busy" waits for the hot block.
Related Object
SQL statement with SQL_ID 326up1aym56dd.
UPDATE ICX_SESSIONS SET LAST_CONNECT = SYSDATE WHERE SESSION_ID = :B1
Recommendation 2: Schema Changes
Estimated benefit is .56 active sessions, 10.87% of total activity.
Action
Consider rebuilding the TABLE "ICX.ICX_SESSIONS" with object ID 37562
using a higher value for PCTFREE.
Related Object
Database object with ID 37562.
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is 2.53 active sessions, 48.87% of total activity.
Finding 6: Undersized SGA
Impact is .38 active sessions, 7.37% of total activity.
The SGA was inadequately sized, causing additional I/O or hard parses.
The value of parameter "sga_target" was "4096 M" during the analysis period.
Recommendation 1: Database Configuration
Estimated benefit is .12 active sessions, 2.33% of total activity.
Action
Increase the size of the SGA by setting the parameter "sga_target" to
4608 M.
Symptoms That Led to the Finding:
Wait class "User I/O" was consuming significant database time.
Impact is .7 active sessions, 13.57% of total activity.
Hard parsing of SQL statements was consuming significant database time.
Impact is .13 active sessions, 2.51% of total activity.
Contention for latches related to the shared pool was consuming
significant database time.
Impact is 0 active sessions, .03% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is 2.53 active sessions, 48.87% of total activity.
Finding 7: Commits and Rollbacks
Impact is .28 active sessions, 5.42% of total activity.
Waits on event "log file sync" while performing COMMIT and ROLLBACK operations
were consuming significant database time.
Recommendation 1: Host Configuration
Estimated benefit is .28 active sessions, 5.42% of total activity.
Action
Investigate the possibility of improving the performance of I/O to the
online redo log files.
Rationale
The average size of writes to the online redo log files was 163 K and
the average time per write was 68 milliseconds.
Symptoms That Led to the Finding:
Wait class "Commit" was consuming significant database time.
Impact is .28 active sessions, 5.42% of total activity.
Finding 8: Undo I/O
Impact is .18 active sessions, 3.53% of total activity.
Undo I/O was a significant portion (26%) of the total database I/O.
No recommendations are available.
Symptoms That Led to the Finding:
The throughput of the I/O subsystem was significantly lower than
expected.
Impact is .08 active sessions, 1.46% of total activity.
Wait class "User I/O" was consuming significant database time.
Impact is .7 active sessions, 13.57% of total activity.
Finding 9: CPU Usage
Impact is .13 active sessions, 2.57% of total activity.
Time spent on the CPU by the instance was responsible for a substantial part
of database time.
Recommendation 1: SQL Tuning
Estimated benefit is .13 active sessions, 2.57% of total activity.
Finding 10: Top SQL By I/O
Impact is .11 active sessions, 2.21% of total activity.
Individual SQL statements responsible for significant user I/O wait were
found.
Recommendation 1: SQL Tuning
Estimated benefit is .11 active sessions, 2.22% of total activity.
Action
Run SQL Tuning Advisor on the SQL statement with SQL_ID "b3pnc5yctv2z5".
Related Object
SQL statement with SQL_ID b3pnc5yctv2z5.
INSERT INTO ZX_TRANSACTION_LINES_GT( APPLICATION_ID ,ENTITY_CODE
,EVENT_CLASS_CODE ,TRX_ID ,TRX_LEVEL_TYPE ,TRX_LINE_ID ,LINE_CLASS
,LINE_LEVEL_ACTION ,TRX_LINE_TYPE ,TRX_LINE_DATE
,LINE_AMT_INCLUDES_TAX_FLAG ,LINE_AMT ,TRX_LINE_QUANTITY ,UNIT_PRICE
,PRODUCT_ID ,PRODUCT_ORG_ID ,UOM_CODE ,PRODUCT_CODE ,SHIP_TO_PARTY_ID
,SHIP_FROM_PARTY_ID ,BILL_TO_PARTY_ID ,BILL_FROM_PARTY_ID
,SHIP_FROM_PARTY_SITE_ID ,BILL_FROM_PARTY_SITE_ID
,SHIP_TO_LOCATION_ID ,SHIP_FROM_LOCATION_ID ,BILL_TO_LOCATION_ID
,SHIP_THIRD_PTY_ACCT_ID ,SHIP_THIRD_PTY_ACCT_SITE_ID ,HISTORICAL_FLAG
,TRX_LINE_CURRENCY_CODE ,TRX_LINE_CURRENCY_CONV_DATE
,TRX_LINE_CURRENCY_CONV_RATE ,TRX_LINE_CURRENCY_CONV_TYPE
,TRX_LINE_MAU ,TRX_LINE_PRECISION ,HISTORICAL_TAX_CODE_ID
,TRX_BUSINESS_CATEGORY ,PRODUCT_CATEGORY ,PRODUCT_FISC_CLASSIFICATION
,LINE_INTENDED_USE ,PRODUCT_TYPE ,USER_DEFINED_FISC_CLASS
,ASSESSABLE_VALUE ,INPUT_TAX_CLASSIFICATION_CODE ,ACCOUNT_CCID
,BILL_THIRD_PTY_ACCT_ID ,BILL_THIRD_PTY_ACCT_SITE_ID ,TRX_LINE_NUMBER
,TRX_LINE_DESCRIPTION ,PRODUCT_DESCRIPTION ,USER_UPD_DET_FACTORS_FLAG
,DEFAULTING_ATTRIBUTE1 ) SELECT :B4 ,:B3 ,:B2
,PRL.REQUISITION_HEADER_ID ,:B1 ,PRL.REQUISITION_LINE_ID ,'INVOICE'
,NVL(PRL.TAX_ATTRIBUTE_UPDATE_CODE,'UPDATE') ,'ITEM'
,NVL(PRL.NEED_BY_DATE, SYSDATE) ,'N' ,NVL(PRL.AMOUNT,
PRL.UNIT_PRICE*PRL.QUANTITY) ,PRL.QUANTITY ,PRL.UNIT_PRICE
,PRL.ITEM_ID ,(SELECT FSP.INVENTORY_ORGANIZATION_ID FROM
FINANCIALS_SYSTEM_PARAMS_ALL FSP WHERE FSP.ORG_ID=PRL.ORG_ID)
,(SELECT MUM.UOM_CODE FROM MTL_UNITS_OF_MEASURE MUM WHERE
MUM.UNIT_OF_MEASURE=PRL.UNIT_MEAS_LOOKUP_CODE) ,MSIB.SEGMENT1
,PRL.DESTINATION_ORGANIZATION_ID ,PV.PARTY_ID ,PRH.ORG_ID
,PV.PARTY_ID ,PVS.PARTY_SITE_ID ,PVS.PARTY_SITE_ID
,PRL.DELIVER_TO_LOCATION_ID ,(SELECT HZPS.LOCATION_ID FROM
HZ_PARTY_SITES HZPS WHERE HZPS.PARTY_SITE_ID = PVS.PARTY_SITE_ID)
,(SELECT LOCATION_ID FROM HR_ALL_ORGANIZATION_UNITS WHERE
ORGANIZATION_ID=PRH.ORG_ID) ,PRL.VENDOR_ID ,PRL.VENDOR_SITE_ID ,NULL
,NVL(PRL.CURRENCY_CODE, :B9 ) ,NVL2(PRL.CURRENCY_CODE, PRL.RATE_DATE,
SYSDATE) ,NVL2(PRL.CURRENCY_CODE, PRL.RATE, :B8 )
,NVL2(PRL.CURRENCY_CODE, PRL.RATE_TYPE, :B7 )
,FC.MINIMUM_ACCOUNTABLE_UNIT ,NVL(FC.PRECISION, 2) ,NULL
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.TRX_BUSINESS_CATEGORY, NULL),
NULL ) ,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.PRODUCT_CATEGORY, NULL), NULL )
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.PRODUCT_FISC_CLASSIFICATION,
NULL), NULL ) ,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.LINE_INTENDED_USE, NULL), NULL )
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.PRODUCT_TYPE, NULL), NULL )
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.USER_DEFINED_FISC_CLASS, NULL),
NULL ) ,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.ASSESSABLE_VALUE, NULL), NULL )
,DECODE(:B6 , 'REQIMPORT', PRL.TAX_NAME,
DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.INPUT_TAX_CLASSIFICATION_CODE,
NULL), NULL ) ) ,NVL((SELECT PRD.CODE_COMBINATION_ID FROM
PO_REQ_DISTRIBUTIONS_ALL PRD WHERE PRD.REQUISITION_LINE_ID =
PRL.REQUISITION_LINE_ID AND ROWNUM = 1), MSIB.EXPENSE_ACCOUNT )
,PV.VENDOR_ID ,PVS.VENDOR_SITE_ID ,PRL.LINE_NUM ,PRL.ITEM_DESCRIPTION
,PRL.ITEM_DESCRIPTION ,(SELECT 'Y' FROM DUAL WHERE :B6 = 'REQIMPORT'
AND PRL.TAX_NAME IS NOT NULL) ,PRL.DESTINATION_ORGANIZATION_ID FROM
PO_REQUISITION_HEADERS_ALL PRH, PO_REQUISITION_LINES_ALL PRL,
ZX_LINES_DET_FACTORS ZXLDET, PO_VENDORS PV, PO_VENDOR_SITES_ALL PVS,
MTL_SYSTEM_ITEMS_B MSIB, FND_CURRENCIES FC WHERE
PRH.REQUISITION_HEADER_ID = :B5 AND PRH.REQUISITION_HEADER_ID =
PRL.REQUISITION_HEADER_ID AND ZXLDET.APPLICATION_ID(+) = :B4 AND
ZXLDET.ENTITY_CODE(+) = :B3 AND ZXLDET.EVENT_CLASS_CODE(+) = :B2 AND
ZXLDET.TRX_LEVEL_TYPE(+) = :B1 AND ZXLDET.TRX_LINE_ID(+) =
PRL.PARENT_REQ_LINE_ID AND PV.VENDOR_ID(+) = PRL.VENDOR_ID AND
PVS.VENDOR_SITE_ID(+) = PRL.VENDOR_SITE_ID AND
MSIB.INVENTORY_ITEM_ID(+) = PRL.ITEM_ID AND MSIB.ORGANIZATION_ID(+) =
PRL.ORG_ID AND FC.CURRENCY_CODE(+) = PRL.CURRENCY_CODE AND
NVL(PRL.MODIFIED_BY_AGENT_FLAG, 'N') = 'N' AND NVL(PRL.CANCEL_FLAG,
'N') = 'N' AND NVL(PRL.CLOSED_CODE, 'OPEN') <> 'FINALLY CLOSED' AND
PRL.LINE_LOCATION_ID IS NULL AND PRL.AT_SOURCING_FLAG IS NULL
Rationale
SQL statement with SQL_ID "b3pnc5yctv2z5" was executed 3 times and had
an average elapsed time of 138 seconds.
Rationale
Average time spent in User I/O wait events per execution was 137
seconds.
Symptoms That Led to the Finding:
Wait class "User I/O" was consuming significant database time.
Impact is .7 active sessions, 13.57% of total activity.
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Network" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
The database's maintenance windows were active during 100% of the analysis
period.
Regards
AthishFew days are am facing serious of performance problem in our Production instanceFor production issues, please log a SR.
Was this working before? If yes, any changes been done recently?
Do you have the statistics collected up to date?
Please see these docs.
AutoInvoice Performance Issue When Processing Tax [ID 1059275.1]
R12 : System Hangs When Attempting To Save Blanket Release After Applying Patch 11817843 [ID 1333336.1]
Thanks,
Hussein -
Performance issue with pl/sql code
Hi Oracle Gurus,
I am in need of your recommendations for a performance issue that I am facing in production envrionment. There is a pl/sql procedure which executes with different elapsed time at different executions. Elapsed Times are 30minutes , 40 minutes, 65 minutes , 3 minutes ,3 seconds.
Expected elapsed time is maximum of 3 minutes. ( But some times it took 3 seconds too...! )
Output on all different executions are same that is deletion and insertion of 12K records into a table.
Here is the auto trace details of two different scenarios.
Slow execution - 33.65 minutes
Stat Name Statement Per Execution % Snap
Elapsed Time (ms) 1,712,343 1,712,342.6 41.4
CPU Time (ms) 1,679,689 1,679,688.6 44.7
Executions 1 N/A N/A
Buffer Gets ########## 167,257,973.0 86.9
Disk Reads 1,284 1,284.0 0.4
Parse Calls 1 1.0 0.0
User I/O Wait Time (ms) 4,264 N/A N/A
Cluster Wait Time (ms) 3,468 N/A N/A
Application Wait Time (ms) 0 N/A N/A
Concurrency Wait Time (ms) 6 N/A N/A
Invalidations 0 N/A N/A
Version Count 4 N/A N/A
Sharable Mem(KB) 85 N/A N/A
-------------------------------------------------------------Fast Exection : 5 seconds
Stat Name Statement Per Execution % Snap
Elapsed Time (ms) 41,550 41,550.3 0.7
CPU Time (ms) 40,776 40,776.3 1.0
Executions 1 N/A N/A
Buffer Gets 2,995,677 2,995,677.0 4.2
Disk Reads 22 22.0 0.0
Parse Calls 1 1.0 0.0
User I/O Wait Time (ms) 162 N/A N/A
Cluster Wait Time (ms) 621 N/A N/A
Application Wait Time (ms) 0 N/A N/A
Concurrency Wait Time (ms) 55 N/A N/A
Invalidations 0 N/A N/A
Version Count 4 N/A N/A
Sharable Mem(KB) 85 N/A N/A
-------------------------------------------------------------For security reasons, I cannot share the actual code. Its a report generating code that deletes and load the data into table using insert into select statement.
Delete from table ;
cursor X to get the master data ( 98 records )
For each X loop
insert into tableA select * from tables where a= X.a and b= X.b and c=X.c ..... ;
-- 12 K records inserted on average
insert into tableB select * from tables where a= X.a and b= X.b and c=X.c ..... ;
-- 12 K records inserted on average
end loop ;1. The select query is complex with bind variables ( explain plan varies for each values )
2. I have checked the tablespace of the tables involved, it is 82% used. DBA confirmed that it is not the reason.
3. Disk reads are high during long execution.
4. At long running times, I can see a db sequential read wait event on a index object. This index is on the table where data is inserted.
All I need to find is why this code is taking 3 seconds and 60 minutes on the same day and on the consecutive executions ?
Is there any other approach to find the root cause of this behaviour and to fix it ? Kindly adivse.
Thanks in advance your help.
Regards,
Hari
Edited by: BluShadow on 26-Sep-2012 08:24
edited to add {noformat}{noformat} tags. You've been a member long enough to know to do this yourself... so please do so in future. ({message:id=9360002})Hariharan ST wrote:
Hi Oracle Gurus,
I am in need of your recommendations for a performance issue that I am facing in production envrionment. There is a pl/sql procedure which executes with different elapsed time at different executions. Please reedit your post and add some code tags around the trace information. This would improve readability greatly and will help us to help you
example
{<b></b>code}
select * from dual;{<b></b>code}
Based upon your description I can imagine two things.
a) The execution plan for the select query does change frequently.
A typical reason can be not up to date statistics.
b) Some locking / wait conflict. For example upon a UK index.
Are there any other operations going on while it is slow? If anybody inserts a value, then your session will wait, if the same (PK/UK) value also is to be inserted.
Those wait events can be recognized using standard tools like oracle sql developer or enterprise manager while the query is slow.
Also go through the links that are in the FAQ. They tell you how to get better information for makeing a tuning request.
SQL and PL/SQL FAQ
Edited by: Sven W. on Sep 25, 2012 6:41 PM -
Performance issue showing read by other session Event
Hi All,
we are having a severe performance issue in my database when we are running batch jobs.
This was a new database(11.2.0.2) and we are testing the performance by running some batch jobs. These batch jobs included some inserts and updates.
I am seeing read by other session in top 5 timed events and cache buffers chains in Latch Miss Sources section.
Please help me to solve this out.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5605 29-Feb-12 03:00:27 63 4.5
End Snap: 5614 29-Feb-12 12:00:47 63 4.3
Elapsed: 540.32 (mins)
DB Time: 1,774.23 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 3.3 0.8 0.02 0.05
DB CPU(s): 1.1 0.3 0.01 0.02
Redo size: 55,763.8 13,849.3
Logical reads: 23,906.6 5,937.4
Block changes: 325.7 80.9
Physical reads: 665.6 165.3
Physical writes: 40.4 10.0
User calls: 60.7 15.1
Parses: 10.6 2.6
Hard parses: 1.1 0.3
W/A MB processed: 0.6 0.2
Logons: 0.1 0.0
Executes: 151.2 37.6
Rollbacks: 0.0 0.0
Transactions: 4.0
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.94 Redo NoWait %: 100.00
Buffer Hit %: 97.90 In-memory Sort %: 100.00
Library Hit %: 98.06 Soft Parse %: 90.16
Execute to Parse %: 92.96 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 76.71 % Non-Parse CPU: 98.57
Shared Pool Statistics Begin End
Memory Usage %: 89.38 87.96
% SQL with executions>1: 97.14 95.15
% Memory for SQL w/exec>1: 96.05 92.46
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 14,092,706 65,613 5 61.6 User I/O
DB CPU 34,819 32.7
read by other session 308,534 1,260 4 1.2 User I/O
direct path read 97,454 987 10 .9 User I/O
db file scattered read 71,870 910 13 .9 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
0.43 0.36 13.7 0.6 9.7 85.7
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.5
% of busy CPU for Instance: 94.2
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 506.5 629.1
% Host Mem used for SGA+PGA: 7.28 7.53
Time Model Statistics
-> Total time in database user-calls (DB Time): 106453.8s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 105,531.1 99.1
DB CPU 34,818.8 32.7
parse time elapsed 714.7 .7
hard parse elapsed time 684.8 .6
PL/SQL execution elapsed time 161.9 .2
PL/SQL compilation elapsed time 44.2 .0
connection management call elapsed time 16.9 .0
hard parse (sharing criteria) elapsed time 10.2 .0
hard parse (bind mismatch) elapsed time 9.4 .0
sequence load elapsed time 2.9 .0
repeated bind elapsed time 0.5 .0
failed parse elapsed time 0.0 .0
DB time 106,453.8
background elapsed time 1,753.9
background cpu time 61.7
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 3,704,415
IDLE_TIME 22,203,740
IOWAIT_TIME 2,517,864
NICE_TIME 3
SYS_TIME 145,696
USER_TIME 3,557,758
LOAD 0 0
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 358,813,045,760
VM_OUT_BYTES 29,514,830,848
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 03:00:27 0.4 N/A N/A N/A N/A N/A
29-Feb 04:00:35 1.4 11.9 11.2 0.6 88.1 14.3
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
29-Feb 06:00:48 1.5 14.0 13.5 0.6 86.0 12.3
29-Feb 07:01:00 1.8 16.3 15.8 0.5 83.7 10.4
29-Feb 08:00:12 2.6 23.2 22.5 0.6 76.8 12.6
29-Feb 09:00:26 1.3 16.6 16.0 0.5 83.4 5.7
29-Feb 10:00:33 1.2 13.8 13.3 0.5 86.2 2.0
29-Feb 11:00:43 1.3 14.5 14.0 0.5 85.5 3.8
29-Feb 12:00:47 0.4 4.9 4.2 0.7 95.1 10.6
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.9% of Total DB time 106,453.79 (s)
-> Total FG Wait Time: 69,415.64 (s) DB CPU time: 34,818.79 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 14,693,843 0 69,222 5 65.0
DB CPU 34,819 32.7
Commit 40,629 0 119 3 0.1
System I/O 26,504 0 57 2 0.1
Network 1,945,010 0 11 0 0.0
Other 125,200 99 4 0 0.0
Application 2,673 0 2 1 0.0
Concurrency 3,059 0 1 0 0.0
Configuration 31 19 0 15 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 14,092,706 0 65,613 5 108.0 61.6
read by other session 308,534 0 1,260 4 2.4 1.2
direct path read 97,454 0 987 10 0.7 .9
db file scattered read 71,870 0 910 13 0.6 .9
db file parallel read 35,001 0 372 11 0.3 .3
log file sync 40,629 0 119 3 0.3 .1
control file sequential re 26,504 0 57 2 0.2 .1
direct path read temp 14,499 0 49 3 0.1 .0
direct path write temp 9,186 0 28 3 0.1 .0
SQL*Net message to client 1,923,973 0 5 0 14.7 .0
SQL*Net message from dblin 1,056 0 5 5 0.0 .0
Disk file operations I/O 8,848 0 2 0 0.1 .0
ASM file metadata operatio 36 0 2 54 0.0 .0
SQL*Net break/reset to cli 2,636 0 1 1 0.0 .0
ADR block file read 472 0 1 1 0.0 .0
os thread startup 8 0 1 74 0.0 .0
SQL*Net more data to clien 17,656 0 1 0 0.1 .0
asynch descriptor resize 123,852 100 0 0 0.9 .0
local write wait 110 0 0 4 0.0 .0
utl_file I/O 55,635 0 0 0 0.4 .0
log file switch (private s 8 0 0 52 0.0 .0
cursor: pin S wait on X 2 0 0 142 0.0 .0
enq: KO - fast object chec 13 0 0 20 0.0 .0
PX Deq: Slave Session Stat 248 0 0 1 0.0 .0
enq: RO - fast object reus 18 0 0 11 0.0 .0
latch: cache buffers chain 2,511 0 0 0 0.0 .0
latch: shared pool 195 0 0 1 0.0 .0
CSS initialization 12 0 0 8 0.0 .0
PX qref latch 54 100 0 2 0.0 .0
SQL*Net more data from cli 995 0 0 0 0.0 .0
SQL*Net more data from dbl 300 0 0 0 0.0 .0
kksfbc child completion 1 100 0 56 0.0 .0
library cache: mutex X 244 0 0 0 0.0 .0
PX Deq: Signal ACK RSG 124 0 0 0 0.0 .0
undo segment extension 6 100 0 7 0.0 .0
PX Deq: Signal ACK EXT 124 0 0 0 0.0 .0
library cache load lock 3 0 0 9 0.0 .0
ADR block file write 45 0 0 1 0.0 .0
CSS operation: action 12 0 0 2 0.0 .0
reliable message 28 0 0 1 0.0 .0
CSS operation: query 72 0 0 0 0.0 .0
latch: row cache objects 14 0 0 1 0.0 .0
enq: SQ - contention 17 0 0 0 0.0 .0
latch free 32 0 0 0 0.0 .0
buffer busy waits 52 0 0 0 0.0 .0
enq: PS - contention 16 0 0 0 0.0 .0
enq: TX - row lock content 6 0 0 1 0.0 .0
SQL*Net message to dblink 1,018 0 0 0 0.0 .0
cursor: pin S 23 0 0 0 0.0 .0
latch: cache buffers lru c 8 0 0 0 0.0 .0
SQL*Net message from clien 1,923,970 0 944,508 491 14.7
jobq slave wait 66,732 100 33,334 500 0.5
Streams AQ: waiting for me 6,481 100 32,412 5001 0.0
wait for unread message on 32,858 98 32,411 986 0.3
PX Deq: Execution Msg 1,448 0 190 131 0.0
PX Deq: Execute Reply 1,196 0 74 62 0.0
HS message to agent 228 0 4 19 0.0
single-task message 42 0 4 97 0.0
PX Deq Credit: send blkd 904 0 2 3 0.0
PX Deq Credit: need buffer 205 0 1 3 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
PX Deq: Table Q Normal 4,291 0 1 0 0.0
PX Deq: Join ACK 124 0 0 1 0.0
PX Deq: Parse Reply 124 0 0 0 0.0
KSV master wait 256 0 0 0 0.0
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
ASM map operation freeli kffmTranslate2 0 2 0
DML lock allocation ktadmc 0 2 0
FOB s.o list latch ksfd_allfob 0 2 2
In memory undo latch ktiFlushMe 0 5 0
In memory undo latch ktichg: child 0 3 0
PC and Classifier lists No latch 0 6 0
Real-time plan statistic keswxAddNewPlanEntry 0 20 20
SQL memory manager worka qesmmIRegisterWorkArea:1 0 1 1
active service list kswslogon: session logout 0 23 12
active service list kswssetsvc: PX session swi 0 6 1
active service list kswsite: service iterator 0 1 0
archive process latch kcrrgpll 0 3 3
cache buffers chains kcbgtcr_2 0 1,746 573
cache buffers chains kcbgtcr: fast path (cr pin 0 1,024 2,126
cache buffers chains kcbgcur_2 0 60 8
cache buffers chains kcbchg1: kslbegin: bufs no 0 16 3
cache buffers chains kcbgtcr: fast path 0 14 20
cache buffers chains kcbzibmlt: multi-block rea 0 10 0
cache buffers chains kcbrls_2 0 9 53
cache buffers chains kcbgtcr: kslbegin shared 0 8 1
cache buffers chains kcbrls_1 0 7 84
cache buffers chains kcbgtcr: kslbegin excl 0 6 14
cache buffers chains kcbnew: new latch again 0 6 0
cache buffers chains kcbzgb: scan from tail. no 0 6 0
cache buffers chains kcbzwb 0 5 8
cache buffers chains kcbgcur: fast path (shr) 0 3 0
cache buffers chains kcbget: pin buffer 0 3 0
cache buffers chains kcbzhngcbk2_1 0 1 0
cache buffers lru chain kcbzgws 0 19 0
cache buffers lru chain kcbo_link_q 0 3 0
call allocation ksuxds 0 14 10
call allocation ksudlp: top call 0 2 3
enqueue hash chains ksqgtl3 0 2 1
enqueue hash chains ksqrcl 0 1 2
enqueues ksqgel: create enqueue 0 1 0
object queue header oper kcbo_unlink_q 0 5 2
object queue header oper kcbo_sw_buf 0 2 0
object queue header oper kcbo_link_q 0 1 2
object queue header oper kcbo_switch_cq 0 1 2
object queue header oper kcbo_switch_mq_bg 0 1 4
parallel query alloc buf kxfpbalo 0 1 1
process allocation ksucrp:1 0 2 0
process queue reference kxfpqrsnd 0 1 0
qmn task queue latch kwqmnmvtsks: delay to read 0 1 0
redo allocation kcrfw_redo_gen: redo alloc 0 17 0
row cache objects kqreqd: reget 0 6 0
row cache objects kqrpre: find obj 0 6 13
row cache objects kqrso 0 2 0
row cache objects kqreqd 0 1 2
row cache objects kqrpre: init complete 0 1 1
shared pool kghalo 0 199 106
shared pool kghupr1 0 39 109
shared pool kghfre 0 18 19
shared pool kghalp 0 7 29
space background task la ktsj_grab_task 0 21 27
Mutex Sleep Summary
-> ordered by number of sleeps desc
Wait
Mutex Type Location Sleeps Time (ms)
Library Cache kglhdgn2 106 338 12
Library Cache kgllkc1 57 259 10
Library Cache kgllkdl1 85 123 21
Cursor Pin kkslce [KKSCHLPIN2] 70 286
Library Cache kglget2 2 31 1
Library Cache kglhdgn1 62 31 2
Library Cache kglpin1 4 26 1
Library Cache kglpnal1 90 18 0
Library Cache kglpndl1 95 15 2
Library Cache kgllldl2 112 6 0
Library Cache kglini1 32 1 0
-------------------------------------------------------------Thanks in advance.Hi,
Thanks for reply.
I provided one hour report.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5606 29-Feb-12 04:00:35 63 3.7
End Snap: 5607 29-Feb-12 05:00:41 63 3.6
Elapsed: 60.11 (mins)
DB Time: 382.67 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 6.4 0.8 0.03 0.03
DB CPU(s): 1.0 0.1 0.00 0.00
Redo size: 84,539.3 10,425.6
Logical reads: 23,345.6 2,879.1
Block changes: 386.5 47.7
Physical reads: 1,605.0 197.9
Physical writes: 7.1 0.9
User calls: 233.9 28.9
Parses: 4.0 0.5
Hard parses: 0.1 0.0
W/A MB processed: 0.1 0.0
Logons: 0.1 0.0
Executes: 210.9 26.0
Rollbacks: 0.0 0.0
Transactions: 8.1
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.62 Redo NoWait %: 100.00
Buffer Hit %: 95.57 In-memory Sort %: 100.00
Library Hit %: 99.90 Soft Parse %: 98.68
Execute to Parse %: 98.10 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 32.08 % Non-Parse CPU: 99.90
Shared Pool Statistics Begin End
Memory Usage %: 89.25 89.45
% SQL with executions>1: 96.79 97.52
% Memory for SQL w/exec>1: 95.67 96.56
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 3,054,464 17,002 6 74.0 User I/O
DB CPU 3,748 16.3
read by other session 199,603 796 4 3.5 User I/O
direct path read 46,301 439 9 1.9 User I/O
db file scattered read 21,113 269 13 1.2 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
1.45 1.67 13.2 0.6 15.8 86.2
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.0
% of busy CPU for Instance: 94.7
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 513.5 467.7
% Host Mem used for SGA+PGA: 7.29 7.20
Time Model Statistics
-> Total time in database user-calls (DB Time): 22960.5s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 22,835.9 99.5
DB CPU 3,748.4 16.3
parse time elapsed 15.4 .1
hard parse elapsed time 14.3 .1
PL/SQL execution elapsed time 7.5 .0
PL/SQL compilation elapsed time 6.0 .0
connection management call elapsed time 1.6 .0
sequence load elapsed time 0.4 .0
hard parse (sharing criteria) elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 22,960.5
background elapsed time 238.1
background cpu time 4.9
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 396,506
IDLE_TIME 2,483,725
IOWAIT_TIME 455,495
NICE_TIME 0
SYS_TIME 16,163
USER_TIME 380,052
LOAD 1 2
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 95,646,943,232
VM_OUT_BYTES 1,686,059,008
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 04:00:35 1.4 N/A N/A N/A N/A N/A
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.6% of Total DB time 22,960.46 (s)
-> Total FG Wait Time: 18,651.75 (s) DB CPU time: 3,748.35 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 3,327,253 0 18,576 6 80.9
DB CPU 3,748 16.3
Commit 23,882 0 69 3 0.3
System I/O 1,035 0 3 3 0.0
Network 842,393 0 2 0 0.0
Other 10,120 99 0 0 0.0
Configuration 3 0 0 58 0.0
Application 264 0 0 1 0.0
Concurrency 1,482 0 0 0 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 3,054,464 0 17,002 6 104.5 74.0
read by other session 199,603 0 796 4 6.8 3.5
direct path read 46,301 0 439 9 1.6 1.9
db file scattered read 21,113 0 269 13 0.7 1.2
log file sync 23,882 0 69 3 0.8 .3
db file parallel read 4,727 0 68 14 0.2 .3
control file sequential re 1,035 0 3 3 0.0 .0
SQL*Net message to client 840,792 0 2 0 28.8 .0
direct path read temp 95 0 2 18 0.0 .0
local write wait 79 0 0 4 0.0 .0
Disk file operations I/O 870 0 0 0 0.0 .0
ASM file metadata operatio 4 0 0 50 0.0 .0
log file switch (private s 3 0 0 58 0.0 .0
ADR block file read 36 0 0 3 0.0 .0
enq: RO - fast object reus 5 0 0 16 0.0 .0
latch: cache buffers chain 1,465 0 0 0 0.1 .0
SQL*Net break/reset to cli 256 0 0 0 0.0 .0
asynch descriptor resize 10,059 100 0 0 0.3 .0
SQL*Net more data to clien 1,510 0 0 0 0.1 .0
enq: KO - fast object chec 3 0 0 8 0.0 .0
SQL*Net more data from cli 91 0 0 0 0.0 .0
latch: shared pool 14 0 0 0 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
reliable message 8 0 0 0 0.0 .0
direct path write temp 1 0 0 2 0.0 .0
SQL*Net message from clien 840,794 0 68,885 82 28.8
jobq slave wait 7,365 100 3,679 499 0.3
Streams AQ: waiting for me 721 100 3,605 5000 0.0
wait for unread message on 3,648 98 3,603 988 0.1
KSV master wait 20 0 0 0 0.0
Background Wait Events
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
log file parallel write 29,353 0 83 3 1.0 34.8
db file parallel write 5,753 0 17 3 0.2 6.9
db file sequential read 1,638 0 15 9 0.1 6.1
control file sequential re 5,142 0 13 2 0.2 5.4
os thread startup 140 0 8 58 0.0 3.4
control file parallel writ 1,440 0 8 6 0.0 3.4
log file sequential read 304 0 8 26 0.0 3.3
db file scattered read 214 0 2 9 0.0 .8
ASM file metadata operatio 1,199 0 1 1 0.0 .3
direct path write 35 0 0 6 0.0 .1
direct path read 41 0 0 5 0.0 .1
kfk: async disk IO 6 0 0 9 0.0 .0
Disk file operations I/O 1,266 0 0 0 0.0 .0
ADR block file read 16 0 0 2 0.0 .0
read by other session 3 0 0 8 0.0 .0
Log archive I/O 2 0 0 10 0.0 .0
log file sync 3 0 0 5 0.0 .0
asynch descriptor resize 341 100 0 0 0.0 .0
CSS initialization 1 0 0 6 0.0 .0
log file single write 4 0 0 1 0.0 .0
latch: redo allocation 3 0 0 1 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
LGWR wait for redo copy 45 0 0 0 0.0 .0
CSS operation: query 6 0 0 0 0.0 .0
CSS operation: action 1 0 0 1 0.0 .0
SQL*Net message to client 420 0 0 0 0.0 .0
rdbms ipc message 47,816 39 61,046 1277 1.6
DIAG idle wait 7,200 100 7,200 1000 0.2
Space Manager: slave idle 1,146 98 5,674 4951 0.0
class slave wait 284 0 3,983 14026 0.0
dispatcher timer 61 100 3,660 60006 0.0
Streams AQ: qmn coordinato 258 50 3,613 14003 0.0
Streams AQ: qmn slave idle 130 0 3,613 27789 0.0
Streams AQ: waiting for ti 7 71 3,608 515430 0.0
wait for unread message on 3,605 100 3,606 1000 0.1
pmon timer 1,201 100 3,604 3001 0.0
smon timer 15 73 3,603 240207 0.0
ASM background timer 754 0 3,602 4777 0.0
shared server idle wait 120 100 3,601 30006 0.0
SQL*Net message from clien 554 0 4 7 0.0
KSV master wait 101 0 0 2 0.0
Wait Event Histogram
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 52 73.1 1.9 9.6 13.5 1.9
ADR block file write 10 100.0
ADR file lock 12 100.0
ARCH wait for archivelog l 3 100.0
ASM file metadata operatio 1203 97.3 .5 .7 .3 .2 .9
CSS initialization 1 100.0
CSS operation: action 1 100.0
CSS operation: query 6 83.3 16.7
Disk file operations I/O 2118 95.4 4.5 .1
LGWR wait for redo copy 45 100.0
Log archive I/O 2 100.0
SQL*Net break/reset to cli 256 99.6 .4
SQL*Net message to client 839.9 100.0 .0
SQL*Net more data from cli 91 100.0
SQL*Net more data to clien 1503 100.0
asynch descriptor resize 10.4K 100.0
buffer busy waits 2 100.0
control file parallel writ 1440 5.7 35.1 24.0 16.3 12.0 5.5 1.5
control file sequential re 6177 69.4 7.5 5.9 8.1 7.1 1.7 .3
db file parallel read 4727 1.7 3.2 3.2 10.1 46.6 33.3 1.8
db file parallel write 5755 42.3 21.3 18.6 11.2 4.6 1.4 .5
db file scattered read 21.5K 8.4 4.3 11.9 18.9 26.3 25.3 4.9
db file sequential read 3053. 28.7 15.1 11.1 17.9 21.5 5.4 .3 .0
direct path read 46.3K 9.9 8.8 18.5 21.7 22.8 15.7 2.7
direct path read temp 95 9.5 9.5 23.2 49.5 8.4
direct path write 35 11.4 31.4 17.1 22.9 11.4 2.9 2.9
direct path write temp 1 100.0
enq: KO - fast object chec 3 66.7 33.3
enq: RO - fast object reus 5 20.0 20.0 20.0 20.0 20.0
kfk: async disk IO 6 50.0 16.7 16.7 16.7
latch free 3 100.0
latch: cache buffers chain 1465 100.0
latch: cache buffers lru c 1 100.0
latch: object queue header 2 100.0
latch: redo allocation 3 33.3 33.3 33.3
latch: row cache objects 2 100.0
latch: shared pool 15 93.3 6.7
local write wait 79 35.4 34.2 21.5 8.9
log file parallel write 29.4K 47.8 21.7 11.9 9.9 6.8 1.6 .3
log file sequential read 304 6.3 3.0 3.6 10.2 23.4 24.3 29.3
log file single write 4 25.0 75.0
log file switch (private s 3 100.0
log file sync 23.9K 40.9 28.0 12.9 9.7 6.7 1.5 .3
os thread startup 140 100.0
read by other session 199.6 37.1 19.9 12.9 13.1 13.8 3.1 .2
reliable message 8 100.0
ASM background timer 755 2.9 .4 .1 .1 .3 .1 .3 95.8
DIAG idle wait 7196 100.0
KSV master wait 121 88.4 2.5 3.3 2.5 .8 .8 1.7
SQL*Net message from clien 840.1 97.1 1.8 .5 .2 .2 .1 .0 .1
Space Manager: slave idle 1147 .1 .5 99.4
Streams AQ: qmn coordinato 258 49.6 .4 50.0
Streams AQ: qmn slave idle 130 .8 99.2
Streams AQ: waiting for me 721 100.0
Streams AQ: waiting for ti 7 28.6 42.9 28.6
class slave wait 283 39.9 2.5 2.5 3.5 4.9 9.2 15.2 22.3
dispatcher timer 60 100.0
jobq slave wait 7360 .0 .0 .0 99.9
pmon timer 1201 100.0
rdbms ipc message 47.8K 2.7 31.6 17.4 1.1 1.1 .9 20.9 24.3
Wait Event Histogram DB/Inst: I2KPROD/I2KPROD Snaps: 5606-5607
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
shared server idle wait 120 100.0
smon timer 16 6.3 93.8
wait for unread message on 7250 .1 99.9
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
In memory undo latch ktichg: child 0 1 0
active service list kswslogon: session logout 0 2 0
cache buffers chains kcbgtcr_2 0 1,123 483
cache buffers chains kcbgtcr: fast path (cr pin 0 496 1,131
cache buffers chains kcbrls_2 0 5 6
cache buffers chains kcbgcur_2 0 4 0
cache buffers chains kcbgtcr: fast path 0 3 1
cache buffers chains kcbzwb 0 2 4
cache buffers chains kcbchg1: kslbegin: bufs no 0 1 0
cache buffers chains kcbnew: new latch again 0 1 0
cache buffers chains kcbrls_1 0 1 6
cache buffers chains kcbzgb: scan from tail. no 0 1 0
cache buffers lru chain kcbzgws 0 1 0
object queue header oper kcbo_switch_cq 0 1 0
object queue header oper kcbo_switch_mq_bg 0 1 2
redo allocation kcrfw_redo_gen: redo alloc 0 3 0
row cache objects kqrpre: find obj 0 1 1
row cache objects kqrso 0 1 0
shared pool kghalo 0 13 3
shared pool kghupr1 0 4 15
shared pool kghalp 0 1 0
space background task la ktsj_grab_task 0 2 2
------------------------------------------------------------- -
Oracle Apps Database severe Performance Issue
Hi Gurus,
This is regarding a severe performance issue running in our Production E-Business Suite Instance.
its an R12.1.3 setup installed with 11.2.0.1 Database. All the servers are Solaris Sparc 64 (Solaris 10)
Let me brief you about the instance first:
2 Node Application
- Main Application Server hosting web/forms/concurrent/admin servers
- iSupplier server hosting web services (placed in DMZ, used by external suppliers via Internet)
1 Node Database Server
Database Server Specs
Memory: 144G phys mem 20G total swap
- CPUs (8Px4cores, 2Px2cores)
- I/O - fiber channel hard disk (hitachi SAN Storage) - 7 DATA_TOPs (7 drives with RAID 5) - current DB size 1.6 TB
- at peak load, around 1000 concurrent forms session and 2000 web sessions.
We have been facing some serious performance issues and we raised an SR with Oracle Support.
The Support analyzed a bunch of AWR Reports we provided them and they asked us to increase the DB_CACHE from its current usage of 27G to 40G
So, we changed SGA_TARGET from 35G to 50G and PGA was increased from 35G to 40G as v$pgastat was also suggesting some lack of memory.
We made these changes last night.
Today morning we observed the following:
1. after start of office hours, we checked in the home page of EM DB Console that ADDM was showing reduced impact due to lack of SGA memory which seemed to be a good sign. Earlier it was around 25% which was now at 12%.
However, negative aspects were:
1. lot of swapping was reported by the System Administrators on the DB Server
2. High CPU Usage
3. EM DB Console showed a lot of "Concurrency Wait Class" events ...throughout the day lot of blocking sessions were reported which were making other sessions to wait.
in the AWR Report, following foreground reports were listed:
Top 5 Timed Foreground Events
Event
Waits
Time(s)
Avg wait (ms)
% DB time
Wait Class
DB CPU
132,577
61.46
library cache lock
3,539
40,683
11496
18.86
Concurrency
library cache: mutex X
4,014,083
21,011
5
9.74
Concurrency
db file sequential read
4,138,014
20,767
5
9.63
User I/O
latch free
381,916
5,897
15
2.73
Other
This is showing "library cache lock" events as the main culprit apart from the usual suspect, the CPU.
I am attaching the AWR Report. Please let me know if i should revert back the memory changes or is there anything else i could do.
Please help us resolving it because the performance is going worst.
Regards,
Muneer.Pl do not post duplicates - Oracle Apps Database severe Performance Issue
For all critical production issues, pl work with Support thru SRs - using the forums to troubleshoot production issues is not wise -
Oracle SOA 11g Performance Issue
Hi,
We have set up Oracle SOA Suite on AIX environment. Java which we are using is IBM Jdk 1.6. Recently we are hit with performance issue. Frequently we are getting out of memory exception and we need to restart the server and sometimes physically reboot the machine, because out of 16 GB of RAM 4GB we have given as heap space to Admin Server, 7 GB to SOA Server but it is taking more than 7 GB as heap space. On stopping or killing both the services memory is not getting released
SOA Suite Version : 11.1.1.3
Instance Node: Single Node
I collected the logs and tried to analyze in Thread Dump Analyzer and i could see objects(Reserved) is taking 100% of the CPU Utilization.
We are getting the following error highlighed in the analyzer. There are about 200+ threads got stuck.
"HTTPThreadGroup-42" prio=10 tid=0x6382ba28 nid=0x20bf4 waiting on condition [0x6904f000..0x6904fb94]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:772)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1087)
at java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"HTTPThreadGroup-41" prio=10 tid=0x6ae3cce0 nid=0x20bf0 waiting on condition [0x68d8f000..0x68d8fc14]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:772)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1087)
at java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
Has anyone faced same issue? We are badly hit with this performance issue in UAT
Please concider this on high priority and someone please help us
Regards,
SundarAlways raise a case with Oracle Support for such issues.
Regards,
Anuj -
hI,
I need to build UI using ADobe flex that will connect BPM system. I have concurrent 500 users. Is this is feasible to develop application(UI) using Flex for such kind of application.
Thanks,
VikashHello Victor,
We are using Flex at Taleo for an application that can have much more than 1,000 concurrent users. You just need to make sure that your architecture can scale horizontally and vertically. Flex isn't the best here, we also do have performance issues. If speed is your primary target, I suggests you to go pure HTML (5) and Javascript (jQuery, Dojo, etc.)
Good luck.
Nicolas Boulet-Lavoie
Développeur logiciel
Software Developer
T. 418.524.5665 x 1454
F: 418.524.8899
C. [email protected]
Ce courriel et toute pièce jointe peuvent contenir des renseignements confidentiels, particuliers ou protégés pour l'usage exclusif du destinataire du message original. Toute révision, copie ou distribution du présent courriel (ou de toute pièce jointe) par autrui est formellement interdite. Si vous n'êtes pas le destinataire prévu, veuillez retourner immédiatement le courriel à l'expéditeur et effacer définitivement l'original, les copies et les pièces jointes.
This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient named in the original email to which this message was attached. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please return this email to the sender immediately and permanently delete the original and any copies of this email and any attachments thereto. -
Heaviest performance issue impacting business Free Pct showing 9.41%
Dears,
We are facing heaviest performance issue on our database and and Application, Our Key users Oracle Techno functional consultant complaining that when "Free Pct" was 18% performance was excellent but now the "Free Pct" has reached to 9.41%.
Am wondering if any performance tuning steps can be done to come back to its 18%, your valuable help is required.
Following are the details:-
Tablespace | Size Mgs | Free Mgs |Used Mgs |**Free Pct** | Used Pct |Max Mb
APPS_TS_TX_IDX | 84780.88 | 7973.75 |76807.13 |**9.41** | 90.41 | 84780.88
APPS_TS_TX_DATA | 120540.25 | 16301.88 | 104238.38 |*13.52* | 86.48 | 149276.23
Do I have to rebuild index tablespace? If yes, while doing this what are the precautions can be done like, do I need downtime from the business?
will the command below if triggered resolves the issue:-
alter index apps. APPS_TS_TX_IDX rebuild;
Kind regards,
Mohammed
Edited by: user9007339 on 28/01/2013 04:22 ص
Edited by: user9007339 on 28/01/2013 04:52 ص
Edited by: user9007339 on 28/01/2013 05:48 صHi Helios,
I shall certainly update, here are the details and feedback of our SR:-
======================================================================================================
1, SR 3-6701198251 : Oracle Application PRODUCTION SHUTDOWN THREE TIMES
=== Data Collected ===
Findings and Recommendations
Finding 1: Commits and Rollbacks
Impact is 4.76 active sessions, 52.92% of total activity.
Waits on event "log file sync" while performing COMMIT and ROLLBACK operations
were consuming significant database time.
Recommendation 1: Host Configuration
Estimated benefit is 4.76 active sessions, 52.92% of total activity.
Action
Investigate the possibility of improving the performance of I/O to the
online redo log files.
Rationale
The average size of writes to the online redo log files was 1788 K and
the average time per write was 2766 milliseconds.
Symptoms That Led to the Finding:
Wait class "Commit" was consuming significant database time.
Impact is 4.76 active sessions, 52.92% of total activity.
Finding 2: Top SQL by DB Time
Impact is 3.86 active sessions, 42.9% of total activity.
SQL statements consuming significant database time were found.
Recommendation 1: SQL Tuning
Estimated benefit is 2.29 active sessions, 25.44% of total activity.
Action
Tune the PL/SQL block with SQL_ID "5t39uchjqpyfm". Refer to the "Tuning
PL/SQL Applications" chapter of Oracle's "PL/SQL User's Guide and
Reference".
Related Object
SQL statement with SQL_ID 5t39uchjqpyfm.
BEGIN xla_accounting_pkg.unit_processor_batch(:errbuf,:rc,:A0,:A1,:A2
,:A3,:A4,:A5,:A6,:A7,:A8,:A9,:A10,:A11,:A12,:A13,:A14); END;
log_buffer 10485760
log_checkpoint_interval 100000
log_checkpoint_timeout 1200
log_checkpoints_to_alert TRUE
=== Action Plan ===
Mohammed,
AWR and ADDM reports clearly point to performance issues around redo logs. Top waits were :
op 5 Timed Foreground Events
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
log file sync 1,517 13,265 8744 45.55 Commit
log buffer space 9,048 8,218 908 28.22 Configuration
buffer busy waits 6,519 3,743 574 12.85 Concurrency
DB CPU 2,177 7.48
db file sequential read 54,769 540 10 1.85 User I/O
You have two options here :
1> Increase size of log_buffer . Set to 15M , unset other parameters ( log_checkpoint_interval , log_checkpoint_timeout )
2> Increase size and number of online redo log files. Make sure that these are on fast disks.
3> Run redo generating jobs , like XLAACCUP, during off hours where end-users are not in the system.
======================================================================================================
Dear Helios,
We are waiting for the RAM to come from our vendor to increase from 16GB to 32GB on Application Server it will take 3 weeks from now.
Secondly, Our key users especially "Oracle Techno Functional Consultants and "Oracle Application Developers" are compalining about the APPS_TS_TX_IDX tablespace tat has been decreased by 50% from 18% to 9.41%................I was attempting to find if there any possibiities to tune this tablespace..........Your suggestion is appreciated.
Regards,
Mohammed
Edited by: user9007339 on 29/01/2013 03:02 ص -
Dear experts:
I got a problem with ABAP memory performance issue. I need to export an internal table to ABAP memory, and it's ID is sales order number. So, I will bulid too many ABAP memory in SAP...
Does it will couse performance issue or will it be lost?
tyroluHi Tyrolu,
There are some settings in basis which decides size of your abap memory.
If you are consuming most of the abap memory by exporting heavy objects to abap memory then
you may face a problem as many standard sap program aslo uses abap memory and
if there is no free abap memory then they will start thowing short dumps like "Page allocation error"
or "Could not allocate shared memory" or something like that, I don't remember exact term now.
In some cases you won't have any problem if users are less, but as the no of concurrent users
running your program increased then every user who runs the program would consume abap memory
and if there is no free abap memory then this may again lead to above mentioned problem.
Also ensure FREE MEMORY ID is used which clears abap memory immediatly.
Other wise abap memory is cleared by the garbage collector on random basis and by that time
your memory remains consumed and not available for other programs to use
which may again lead to above mentioned problem.
Regards,
Vishal -
Hi
I am seeing the performance issue on my ebs environment. My ebs database has 3 processor and 9GB RAM. When i run Item/Item Revision User-Defined Attributes Concurrent Program, my database server show 98% of cpu utlilisation and there is some paging also happening in database server.
Is there any bug in "Item/Item Revision User-Defined Attributes" Concurrent Program? or do i need to looking for increasing resources on my server?Hi,
Can you find any errors in the CM/Database log files?
AFAIK, there are no performance issues reported about this concurrent program. As a start, you may enable trace and generate the TKPROF file, and see if it helps. I would also you go through the following thread.
EBS, performence issue
EBS, performence issue
For the hardware resources, I believe it is sufficient and you should not have any issues that could be relevant.
If none of the above helps, please log a SR as the issue is with a standard concurrent program.
Regards,
Hussein -
Performance issues after upgrading to 11g
hello,
We are facing a performance issue after upgrading from 10g to 11g.
as you can see, the tables named APPLSYS.FND_ENV_CONTEXT has grown to 15544880 records,
and oracle is using TABLE ACCESS FULL on it:
DELETE FND_ENV_CONTEXT E
WHERE NOT EXISTS
(SELECT 'X'
FROM FND_CONCURRENT_PROCESSES P
WHERE P.CONCURRENT_PROCESS_ID = E.CONCURRENT_PROCESS_ID)
AND ROWNUM < 10000;
Plan:
DELETE STATEMENT ALL_ROWSCost: 74,345
5 DELETE APPLSYS.FND_ENV_CONTEXT
4 COUNT STOPKEY
3 HASH JOIN RIGHT ANTI Cost: 74,345 Bytes: 3,108,980 Cardinality: 155,449
1 INDEX FULL SCAN INDEX (UNIQUE) APPLSYS.FND_CONCURRENT_PROCESSES_U1 Cost: 166 Bytes: 315,450 Cardinality: 63,090
2 TABLE ACCESS FULL TABLE APPLSYS.FND_ENV_CONTEXT Cost: 69,124 Bytes: 233,173,200 Cardinality: 15,544,880
select num_rows,last_analyzed from dba_tables where table_name='FND_ENV_CONTEXT'
15544880 07/07/2011 12:08:55
please advice.
OferPlease see these docs.
Table Fnd_env_context Growing Very Fast [ID 419990.1]
FND Related Tablespaces Growing at Rapid and Excessive Rate [ID 189800.1]
FNDCPPUR Request Does Not Always Delete Files From The File System, Many Files Will Be Denoted As "deletion failed" [ID 1335304.1]
Concurrent Processing - Best Practices for Performance for Concurrent Managers in E-Business Suite [ID 1057802.1]
Concurrent Processing - Purge Concurrent Request and/or Manager Data Program (FNDCPPUR) [ID 104282.1]
Thanks,
Hussein
Maybe you are looking for
-
Prompt valuese are in row, obiee 11g
Hi , How to show prompt values are in single row. Ex: We have a year prompt , usually when we create a prompt. Prompt Name: 2010 2011 2013 I have to show that radio button 2010 radiobutton 2011 radionbutton2013. Can you please share your thoughts if
-
More No of GRN with single excise invoice
Dear Sir we want to do goods receipt for 4 lots with single excise invoice.we have done multible goods receipt with multi credit settings in J1id for that perticular material.we tried to do goods receipt with reference to out bound delivery NO.first
-
Limitations to a Lookup Property Table
I am relatively new to Hyperion DRM. I have a question regarding any limitations to a lookup property table. How many members or rows can a look up table contain. I have a need to cross reference our sku numbers to a platform. The first thing that ca
-
I've exported a sample calendar with .ics and looked at it in Text Edit. I've then modified that event DTSTART;....20060416T120000, SUMMARY:Custom Event From Text Edit, etc... I save the file as an .ics and iCal re-imports it fine. I've further taken
-
I did the download on the Safari 4 update 4.0.3 and now Safari won't even open. It will only open about half way then go no further. If I do a force quit it tells me Safari (not responding) so that I have to make it stop or it simply keeps spinning t