Performance issue in DB need help with analysing this ADDM report

Question

Hi,
My environment:
Os: RHEL5U3 / 11.1.0.7 64 bit / R12.1.1 64 bit
Issue:
Few days are am facing serious of performance problem in our Production instance. Normally the issue will occur 5 to 10 minutes occasionally per day. At the time of issue we not able to access the EBS application its taking time to load. But backend all the oracle, listener and apps services are up and running. No locks at table and session level. Cpu and memory usage is normal.
We have monitored using "Enterprise Manager" for this issue and we found the wait session present more in Active session tab. At this time EBS application is not able access its loading too time. After some time the in Active session tab the wait session came normal and when we try to access the EBS application its working fine.
We try to find the cause of the issue by running addm report. But am not able to understand what its says. Kindly suggests me
ADDM Report for Task 'TASK_42656'
Analysis Period
AWR snapshot range from 14754 to 14755.
Time period starts at 17-APR-12 11.00.22 AM
Time period ends at 17-APR-12 12.00.33 PM
Analysis Target
Database 'PRD' with DB ID 1789440879.
Database version 11.1.0.7.0.
ADDM performed an analysis of instance PRD, numbered 1 and hosted at
advgrpdb.advgroup.ae.
Activity During the Analysis Period
Total database time was 18674 seconds.
The average number of active sessions was 5.17.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 Top SQL by DB Time 3.43 | 66.33 5
2 Buffer Busy 2.52 | 48.81 5
3 Buffer Busy 1.39 | 26.81 2
4 Log File Switches .91 | 17.56 1
5 Buffer Busy .56 | 10.87 2
6 Undersized SGA .38 | 7.37 1
7 Commits and Rollbacks .28 | 5.42 1
8 Undo I/O .18 | 3.53 0
9 CPU Usage .13 | 2.57 1
10 Top SQL By I/O .11 | 2.21 1
Findings and Recommendations
Finding 1: Top SQL by DB Time
Impact is 3.43 active sessions, 66.33% of total activity.
SQL statements consuming significant database time were found.
Recommendation 1: SQL Tuning
Estimated benefit is 1.59 active sessions, 30.8% of total activity.
Action
Investigate the SQL statement with SQL_ID "a49xsqhv0h31b" for possible
performance improvements.
Related Object
SQL statement with SQL_ID a49xsqhv0h31b.
SELECT R.Conc_Login_Id, R.Request_Id, R.Phase_Code, R.Status_Code,
P.Application_ID, P.Concurrent_Program_ID, P.Concurrent_Program_Name,
R.Enable_Trace, R.Restart, DECODE(R.Increment_Dates, 'Y', 'Y', 'N'),
R.NLS_Compliant, R.OUTPUT_FILE_TYPE, E.Executable_Name,
E.Execution_File_Name, A2.Basepath, DECODE(R.Stale, 'Y', 'C',
P.Execution_Method_Code), P.Print_Flag, P.Execution_Options,
DECODE(P.Srs_Flag, 'Y', 'Y', 'Q', 'Y', 'N'), P.Argument_Method_Code,
R.Print_Style, R.Argument_Input_Method_Code, R.Queue_Method_Code,
R.Responsibility_ID, R.Responsibility_Application_ID, R.Requested_By,
R.Number_Of_Copies, R.Save_Output_Flag, R.Printer, R.Print_Group,
R.Priority, U.User_Name, O.Oracle_Username,
O.Encrypted_Oracle_Password, R.Cd_Id, A.Basepath,
A.Application_Short_Name, TO_CHAR(R.Requested_Start_Date,'YYYY/MM/DD
HH24:MI:SS'), R.Nls_Language, R.Nls_Territory,
R.Nls_Numeric_Characters, DECODE(R.Parent_Request_ID, NULL, 0,
R.Parent_Request_ID), R.Priority_Request_ID, R.Single_Thread_Flag,
R.Has_Sub_Request, R.Is_Sub_Request, R.Req_Information,
R.Description, R.Resubmit_Time, TO_CHAR(R.Resubmit_Interval),
R.Resubmit_Interval_Type_Code, R.Resubmit_Interval_Unit_Code,
TO_CHAR(R.Resubmit_End_Date,'YYYY/MM/DD HH24:MI:SS'),
Decode(E.Execution_File_Name, NULL, 'N', Decode(E.Subroutine_Name,
NULL, Decode(E.Execution_Method_Code, 'I', 'Y', 'J', 'Y', 'N'),
'Y')), R.Argument1, R.Argument2, R.Argument3, R.Argument4,
R.Argument5, R.Argument6, R.Argument7, R.Argument8, R.Argument9,
R.Argument10, R.Argument11, R.Argument12, R.Argument13, R.Argument14,
R.Argument15, R.Argument16, R.Argument17, R.Argument18, R.Argument19,
R.Argument20, R.Argument21, R.Argument22, R.Argument23, R.Argument24,
R.Argument25, X.Argument26, X.Argument27, X.Argument28, X.Argument29,
X.Argument30, X.Argument31, X.Argument32, X.Argument33, X.Argument34,
X.Argument35, X.Argument36, X.Argument37, X.Argument38, X.Argument39,
X.Argument40, X.Argument41, X.Argument42, X.Argument43, X.Argument44,
X.Argument45, X.Argument46, X.Argument47, X.Argument48, X.Argument49,
X.Argument50, X.Argument51, X.Argument52, X.Argument53, X.Argument54,
X.Argument55, X.Argument56, X.Argument57, X.Argument58, X.Argument59,
X.Argument60, X.Argument61, X.Argument62, X.Argument63, X.Argument64,
X.Argument65, X.Argument66, X.Argument67, X.Argument68, X.Argument69,
X.Argument70, X.Argument71, X.Argument72, X.Argument73, X.Argument74,
X.Argument75, X.Argument76, X.Argument77, X.Argument78, X.Argument79,
X.Argument80, X.Argument81, X.Argument82, X.Argument83, X.Argument84,
X.Argument85, X.Argument86, X.Argument87, X.Argument88, X.Argument89,
X.Argument90, X.Argument91, X.Argument92, X.Argument93, X.Argument94,
X.Argument95, X.Argument96, X.Argument97, X.Argument98, X.Argument99,
X.Argument100, R.number_of_arguments, C.CD_Name,
NVL(R.Security_Group_ID, 0), NVL(R.org_id, 0) FROM
fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application
A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C,
fnd_concurrent_queues Q, fnd_application A2, fnd_executables E,
fnd_conc_request_arguments X WHERE R.Status_code = 'I' And
((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or
(R.OPS_INSTANCE =
decode(:dcp_on,1,FND_CONC_GLOBAL.OPS_INST_NUM,R.OPS_INSTANCE))) And
R.Request_ID = X.Request_ID(+) And R.Program_Application_Id =
P.Application_Id(+) And R.Concurrent_Program_Id =
P.Concurrent_Program_Id(+) And R.Program_Application_Id =
A.Application_Id(+) And P.Executable_Application_Id =
E.Application_Id(+) And P.Executable_Id =
E.Executable_Id(+) And P.Executable_Application_Id =
A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id
= C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id =
:q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag
is NULL OR P.Enabled_Flag = 'Y') And R.Hold_Flag = 'N' And
R.Requested_Start_Date <= Sysdate And ( R.Enforce_Seriality_Flag =
'N' OR ( C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag =
'N' OR Not Exists (Select Null From Fnd_Concurrent_Requests Sr
Where Sr.Status_Code In ('R', 'T') And Sr.Enforce_Seriality_Flag =
'Y' And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <=
Q.Max_Processes And R.Rowid = :reqname And
((P.Execution_Method_Code != 'S' OR
(R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN
((0,98),(0,100),(0,31721),(0,31722),(0,31757))) AND
((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN
((510,40112),(510,40113),(510,41497),(510,41498),(530,41859),(530,418
60),(535,41492),(535,41493),(535,41494)))) FOR UPDATE OF
R.status_code NoWait
Rationale
SQL statement with SQL_ID "a49xsqhv0h31b" was executed 4686 times and
had an average elapsed time of 1.2 seconds.
Rationale
Waiting for event "buffer busy waits" in wait class "Concurrency"
accounted for 85% of the database time spent in processing the SQL
statement with SQL_ID "a49xsqhv0h31b".
Rationale
Waiting for event "log file switch (checkpoint incomplete)" in wait
class "Configuration" accounted for 9% of the database time spent in
processing the SQL statement with SQL_ID "a49xsqhv0h31b".
Recommendation 3: SQL Tuning
Estimated benefit is .56 active sessions, 10.91% of total activity.
Action
Investigate the SQL statement with SQL_ID "5d7957yktf3nn" for possible
performance improvements.
Related Object
SQL statement with SQL_ID 5d7957yktf3nn.
UPDATE ICX_SESSIONS SET TIME_OUT = :B2 WHERE SESSION_ID = :B1
Rationale
SQL statement with SQL_ID "5d7957yktf3nn" was executed 266 times and had
an average elapsed time of 7.6 seconds.
Rationale
Waiting for event "buffer busy waits" in wait class "Concurrency"
accounted for 86% of the database time spent in processing the SQL
statement with SQL_ID "5d7957yktf3nn".
Rationale
Waiting for event "log file switch (checkpoint incomplete)" in wait
class "Configuration" accounted for 7% of the database time spent in
processing the SQL statement with SQL_ID "5d7957yktf3nn".
Finding 2: Buffer Busy
Impact is 2.52 active sessions, 48.81% of total activity.
Read and write contention on database blocks was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is 1.42 active sessions, 27.44% of total activity.
Action
Trace the cause of object contention due to SELECT statements in the
application using the information provided.
Related Object
Database object with ID 34562.
Rationale
The SELECT statement with SQL_ID "a49xsqhv0h31b" was significantly
affected by "buffer busy" waits.
Related Object
SQL statement with SQL_ID a49xsqhv0h31b.
SELECT R.Conc_Login_Id, R.Request_Id, R.Phase_Code, R.Status_Code,
P.Application_ID, P.Concurrent_Program_ID, P.Concurrent_Program_Name,
R.Enable_Trace, R.Restart, DECODE(R.Increment_Dates, 'Y', 'Y', 'N'),
R.NLS_Compliant, R.OUTPUT_FILE_TYPE, E.Executable_Name,
E.Execution_File_Name, A2.Basepath, DECODE(R.Stale, 'Y', 'C',
P.Execution_Method_Code), P.Print_Flag, P.Execution_Options,
DECODE(P.Srs_Flag, 'Y', 'Y', 'Q', 'Y', 'N'), P.Argument_Method_Code,
R.Print_Style, R.Argument_Input_Method_Code, R.Queue_Method_Code,
R.Responsibility_ID, R.Responsibility_Application_ID, R.Requested_By,
R.Number_Of_Copies, R.Save_Output_Flag, R.Printer, R.Print_Group,
R.Priority, U.User_Name, O.Oracle_Username,
O.Encrypted_Oracle_Password, R.Cd_Id, A.Basepath,
A.Application_Short_Name, TO_CHAR(R.Requested_Start_Date,'YYYY/MM/DD
HH24:MI:SS'), R.Nls_Language, R.Nls_Territory,
R.Nls_Numeric_Characters, DECODE(R.Parent_Request_ID, NULL, 0,
R.Parent_Request_ID), R.Priority_Request_ID, R.Single_Thread_Flag,
R.Has_Sub_Request, R.Is_Sub_Request, R.Req_Information,
R.Description, R.Resubmit_Time, TO_CHAR(R.Resubmit_Interval),
R.Resubmit_Interval_Type_Code, R.Resubmit_Interval_Unit_Code,
TO_CHAR(R.Resubmit_End_Date,'YYYY/MM/DD HH24:MI:SS'),
Decode(E.Execution_File_Name, NULL, 'N', Decode(E.Subroutine_Name,
NULL, Decode(E.Execution_Method_Code, 'I', 'Y', 'J', 'Y', 'N'),
'Y')), R.Argument1, R.Argument2, R.Argument3, R.Argument4,
R.Argument5, R.Argument6, R.Argument7, R.Argument8, R.Argument9,
R.Argument10, R.Argument11, R.Argument12, R.Argument13, R.Argument14,
R.Argument15, R.Argument16, R.Argument17, R.Argument18, R.Argument19,
R.Argument20, R.Argument21, R.Argument22, R.Argument23, R.Argument24,
R.Argument25, X.Argument26, X.Argument27, X.Argument28, X.Argument29,
X.Argument30, X.Argument31, X.Argument32, X.Argument33, X.Argument34,
X.Argument35, X.Argument36, X.Argument37, X.Argument38, X.Argument39,
X.Argument40, X.Argument41, X.Argument42, X.Argument43, X.Argument44,
X.Argument45, X.Argument46, X.Argument47, X.Argument48, X.Argument49,
X.Argument50, X.Argument51, X.Argument52, X.Argument53, X.Argument54,
X.Argument55, X.Argument56, X.Argument57, X.Argument58, X.Argument59,
X.Argument60, X.Argument61, X.Argument62, X.Argument63, X.Argument64,
X.Argument65, X.Argument66, X.Argument67, X.Argument68, X.Argument69,
X.Argument70, X.Argument71, X.Argument72, X.Argument73, X.Argument74,
X.Argument75, X.Argument76, X.Argument77, X.Argument78, X.Argument79,
X.Argument80, X.Argument81, X.Argument82, X.Argument83, X.Argument84,
X.Argument85, X.Argument86, X.Argument87, X.Argument88, X.Argument89,
X.Argument90, X.Argument91, X.Argument92, X.Argument93, X.Argument94,
X.Argument95, X.Argument96, X.Argument97, X.Argument98, X.Argument99,
X.Argument100, R.number_of_arguments, C.CD_Name,
NVL(R.Security_Group_ID, 0), NVL(R.org_id, 0) FROM
fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application
A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C,
fnd_concurrent_queues Q, fnd_application A2, fnd_executables E,
fnd_conc_request_arguments X WHERE R.Status_code = 'I' And
((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or
(R.OPS_INSTANCE =
decode(:dcp_on,1,FND_CONC_GLOBAL.OPS_INST_NUM,R.OPS_INSTANCE))) And
R.Request_ID = X.Request_ID(+) And R.Program_Application_Id =
P.Application_Id(+) And R.Concurrent_Program_Id =
P.Concurrent_Program_Id(+) And R.Program_Application_Id =
A.Application_Id(+) And P.Executable_Application_Id =
E.Application_Id(+) And P.Executable_Id =
E.Executable_Id(+) And P.Executable_Application_Id =
A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id
= C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id =
:q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag
is NULL OR P.Enabled_Flag = 'Y') And R.Hold_Flag = 'N' And
R.Requested_Start_Date <= Sysdate And ( R.Enforce_Seriality_Flag =
'N' OR ( C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag =
'N' OR Not Exists (Select Null From Fnd_Concurrent_Requests Sr
Where Sr.Status_Code In ('R', 'T') And Sr.Enforce_Seriality_Flag =
'Y' And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <=
Q.Max_Processes And R.Rowid = :reqname And
((P.Execution_Method_Code != 'S' OR
(R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN
((0,98),(0,100),(0,31721),(0,31722),(0,31757))) AND
((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN
((510,40112),(510,40113),(510,41497),(510,41498),(530,41859),(530,418
60),(535,41492),(535,41493),(535,41494)))) FOR UPDATE OF
R.status_code NoWait
UPDATE ICX_SESSIONS SET LAST_CONNECT = SYSDATE WHERE SESSION_ID = :B1
Recommendation 1: Schema Changes
Estimated benefit is .03 active sessions, .62% of total activity.
Action
Consider rebuilding the TABLE "APPLSYS.FND_LOGIN_RESP_FORMS" with object
ID 34651 using a higher value for PCTFREE.
Related Object
Database object with ID 34651.
Rationale
The UPDATE statement with SQL_ID "cqc5crhxxt36t" was significantly
affected by "buffer busy" waits.
Related Object
SQL statement with SQL_ID cqc5crhxxt36t.
UPDATE FND_LOGIN_RESP_FORMS FLRF SET END_TIME = SYSDATE WHERE
FLRF.LOGIN_ID = :B2 AND FLRF.LOGIN_RESP_ID = :B1 AND FLRF.END_TIME IS
NULL AND (FLRF.FORM_ID, FLRF.FORM_APPL_ID) = (SELECT F.FORM_ID,
F.APPLICATION_ID FROM FND_FORM F, FND_APPLICATION A WHERE F.FORM_NAME
= :B4 AND F.APPLICATION_ID = A.APPLICATION_ID AND
A.APPLICATION_SHORT_NAME = :B3 )
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is 2.53 active sessions, 48.87% of total activity.
Finding 4: Log File Switches
Impact is .91 active sessions, 17.56% of total activity.
Log file switch operations were consuming significant database time while
waiting for checkpoint completion.
This problem can be caused by use of hot backup mode on tablespaces. DML to
tablespaces in hot backup mode causes generation of additional redo.
Recommendation 1: Database Configuration
Estimated benefit is .91 active sessions, 17.56% of total activity.
Action
Verify whether incremental shipping was used for standby databases.
Symptoms That Led to the Finding:
Wait class "Configuration" was consuming significant database time.
Impact is .91 active sessions, 17.63% of total activity.
Finding 5: Buffer Busy
Impact is .56 active sessions, 10.87% of total activity.
A hot data block with concurrent read and write activity was found. The block
belongs to segment "ICX.ICX_SESSIONS" and is block 243489 in file 36.
Recommendation 1: Application Analysis
Estimated benefit is .56 active sessions, 10.87% of total activity.
Action
Investigate application logic to find the cause of high concurrent read
and write activity to the data present in this block.
Related Object
Database block with object number 37562, file number 36 and block
number 243489.
Rationale
The SQL statement with SQL_ID "5d7957yktf3nn" spent significant time on
"buffer busy" waits for the hot block.
Related Object
SQL statement with SQL_ID 5d7957yktf3nn.
UPDATE ICX_SESSIONS SET TIME_OUT = :B2 WHERE SESSION_ID = :B1
Rationale
The SQL statement with SQL_ID "326up1aym56dd" spent significant time on
"buffer busy" waits for the hot block.
Related Object
SQL statement with SQL_ID 326up1aym56dd.
UPDATE ICX_SESSIONS SET LAST_CONNECT = SYSDATE WHERE SESSION_ID = :B1
Recommendation 2: Schema Changes
Estimated benefit is .56 active sessions, 10.87% of total activity.
Action
Consider rebuilding the TABLE "ICX.ICX_SESSIONS" with object ID 37562
using a higher value for PCTFREE.
Related Object
Database object with ID 37562.
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is 2.53 active sessions, 48.87% of total activity.
Finding 6: Undersized SGA
Impact is .38 active sessions, 7.37% of total activity.
The SGA was inadequately sized, causing additional I/O or hard parses.
The value of parameter "sga_target" was "4096 M" during the analysis period.
Recommendation 1: Database Configuration
Estimated benefit is .12 active sessions, 2.33% of total activity.
Action
Increase the size of the SGA by setting the parameter "sga_target" to
4608 M.
Symptoms That Led to the Finding:
Wait class "User I/O" was consuming significant database time.
Impact is .7 active sessions, 13.57% of total activity.
Hard parsing of SQL statements was consuming significant database time.
Impact is .13 active sessions, 2.51% of total activity.
Contention for latches related to the shared pool was consuming
significant database time.
Impact is 0 active sessions, .03% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is 2.53 active sessions, 48.87% of total activity.
Finding 7: Commits and Rollbacks
Impact is .28 active sessions, 5.42% of total activity.
Waits on event "log file sync" while performing COMMIT and ROLLBACK operations
were consuming significant database time.
Recommendation 1: Host Configuration
Estimated benefit is .28 active sessions, 5.42% of total activity.
Action
Investigate the possibility of improving the performance of I/O to the
online redo log files.
Rationale
The average size of writes to the online redo log files was 163 K and
the average time per write was 68 milliseconds.
Symptoms That Led to the Finding:
Wait class "Commit" was consuming significant database time.
Impact is .28 active sessions, 5.42% of total activity.
Finding 8: Undo I/O
Impact is .18 active sessions, 3.53% of total activity.
Undo I/O was a significant portion (26%) of the total database I/O.
No recommendations are available.
Symptoms That Led to the Finding:
The throughput of the I/O subsystem was significantly lower than
expected.
Impact is .08 active sessions, 1.46% of total activity.
Wait class "User I/O" was consuming significant database time.
Impact is .7 active sessions, 13.57% of total activity.
Finding 9: CPU Usage
Impact is .13 active sessions, 2.57% of total activity.
Time spent on the CPU by the instance was responsible for a substantial part
of database time.
Recommendation 1: SQL Tuning
Estimated benefit is .13 active sessions, 2.57% of total activity.
Finding 10: Top SQL By I/O
Impact is .11 active sessions, 2.21% of total activity.
Individual SQL statements responsible for significant user I/O wait were
found.
Recommendation 1: SQL Tuning
Estimated benefit is .11 active sessions, 2.22% of total activity.
Action
Run SQL Tuning Advisor on the SQL statement with SQL_ID "b3pnc5yctv2z5".
Related Object
SQL statement with SQL_ID b3pnc5yctv2z5.
INSERT INTO ZX_TRANSACTION_LINES_GT( APPLICATION_ID ,ENTITY_CODE
,EVENT_CLASS_CODE ,TRX_ID ,TRX_LEVEL_TYPE ,TRX_LINE_ID ,LINE_CLASS
,LINE_LEVEL_ACTION ,TRX_LINE_TYPE ,TRX_LINE_DATE
,LINE_AMT_INCLUDES_TAX_FLAG ,LINE_AMT ,TRX_LINE_QUANTITY ,UNIT_PRICE
,PRODUCT_ID ,PRODUCT_ORG_ID ,UOM_CODE ,PRODUCT_CODE ,SHIP_TO_PARTY_ID
,SHIP_FROM_PARTY_ID ,BILL_TO_PARTY_ID ,BILL_FROM_PARTY_ID
,SHIP_FROM_PARTY_SITE_ID ,BILL_FROM_PARTY_SITE_ID
,SHIP_TO_LOCATION_ID ,SHIP_FROM_LOCATION_ID ,BILL_TO_LOCATION_ID
,SHIP_THIRD_PTY_ACCT_ID ,SHIP_THIRD_PTY_ACCT_SITE_ID ,HISTORICAL_FLAG
,TRX_LINE_CURRENCY_CODE ,TRX_LINE_CURRENCY_CONV_DATE
,TRX_LINE_CURRENCY_CONV_RATE ,TRX_LINE_CURRENCY_CONV_TYPE
,TRX_LINE_MAU ,TRX_LINE_PRECISION ,HISTORICAL_TAX_CODE_ID
,TRX_BUSINESS_CATEGORY ,PRODUCT_CATEGORY ,PRODUCT_FISC_CLASSIFICATION
,LINE_INTENDED_USE ,PRODUCT_TYPE ,USER_DEFINED_FISC_CLASS
,ASSESSABLE_VALUE ,INPUT_TAX_CLASSIFICATION_CODE ,ACCOUNT_CCID
,BILL_THIRD_PTY_ACCT_ID ,BILL_THIRD_PTY_ACCT_SITE_ID ,TRX_LINE_NUMBER
,TRX_LINE_DESCRIPTION ,PRODUCT_DESCRIPTION ,USER_UPD_DET_FACTORS_FLAG
,DEFAULTING_ATTRIBUTE1 ) SELECT :B4 ,:B3 ,:B2
,PRL.REQUISITION_HEADER_ID ,:B1 ,PRL.REQUISITION_LINE_ID ,'INVOICE'
,NVL(PRL.TAX_ATTRIBUTE_UPDATE_CODE,'UPDATE') ,'ITEM'
,NVL(PRL.NEED_BY_DATE, SYSDATE) ,'N' ,NVL(PRL.AMOUNT,
PRL.UNIT_PRICE*PRL.QUANTITY) ,PRL.QUANTITY ,PRL.UNIT_PRICE
,PRL.ITEM_ID ,(SELECT FSP.INVENTORY_ORGANIZATION_ID FROM
FINANCIALS_SYSTEM_PARAMS_ALL FSP WHERE FSP.ORG_ID=PRL.ORG_ID)
,(SELECT MUM.UOM_CODE FROM MTL_UNITS_OF_MEASURE MUM WHERE
MUM.UNIT_OF_MEASURE=PRL.UNIT_MEAS_LOOKUP_CODE) ,MSIB.SEGMENT1
,PRL.DESTINATION_ORGANIZATION_ID ,PV.PARTY_ID ,PRH.ORG_ID
,PV.PARTY_ID ,PVS.PARTY_SITE_ID ,PVS.PARTY_SITE_ID
,PRL.DELIVER_TO_LOCATION_ID ,(SELECT HZPS.LOCATION_ID FROM
HZ_PARTY_SITES HZPS WHERE HZPS.PARTY_SITE_ID = PVS.PARTY_SITE_ID)
,(SELECT LOCATION_ID FROM HR_ALL_ORGANIZATION_UNITS WHERE
ORGANIZATION_ID=PRH.ORG_ID) ,PRL.VENDOR_ID ,PRL.VENDOR_SITE_ID ,NULL
,NVL(PRL.CURRENCY_CODE, :B9 ) ,NVL2(PRL.CURRENCY_CODE, PRL.RATE_DATE,
SYSDATE) ,NVL2(PRL.CURRENCY_CODE, PRL.RATE, :B8 )
,NVL2(PRL.CURRENCY_CODE, PRL.RATE_TYPE, :B7 )
,FC.MINIMUM_ACCOUNTABLE_UNIT ,NVL(FC.PRECISION, 2) ,NULL
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.TRX_BUSINESS_CATEGORY, NULL),
NULL ) ,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.PRODUCT_CATEGORY, NULL), NULL )
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.PRODUCT_FISC_CLASSIFICATION,
NULL), NULL ) ,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.LINE_INTENDED_USE, NULL), NULL )
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.PRODUCT_TYPE, NULL), NULL )
,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.USER_DEFINED_FISC_CLASS, NULL),
NULL ) ,DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.ASSESSABLE_VALUE, NULL), NULL )
,DECODE(:B6 , 'REQIMPORT', PRL.TAX_NAME,
DECODE(PRL.TAX_ATTRIBUTE_UPDATE_CODE, 'CREATE',
NVL2(PRL.PARENT_REQ_LINE_ID, ZXLDET.INPUT_TAX_CLASSIFICATION_CODE,
NULL), NULL ) ) ,NVL((SELECT PRD.CODE_COMBINATION_ID FROM
PO_REQ_DISTRIBUTIONS_ALL PRD WHERE PRD.REQUISITION_LINE_ID =
PRL.REQUISITION_LINE_ID AND ROWNUM = 1), MSIB.EXPENSE_ACCOUNT )
,PV.VENDOR_ID ,PVS.VENDOR_SITE_ID ,PRL.LINE_NUM ,PRL.ITEM_DESCRIPTION
,PRL.ITEM_DESCRIPTION ,(SELECT 'Y' FROM DUAL WHERE :B6 = 'REQIMPORT'
AND PRL.TAX_NAME IS NOT NULL) ,PRL.DESTINATION_ORGANIZATION_ID FROM
PO_REQUISITION_HEADERS_ALL PRH, PO_REQUISITION_LINES_ALL PRL,
ZX_LINES_DET_FACTORS ZXLDET, PO_VENDORS PV, PO_VENDOR_SITES_ALL PVS,
MTL_SYSTEM_ITEMS_B MSIB, FND_CURRENCIES FC WHERE
PRH.REQUISITION_HEADER_ID = :B5 AND PRH.REQUISITION_HEADER_ID =
PRL.REQUISITION_HEADER_ID AND ZXLDET.APPLICATION_ID(+) = :B4 AND
ZXLDET.ENTITY_CODE(+) = :B3 AND ZXLDET.EVENT_CLASS_CODE(+) = :B2 AND
ZXLDET.TRX_LEVEL_TYPE(+) = :B1 AND ZXLDET.TRX_LINE_ID(+) =
PRL.PARENT_REQ_LINE_ID AND PV.VENDOR_ID(+) = PRL.VENDOR_ID AND
PVS.VENDOR_SITE_ID(+) = PRL.VENDOR_SITE_ID AND
MSIB.INVENTORY_ITEM_ID(+) = PRL.ITEM_ID AND MSIB.ORGANIZATION_ID(+) =
PRL.ORG_ID AND FC.CURRENCY_CODE(+) = PRL.CURRENCY_CODE AND
NVL(PRL.MODIFIED_BY_AGENT_FLAG, 'N') = 'N' AND NVL(PRL.CANCEL_FLAG,
'N') = 'N' AND NVL(PRL.CLOSED_CODE, 'OPEN') <> 'FINALLY CLOSED' AND
PRL.LINE_LOCATION_ID IS NULL AND PRL.AT_SOURCING_FLAG IS NULL
Rationale
SQL statement with SQL_ID "b3pnc5yctv2z5" was executed 3 times and had
an average elapsed time of 138 seconds.
Rationale
Average time spent in User I/O wait events per execution was 137
seconds.
Symptoms That Led to the Finding:
Wait class "User I/O" was consuming significant database time.
Impact is .7 active sessions, 13.57% of total activity.
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Network" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
The database's maintenance windows were active during 100% of the analysis
period.
Regards
Athish

Answer 1

Few days are am facing serious of performance problem in our Production instanceFor production issues, please log a SR.
Was this working before? If yes, any changes been done recently?
Do you have the statistics collected up to date?
Please see these docs.
AutoInvoice Performance Issue When Processing Tax [ID 1059275.1]
R12 : System Hangs When Attempting To Save Blanket Release After Applying Patch 11817843 [ID 1333336.1]
Thanks,
Hussein