ORA-1555 ORA-3136 errors:: elapsed time vs Query Duration

Question

Dear all,
- My Database version is 11.2.0.2, Solaris.
- We have been having a problem in the production database where the front end nodes start going up and down for couple of hours sometimes. ; When node flapping is going on we get connection timed out alerts.
WARNING: inbound connection timed out (ORA-3136) opiodr aborting
process unknown ospid (4342) as a result of ORA-609 opiodr aborting
process unknown ospid (4532) as a result of ORA-609 opiodr aborting
process unknown ospid (4534) as a result of ORA-609 opiodr aborting....
Since this week node flapping is happening every day. Since past 2 days after or during node flapping we are getting ORA-1555 error.
Extract from alert log error:
ORA-01555 caused by SQL statement below (SQL ID: g8804k5pkmtyt, Query Duration=19443 sec, SCN: 0x0001.07bd90ed):
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.parentDeviceId, d.created, d.lastModified AS devLastMod, d.customerId, d.userKey1, d.userKey2, d.userKey4, d
.userKey5, d.firmwareFamily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.userKey6, d.provisioningId, d.status, d.classification, d.population, d.name, d.ipRe
solver, d.ipExpirationTime, d.geoLocationId,contact.firstContactTime, ifaces.id, ifaces.type AS ifaceType, ifaces.lastModified AS ifaceLastMod, ifaces.timeoutname, ifac
es.username1, ifaces.password1, ifaces.username2, ifaces.password2, ifaces.connReqUrl, ifaces.connReqScheme, ifaces.srvNonce, ifaces.deviceNonce, ifaces.phoneNumber,ifa
ces.bootstrapSecMethod, ifaces.srvAuthentication, ifaces.deviceAuthentication, ifaces.userPIN, ifaces.networkID, ifaces.omaSessionID, ifaces.portNum, ifaces.mgtIp, ifac
es.cmtsIp, ifaces.mgtReadCommunity, ifaces.mgtWriteCommunity, ifaces.cmtsReadCommunity, ifaces.cmtsWriteCommunity, devto.name AS devtoName, devto.rebootTimeout, devto.sessionInitiationI run Statspack report from the whole day duration, and looking into the elapsed time in seconds no more than 3739.61 sec (too lower than run duration in the alert log file of 19443 sec); So I would like to know if there is any co-relations between the ORA-3136 errors and the ORA-1555 errors?
   CPU                  CPU per             Elapsd                     Old
  Time (s)   Executions  Exec (s)  %Total   Time (s)    Buffer Gets  Hash Value
tTime <= :3 ) AND (endTime IS NULL OR endTime >= :4 )
   2773.77    7,787,914       0.00    3.4    3739.61     112,671,645 1909376826
Module: JDBC Thin Client
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.pare
ntDeviceId, d.created, d.lastModified AS devLastMod, d.customerI
d, d.userKey1, d.userKey2, d.userKey4, d.userKey5, d.firmwareFam
ily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.user
SQL> show parameter UNDO_MANAGEMENT
NAME                                 TYPE        VALUE
undo_management                      string      AUTO
SQL> show parameter UNDO_RETENTION
NAME                                 TYPE        VALUE
undo_retention                       integer     10800BR,
Diego

Answer 1

Thank you. Please let me know if it is enough or you need more information;
SQL ordered by Gets  DB/Inst: DB01/db01  Snaps: 14835-14846
-> End Buffer Gets Threshold:    100000 Total Buffer Gets:     677,689,568
-> Captured SQL accounts for   73.6% of Total Buffer Gets
-> SQL reported below exceeded  1.0% of Total Buffer Gets
                                                     CPU      Elapsd     Old
  Buffer Gets    Executions  Gets per Exec  %Total Time (s)  Time (s) Hash Value
     21,286,248    2,632,793            8.1    3.4   666.73    666.76 3610154549
Module: JDBC Thin Client
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.pare
ntDeviceId, d.created, d.lastModified AS devLastMod, d.customerI
d, d.userKey1, d.userKey2, d.userKey4, d.userKey5, d.firmwareFam
ily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.user
     17,029,561    1,176,849           14.5    2.7   417.32    416.73 1909376826
Module: JDBC Thin Client
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.pare
ntDeviceId, d.created, d.lastModified AS devLastMod, d.customerI
d, d.userKey1, d.userKey2, d.userKey4, d.userKey5, d.firmwareFam
ily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.user
     17,006,795           37      459,643.1    2.7   367.61    368.95 4045552861
Module: JDBC Thin Client
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.pare
ntDeviceId, d.created, d.lastModified AS devLastMod, d.customerI
d, d.userKey1, d.userKey2, d.userKey4, d.userKey5, d.firmwareFam
ily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.userAnother Statspack report for the whole day shows;
SQL ordered by CPU  DB/Inst: DB01/db01  Snaps: 14822-14847
-> Total DB CPU (s):          82,134
-> Captured SQL accounts for   40.9% of Total DB CPU
-> SQL reported below exceeded  1.0% of Total DB CPU
    CPU                  CPU per             Elapsd                     Old
  Time (s)   Executions  Exec (s)  %Total   Time (s)    Buffer Gets  Hash Value
tTime <= :3 ) AND (endTime IS NULL OR endTime >= :4 )
   2773.77    7,787,914       0.00    3.4    3739.61     112,671,645 1909376826
Module: JDBC Thin Client
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.pare
ntDeviceId, d.created, d.lastModified AS devLastMod, d.customerI
d, d.userKey1, d.userKey2, d.userKey4, d.userKey5, d.firmwareFam
ily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.user
SQL ordered by Gets  DB/Inst: DB01/db01  Snaps: 14822-14847
-> End Buffer Gets Threshold:    100000 Total Buffer Gets:   1,416,456,340
-> Captured SQL accounts for   55.8% of Total Buffer Gets
-> SQL reported below exceeded  1.0% of Total Buffer Gets
                                                     CPU      Elapsd     Old
  Buffer Gets    Executions  Gets per Exec  %Total Time (s)  Time (s) Hash Value
     86,354,963    7,834,326           11.0    6.3  2557.34   2604.08  906944860
Module: JDBC Thin Client
SELECT d.devId, d.vendor, d.model, d.productClass, d.oui, d.pare
ntDeviceId, d.created, d.lastModified AS devLastMod, d.customerI
d, d.userKey1, d.userKey2, d.userKey4, d.userKey5, d.firmwareFam
ily, d.softwareVer, d.serialNum, d.ip, d.mac, d.userKey3, d.user
.....BR,
Diego
Edited by: 899660 on 27-ene-2012 7:43
Edited by: 899660 on 27-ene-2012 7:45