High load, database hangs

It happened that load became very high. There were 7 processes usally taken a second, but that time taken hours of CPU time and unable to complete. After I killed those processes, situation came back to normal. I didn't find any error in alert file. What is possible explanation of such event and how to prevent it in the future?
Thanks,
Stan.

Hi,
this is some things you can look at:
- Does Oracle wait for a free redolog ?
Then the redologs are to small or few.
This can appear as Oracle "hangs".
- Take a look in the views v$system_event, v$session_event and v$session_wait.
- Does your some of tour 7 processes hold locks that prevent each other ?
Good luck
/Uffe

Similar Messages

Weblogic Server hanging at high load!!!!!!!!!!!

Folks,
We have a weblogic server 9.2 which is running fine but after quiet a while at high load we observe that the server goes down and following exception is displayed in the server logs:
####<Nov 13, 2009 2:42:48 AM EST> <Error> <WebLogicServer> <ibmrxcptpnp-wb1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1258098168215> <BEA-000337> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "617" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@1cec1cec", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
     java.net.SocketOutputStream.socketWrite0(Native Method)
     java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105)
     java.net.SocketOutputStream.write(SocketOutputStream.java:149)
     oracle.net.ns.DataPacket.send(Unknown Source)
     oracle.net.ns.NetOutputStream.flush(Unknown Source)
     oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
     oracle.net.ns.NetInputStream.read(Unknown Source)
     oracle.net.ns.NetInputStream.read(Unknown Source)
     oracle.net.ns.NetInputStream.read(Unknown Source)
     oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
     oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
     oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
     oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
     oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
     oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1060)
     oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839)
     oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132)
     oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
     oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3422)
     weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:718)
     weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:460)
     weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1455)
     weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1372)
     weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1362)
     weblogic.common.resourcepool.ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:1767)
     weblogic.common.resourcepool.ResourcePoolImpl.access$700(ResourcePoolImpl.java:37)
     weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:1935)
     weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
     weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
     weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
     weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
Any answers?????

Hi,
I have a similar issue on ODSI 10gR3 (WLS 10.3). I am having stuck threads - but, in my case it is clear that the database is down (not at the startup)
Here is some more info:
1. Test Connections On Reserve - Enabled
2. Default values for other parameters
Using Oracle thin driver for the connection pool "MyPool" -
./wlserver_10.3/server/lib/ojdbc6.jar
My question is:
1. How do I avoid the stuck thread caused when the database is down? During peak load, the stuck threads are causing some service requests to fail.
I did my homework and the closest option I could find is set the statementTimeout (which is set currently to the default value of -1 i.e. never timeout). Oracle documentation (http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/111070_readme.html) says that the ojdbc6.jar supports setQueryTimeout() method.
Is there any better way to attack this problem?
<Dec 21, 2009 12:11:18 AM EST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "MyPool" failed with exception: "java.sql.SQLRecoverableException: Io exception: Read failed: Connection timed out".>
<Dec 21, 2009 12:14:27 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:18:38 AM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@297ac2b", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-52 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: javax.management.remote.rmi.RMIConnector$RMINotifClient@271498f[fat lock]
java.lang.Object.wait(Object.java:485)
com.sun.jmx.remote.internal.ClientNotifForwarder.postReconnection(ClientNotifForwarder.java:304)
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.reconnectNotificationListeners(RMIConnector.java:1488)
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.doStart(RMIConnector.java:1568)
com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:72)
com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1420)
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:857)
weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.getAttribute(ManagedMBeanServerConnection.java:288)
javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:235)
weblogic.management.jmx.MBeanServerInvocationHandler.doInvoke(MBeanServerInvocationHandler.java:477)
weblogic.management.jmx.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:294)
$Proxy69.getServerRuntime(Unknown Source)
weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.lookupServerRuntime(DomainRuntimeServiceMBeanImpl.java:242)
sun.reflect.GeneratedMethodAccessor1880.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:355)
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:255)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:441)
weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:55)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:441)
weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:437)
weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10$1.run(AuthenticatedSubjectInterceptor.java:582)
weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10.run(AuthenticatedSubjectInterceptor.java:576)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor.invoke(AuthenticatedSubjectInterceptor.java:570)
weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:305)
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1378)
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1338)
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:761)
javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:136)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:211)
javax.management.remote.rmi.RMIConnectionImpl_1030_WLStub.invoke(Unknown Source)
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:969)
com.bea.diagnostics.server.MBeanServerUtil.processRecordsFromArchive(MBeanServerUtil.java:177)
com.bea.diagnostics.server.MetricEarliestTimestampLocator$RecordProcessorDesc.identifyEarliestTimestamps(MetricEarliestTimestampLocator.java:261)
com.bea.diagnostics.server.MetricEarliestTimestampLocator.identifyEarliestTimestamps(MetricEarliestTimestampLocator.java:84)
com.bea.diagnostics.server.MetricEarliestTimestampLocator.timerExpired(MetricEarliestTimestampLocator.java:79)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:253)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
<Dec 21, 2009 12:24:04 AM EST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "MyPool" failed with exception: "java.sql.SQLRecoverableException: Io exception: Connection reset".>
<Dec 21, 2009 12:24:04 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:24:04 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:24:05 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:24:07 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:24:12 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:24:17 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
<Dec 21, 2009 12:24:22 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>

Replacing / Compiling Database Triggers in a HA/High Load system

Hi there,
My collegue has just asked me if downtime needs to be scheduled to replace an After IUD Database trigger where I've made a minor change.
If this were a package or procedure, the answer would be an obvious yes, as I'd be wary of my users getting the dreaded: ORA-04068... State of Package has been disguarded.
What about when working with database triggers?
If the trigger is firing at the same time that I perform my Create or Replace Trigger DDL, will everything fall into a screaming heap?
Notes:
Oracle 10g R2 db.
High DML rates on the table that the After IUD Trigger is 'attached' to.
It's a near certainty that when the Create or Replace Trigger DDL runs, the trigger will be active at the time.
The Trigger is 'simple' in that it's use: is to call a DB package with some of the :new values as params.
+(I'm happy to rtfm - if someone can tell me where!!)+
Cheers -
Ron Marks

If this were a package or procedure, the answer would be an obvious yes, as I'd be wary of my users getting the dreaded: ORA-04068... State of Package has been disguarded.Not true. This exception is thrown only if sessions have some state represented in package variables. If there is no associated session state, then no exception would be thrown after recompile.
If the trigger is firing at the same time that I perform my Create or Replace Trigger DDL, will everything fall into a screaming heap?You'll wait - to lock an object in library cache which represents your trigger. Not sure if this would require a library cache lock on underlying table, but I would guess that it is required. BTW, locking a table would make no any sense: CREATE TRIGGER is a DDL, which means before it began it issues a COMMIT => all locks are released, and it is quite possible that a TM lock for CREATE TRIGGER won't be acquired since resource would be busy.
Oracle 11gR2 claims to address problems of application upgrades on a high-load environments with a feature called "Editions" (not a good name - try to find something about it via Google...)

Requirements for High-Load OLTP Database

Hi guys!
Need your Best Practise!
I will install&configure High-Load OLTP Database.
5 million users
500 transactions per second What requirements is need?
Do you have any papers or documents?

Denis :) wrote:
Hi guys!
Need your Best Practise!
I will install&configure High-Load OLTP Database.
5 million users
concurrent users?
500 transactions per second
1* select 5000*60*60/5000000 from dual
SQL> /
5000*60*60/5000000
3.6each user does about 4 transactions per hour
How big is a single transaction?
How much redo is generated every day?
>
What requirements is need? more hardware is more better!
Do you have any papers or documents?

Shutting down database hangs

Dear Gurus,
I m facing a database hang issue while performing the scheduled offline backup.I m able to take the online backup.When the backup starts , the database starts to shutdown.When checked in the log of backup it shows in tail as shutting down database.When checked for user trace , it shows message as "Attempting to kill pid xxxxxxx" and keeps on retrying for n no. of times.Then I have to manually kill the backup and restart the application.Then again try for an offline backup.This time it happens smoothly.This activity we did for couple of weeks .But now from past 3 days even after restarting the backup, it goes to hang mode.In other words unable to perform the backup activity.
The message had been raised to SAP from almost 15 days at very high.They every time ask the logs , and say that the data in log is nt providing much info for analysing the issue or cause of hang.Also enabled the traces of 10400 for trace level 1 ,10 some more traces also.But still the output is not helping even them to analyse the root cause.
Its my observation that this issue is occuring only on ABAP+JAVA systems(CRM &BIW)not in ECC.We are on oracle database 10.2.0.0.4 with latest patch set as suggested by SAP for the same issue.
This issue is h
Please suggest me at the earliest.As need to resolve this at the earliest.
Bye.
Amit

Hi,
No "archive process shutdown avoided" message is there in the log file.
In usertrace i get messages as mentioned below.
bslcrmprddb #more bcp_ora_1257502.trc
Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_1257502.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/BCP/102_64
System name:    AIX
Node name:      bslcrmprddb
Release:        3
Version:        5
Machine:        00C6C4024C00
Instance name: BCP
Redo thread mounted by this instance: 1
Oracle process number: 0
Unix process pid: 1257502, image: oracle@bslcrmprddb
2009-06-05 08:01:36.875
Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown
Failed to acquire instance startup/shutdown serialization primitive
bslcrmprddb #more bcp_ora_2453676.trc
Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_2453676.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/BCP/102_64
System name:    AIX
Node name:      bslcrmprddb
Release:        3
Version:        5
Machine:        00C6C4024C00
Instance name: BCP
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
Unix process pid: 2453676, image: oracle@bslcrmprddb
2009-06-06 07:04:22.494
Dynamic strand is set to TRUE
Running with 2 shared and 88 private strand(s). Zero-copy redo is FALSE
bslcrmprddb #more bcp_ora_1495266.trc
Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_1495266.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/BCP/102_64
System name:    AIX
Node name:      bslcrmprddb
Release:        3
Version:        5
Machine:        00C6C4024C00
Instance name: BCP
Redo thread mounted by this instance: 1
Oracle process number: 213
Unix process pid: 1495266, image: oracle@bslcrmprddb (TNS V1-V3)
ACTION NAME:() 2009-06-06 03:00:22.223
MODULE NAME:(sqlplus@bslcrmprddb (TNS V1-V3)) 2009-06-06 03:00:22.223
SERVICE NAME:(SYS$USERS) 2009-06-06 03:00:22.223
SESSION ID:(517.13229) 2009-06-06 03:00:22.223
ksukia: Starting kill, force = 0
ksukia: killed 274 out of 274 processes.
ksukia: Starting kill, force = 0
ksukia: killed 0 out of 2 processes.
ksukia: Starting kill, force = 0
ksukia: killed 0 out of 1 processes.
ksukia: Starting kill, force = 0
ksukia: killed 0 out of 1 processes.
ksukia: Starting kill, force = 0
Please suggest.
Regards,
Amit

Oracle 10g Database Hanging twice a day

My database server has 10g with MS 2003 server as OS.
Twice a day the database hangs and does disconnects all the users from local and remote locations.
Current SYSTEM tablespace has a size of 470 MB and is 98.86 in use, when I try to add another datafile as 'SYSTEM02.DBF' in the same path where other system data files are, I kept this operation running during weekends but datafile was not added.
Has it anything to do with the MS 2003 server no. of users.
Any other suggestions for SYSTEM Tablespace to be free.
Please reply.
Thanks
Ajaz Ahmed

Hi friends / sameer,
Sorry for the delay. Here is the attached alert.log file Since the data from foxpro was loaded into oracle on to oracle 10 g from 7th may the problem of database hanging started from the next day. I joined the organization on 22nd july.
Current SYSTEM tablespace has a size of 470 MB and is 98.86 in use, when I try to add another datafile as 'SYSTEM02.DBF' in the same path where other system data file is, it does not add. I kept this operation running during weekends but datafile was not added.
Has it anything to do with the MS 2003 server no. of users.
Any other suggestions for SYSTEM Tablespace to be free.
If you can get me your mail id then I can send the
attachment of alert.log file as [email protected] bounces back.
Thanks
Ajaz Ahmed

Strange Oracle Behaviour - only month end or high load periods

Hi SAP Gurus
In one of our systems I am expreriencing a strange behaviour from Oracle, in that it seems to be in reconnect, but yet it is not, it resets and carries on but do produce Dump and SM21 log messages, the dump is not really helpfull as it is the same dump but with different program everytime.
I have looked in oracle alert log and found no event tying back to the SM21 log message or dump.
The following is what I found in the developer trace of one of these insident, but there are many, about 125 - 200 for the day during high load:
Fri Nov 28 10:05:55 2008
C *** ERROR => Failed to register application info: orc=3113
[dboci.c      3042]
C *** ERROR => Error 3114 in stmt_fetch()
[dbsloci.c    13001]
C *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
[dbsloci.c    13002]
C sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB
C SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
C NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
C "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
C GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
C INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
C ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
C TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
C DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
C PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
C SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
C FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
C sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB
C prep=0,lit=0,nsql=0,lobret=0,#exec=1,dbcnt=0,upsh_p=00000000,ocistmth_p=345E20EC
C IN : cols=4,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=17,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=3470A8D8
C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C OUT: cols=105,rmax=58,xcnt=58,rpc=1,rowi=0,rtot=1,upto=-1,rsize=895,vmax=128,bound=1,iobuf_p=0E0A4EC8,vda_p=34F8F7E8
C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
C NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
C "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
C GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
C INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
C ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
C TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
C DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
C PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
C SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
C FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
B ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
B ***LOG BY4=> sql error 3114   performing FET on table CATSDB     [dbtran#10 @ 7292] [dbtran 7292 ]
B ***LOG BY0=> ORA-03114: not connected to ORACLE [dbtran#10 @ 7292] [dbtran 7292 ]
B dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
B   RSLT: {dbsl=99, tran=1}
B   FHDR: {tab='CATSDB', fcode=225, mode=2, bpb=0, dbcnt=0, crsr=1,
B          hold=0, keep=1, xfer=0, pkg=0, upto=0, init:b=0,
B          init:p=00000000, init:#=871, wa:p=0XA000DC88, wa:#=871}
B dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
B   STMT:
B   CRSR: {tab='CATSDB', id=1, hold=0, prop=0x1000, max.in@0=1, fae:blk=1,
B          con:id=0, con:vndr=5, val=2,
B          key:#=4, xfer=0, xin:#=0, row:#=0, upto=0, wa:p=0XA000DC88}
C *** ERROR => Error 3114 in stmt_fetch()
[dbsloci.c    13001]
C *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
[dbsloci.c    13002]
C sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD
C SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
C R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
C sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD
C prep=0,lit=0,nsql=0,lobret=0,#exec=29,dbcnt=0,upsh_p=00000000,ocistmth_p=34B9ADD4
C IN : cols=3,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=45,vmax=32,bound=1,iobuf_p=0E0A4EC8,vda_p=31361B18
C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C OUT: cols=9,rmax=1,xcnt=1,rpc=1,rowi=0,rtot=1,upto=-1,rsize=56,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=345C39B8
C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
C R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
B ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
B ***LOG BY4=> sql error 3114   performing SEL on table REPOLOAD   [dbrepo#6 @ 2657] [dbrepo 2657 ]
B ***LOG BY0=> ORA-03114: not connected to ORACLE [dbrepo#6 @ 2657] [dbrepo 2657 ]
A TH VERBOSE LEVEL FULL
A ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.
A ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.
A ** RABAX: level LEV_RX_VMIT_CLEANUP entered.
A ** RABAX: level LEV_RX_VMIT_CLEANUP completed.
A ** RABAX: level LEV_RX_ROLLBACK entered.
C *** ERROR => OCI-call 'OCITransRollback' failed: rc = 3114
[dboci.c      3127]
B ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
B Reconnect state is entered by connection:
B 0: name = R/3, con_id = 000000000 state = ACTIVE      , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
B hdl_error_on_commit_rollback: DB-ROLLBACK detected RECONNECT state
M ThShortCommit: db unusable
B db_con_reconnect disconnecting connection:
B 0: name = R/3, con_id = 000000000 state = INACTIVE    , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
C Disconnecting from connection 0 ...
C Close user session (con_hdl=0,svchp=045D7FA4,usrhp=345E67A0)
C Detaching from DB Server (con_hdl=0,svchp=045D7FA4,srvhp=045D0174)
C Now I'm disconnected from ORACLE
B db_con_reconnect performing the reconnect for con:
B 0: name = R/3, con_id = 000000000 state = DISCONNECTED, perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
C got NLS_LANG='AMERICAN_AMERICA.US7ASCII' from environment
C Client NLS settings:
C Logon as OPS$-user to get SAPR3's password
C Connecting as /@PRD on connection 0 (nls_hdl 0) ... (dbsl 640 250407)
C Nls CharacterSet                 NationalCharSet              C      EnvHp      ErrHp ErrHpBatch
C    0 US7ASCII                                                  1   045CBB38   0E0A4648   0E0A40B0
C Attaching to DB Server PRD (con_hdl=0,svchp=045
SM51 log entries:
     09:00:32     MS                         Q0I     Operating system call recv failed (error no. 10054)
     09:00:32     DP                         Q0K     Connection to message server (on FCMSPROD) established
     09:00:32     DIA     004     040     181JOLENES     VA02     BYM     SQL error 3114 . Work processes in reconnect status
     09:00:32     DP                         Q1C     MsgServer Hardware ID Was Determined
     09:00:32     DIA     001     040     181ANNELINEJ     VA42     BYY     Work process has left reconnect status
     09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
     09:00:32     DIA     014     040     161ADRIV     VA02     BYY     Work process has left reconnect status
     09:00:32     DIA     013     040     140ELECIA          BYY     Work process has left reconnect status
     09:00:32     DIA     003     040     133MIRANDAV     KO88     BYY     Work process has left reconnect status
     09:00:32     DIA     003     040     133MIRANDAV     KO88     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
     09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB1     > Short dump "081128 090032 FCMSPROD 181ANNELINEJ" generated
     09:00:32     DIA     003     040     133MIRANDAV     KO88     AB1     > Short dump "081128 090032 FCMSPROD 133MIRANDAV " generated
     09:00:32     DIA     009     040     120ELIZEP     FB01     BYY     Work process has left reconnect status
     09:00:33     DIA     009     040     120ELIZEP     FB01     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
     09:00:33     DIA     004     040     181JOLENES     VA02     BYY     Work process has left reconnect status
     09:00:33     DIA     002     040     133KOEKIEJ          BYY     Work process has left reconnect status
     09:00:33     DIA     002     040     133KOEKIEJ          AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
     09:00:33     DIA     004     040     181JOLENES     VA02     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
     09:00:33     DIA     009     040     120ELIZEP     FB01     AB1     > Short dump "081128 090033 FCMSPROD 120ELIZEP " generated
     09:00:33     DIA     005     040     181CLIVEB     VA02     BYY     Work process has left reconnect status
     09:00:33     DIA     003                    R0Z     The update dispatch info was reset
     09:00:33     DIA     003                    R0Z     The update dispatch info was reset
Please can you assist in maybe you have seen this before.
Regards
Pieter

Hi
Yes I have gone through the Oracle Alert log several time, It is huge, but I am pasting an extract for the entire day this was happening into this reply. Oracle version is 9.2.0.8.0 with the only intrim path I can see being installed : 6130293.
Following is the extract from Alert Log for the day this occurred, again I must stress that this happens only on high system load days like month end:
Fri Nov 28 03:43:33 2008
ALTER TABLESPACE "PSAPBTABI" END BACKUP
Fri Nov 28 03:43:33 2008
Completed: ALTER TABLESPACE "PSAPBTABI" END BACKUP
Fri Nov 28 03:43:37 2008
ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
Fri Nov 28 03:53:38 2008
ALTER TABLESPACE "PSAPCLUD" END BACKUP
Fri Nov 28 03:53:38 2008
Completed: ALTER TABLESPACE "PSAPCLUD" END BACKUP
Fri Nov 28 03:53:42 2008
ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
Fri Nov 28 03:54:36 2008
ALTER TABLESPACE "PSAPCLUI" END BACKUP
Fri Nov 28 03:54:36 2008
Completed: ALTER TABLESPACE "PSAPCLUI" END BACKUP
Fri Nov 28 03:54:38 2008
ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
Fri Nov 28 03:55:47 2008
ALTER TABLESPACE "PSAPDDICD" END BACKUP
Fri Nov 28 03:55:47 2008
Completed: ALTER TABLESPACE "PSAPDDICD" END BACKUP
Fri Nov 28 03:55:50 2008
ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
Fri Nov 28 03:56:41 2008
ALTER TABLESPACE "PSAPDDICI" END BACKUP
Fri Nov 28 03:56:41 2008
Completed: ALTER TABLESPACE "PSAPDDICI" END BACKUP
Fri Nov 28 03:56:45 2008
ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
Fri Nov 28 03:56:49 2008
ALTER TABLESPACE "PSAPDOCUD" END BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUD" END BACKUP
Fri Nov 28 03:56:52 2008
ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
Fri Nov 28 03:56:55 2008
ALTER TABLESPACE "PSAPDOCUI" END BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUI" END BACKUP
Fri Nov 28 03:56:59 2008
ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
Fri Nov 28 04:03:46 2008
ALTER TABLESPACE "PSAPEL640D" END BACKUP
Fri Nov 28 04:03:46 2008
Completed: ALTER TABLESPACE "PSAPEL640D" END BACKUP
Fri Nov 28 04:03:50 2008
ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
Fri Nov 28 04:04:37 2008
ALTER TABLESPACE "PSAPEL640I" END BACKUP
Fri Nov 28 04:04:37 2008
Completed: ALTER TABLESPACE "PSAPEL640I" END BACKUP
Fri Nov 28 04:04:40 2008
ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
Fri Nov 28 04:20:17 2008
ALTER TABLESPACE "PSAPES640D" END BACKUP
Fri Nov 28 04:20:17 2008
Completed: ALTER TABLESPACE "PSAPES640D" END BACKUP
Fri Nov 28 04:20:20 2008
ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
Fri Nov 28 04:36:26 2008
ALTER TABLESPACE "PSAPES640I" END BACKUP
Fri Nov 28 04:36:26 2008
Completed: ALTER TABLESPACE "PSAPES640I" END BACKUP
Fri Nov 28 04:36:29 2008
ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
Fri Nov 28 04:36:40 2008
ALTER TABLESPACE "PSAPLOADD" END BACKUP
Fri Nov 28 04:36:40 2008
Completed: ALTER TABLESPACE "PSAPLOADD" END BACKUP
Fri Nov 28 04:36:43 2008
ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
Fri Nov 28 04:36:53 2008
ALTER TABLESPACE "PSAPLOADI" END BACKUP
Completed: ALTER TABLESPACE "PSAPLOADI" END BACKUP
Fri Nov 28 04:36:56 2008
ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
Fri Nov 28 04:41:04 2008
ALTER TABLESPACE "PSAPPOOLD" END BACKUP
Fri Nov 28 04:41:04 2008
Completed: ALTER TABLESPACE "PSAPPOOLD" END BACKUP
Fri Nov 28 04:41:08 2008
ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
Fri Nov 28 04:45:17 2008
ALTER TABLESPACE "PSAPPOOLI" END BACKUP
Fri Nov 28 04:45:17 2008
Completed: ALTER TABLESPACE "PSAPPOOLI" END BACKUP
Fri Nov 28 04:45:21 2008
ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
Fri Nov 28 04:47:25 2008
ALTER TABLESPACE "PSAPPROTD" END BACKUP
Fri Nov 28 04:47:25 2008
Completed: ALTER TABLESPACE "PSAPPROTD" END BACKUP
Fri Nov 28 04:47:29 2008
ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
Fri Nov 28 04:48:09 2008
ALTER TABLESPACE "PSAPPROTI" END BACKUP
Fri Nov 28 04:48:09 2008
Completed: ALTER TABLESPACE "PSAPPROTI" END BACKUP
Fri Nov 28 04:48:13 2008
ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
Fri Nov 28 04:48:46 2008
ALTER TABLESPACE "PSAPSOURCED" END BACKUP
Fri Nov 28 04:48:46 2008
Completed: ALTER TABLESPACE "PSAPSOURCED" END BACKUP
Fri Nov 28 04:48:49 2008
ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
Fri Nov 28 04:49:29 2008
ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
Fri Nov 28 04:49:29 2008
Completed: ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
Fri Nov 28 04:49:34 2008
ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
Fri Nov 28 05:05:16 2008
ALTER TABLESPACE "PSAPSTABD" END BACKUP
Fri Nov 28 05:05:16 2008
Completed: ALTER TABLESPACE "PSAPSTABD" END BACKUP
Fri Nov 28 05:05:23 2008
ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
Fri Nov 28 05:21:15 2008
ALTER TABLESPACE "PSAPSTABI" END BACKUP
Fri Nov 28 05:21:15 2008
Completed: ALTER TABLESPACE "PSAPSTABI" END BACKUP
Fri Nov 28 05:21:19 2008
ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
Fri Nov 28 05:21:42 2008
ALTER TABLESPACE "PSAPUSER1D" END BACKUP
Fri Nov 28 05:21:42 2008
Completed: ALTER TABLESPACE "PSAPUSER1D" END BACKUP
Fri Nov 28 05:21:46 2008
ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
Fri Nov 28 05:23:03 2008
ALTER TABLESPACE "PSAPUSER1I" END BACKUP
Fri Nov 28 05:23:03 2008
Completed: ALTER TABLESPACE "PSAPUSER1I" END BACKUP
Fri Nov 28 05:23:06 2008
ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
Completed: ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
Fri Nov 28 05:24:03 2008
ALTER TABLESPACE "SYSTEM" END BACKUP
Fri Nov 28 05:24:03 2008
Completed: ALTER TABLESPACE "SYSTEM" END BACKUP
Fri Nov 28 05:24:07 2008
ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
Completed: ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
Fri Nov 28 05:42:06 2008
ALTER TABLESPACE "UNDOTBS1" END BACKUP
Fri Nov 28 05:42:06 2008
Completed: ALTER TABLESPACE "UNDOTBS1" END BACKUP
Fri Nov 28 05:42:07 2008
ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program Files\CA\BrightStor ARCserve Backup Enterprise Option for SAP R3 for Oracle\CONTROL.PRD' REUSE
Completed: ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program F
Fri Nov 28 05:42:14 2008
Beginning log switch checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
Thread 1 advanced to log sequence 30706
Current log# 2 seq# 30706 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
Current log# 2 seq# 30706 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
Fri Nov 28 05:42:14 2008
ARC0: Evaluating archive   log 3 thread 1 sequence 30705
ARC0: Beginning to archive log 3 thread 1 sequence 30705
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30705.001'
Fri Nov 28 05:42:27 2008
ARC1: Evaluating archive   log 3 thread 1 sequence 30705
ARC1: Unable to archive log 3 thread 1 sequence 30705
      Log actively being archived by another process
Fri Nov 28 05:42:29 2008
ARC0: Completed archiving log 3 thread 1 sequence 30705
Fri Nov 28 06:11:26 2008
Completed checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
Fri Nov 28 09:20:59 2008
Beginning log switch checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
Thread 1 advanced to log sequence 30707
Current log# 1 seq# 30707 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
Current log# 1 seq# 30707 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
Fri Nov 28 09:20:59 2008
ARC0: Evaluating archive   log 2 thread 1 sequence 30706
ARC0: Beginning to archive log 2 thread 1 sequence 30706
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30706.001'
Fri Nov 28 09:21:18 2008
ARC0: Completed archiving log 2 thread 1 sequence 30706
Fri Nov 28 09:51:06 2008
Completed checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
Fri Nov 28 10:44:43 2008
Beginning log switch checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
Thread 1 advanced to log sequence 30708
Current log# 4 seq# 30708 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
Current log# 4 seq# 30708 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
Fri Nov 28 10:44:43 2008
ARC0: Evaluating archive   log 1 thread 1 sequence 30707
ARC0: Beginning to archive log 1 thread 1 sequence 30707
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30707.001'
Fri Nov 28 10:45:07 2008
ARC1: Evaluating archive   log 1 thread 1 sequence 30707
ARC1: Unable to archive log 1 thread 1 sequence 30707
      Log actively being archived by another process
Fri Nov 28 10:45:09 2008
ARC0: Completed archiving log 1 thread 1 sequence 30707
Fri Nov 28 11:13:20 2008
Completed checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
Fri Nov 28 11:45:09 2008
Beginning log switch checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
Thread 1 advanced to log sequence 30709
Current log# 3 seq# 30709 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
Current log# 3 seq# 30709 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
Fri Nov 28 11:45:09 2008
ARC0: Evaluating archive   log 4 thread 1 sequence 30708
ARC0: Beginning to archive log 4 thread 1 sequence 30708
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30708.001'
Fri Nov 28 11:45:13 2008
ARC1: Evaluating archive   log 4 thread 1 sequence 30708
ARC1: Unable to archive log 4 thread 1 sequence 30708
      Log actively being archived by another process
Fri Nov 28 11:45:29 2008
ARC0: Completed archiving log 4 thread 1 sequence 30708
Fri Nov 28 12:15:17 2008
Completed checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
Fri Nov 28 12:50:23 2008
Beginning log switch checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
Thread 1 advanced to log sequence 30710
Current log# 2 seq# 30710 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
Current log# 2 seq# 30710 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
Fri Nov 28 12:50:24 2008
ARC0: Evaluating archive   log 3 thread 1 sequence 30709
ARC0: Beginning to archive log 3 thread 1 sequence 30709
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30709.001'
Fri Nov 28 12:50:40 2008
ARC0: Completed archiving log 3 thread 1 sequence 30709
Fri Nov 28 13:20:29 2008
Completed checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
Fri Nov 28 14:27:01 2008
Beginning log switch checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
Thread 1 advanced to log sequence 30711
Current log# 1 seq# 30711 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
Current log# 1 seq# 30711 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
Fri Nov 28 14:27:01 2008
ARC0: Evaluating archive   log 2 thread 1 sequence 30710
ARC0: Beginning to archive log 2 thread 1 sequence 30710
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30710.001'
Fri Nov 28 14:27:25 2008
ARC0: Completed archiving log 2 thread 1 sequence 30710
Fri Nov 28 14:57:08 2008
Completed checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
Fri Nov 28 15:57:26 2008
Beginning log switch checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
Thread 1 advanced to log sequence 30712
Current log# 4 seq# 30712 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
Current log# 4 seq# 30712 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
Fri Nov 28 15:57:26 2008
ARC0: Evaluating archive   log 1 thread 1 sequence 30711
ARC0: Beginning to archive log 1 thread 1 sequence 30711
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30711.001'
Fri Nov 28 15:57:40 2008
ARC1: Evaluating archive   log 1 thread 1 sequence 30711
ARC1: Unable to archive log 1 thread 1 sequence 30711
      Log actively being archived by another process
Fri Nov 28 15:57:40 2008
ARC0: Completed archiving log 1 thread 1 sequence 30711
Fri Nov 28 16:27:31 2008
Completed checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
Fri Nov 28 17:02:12 2008
Beginning log switch checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
Thread 1 advanced to log sequence 30713
Current log# 3 seq# 30713 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
Current log# 3 seq# 30713 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
Fri Nov 28 17:02:12 2008
ARC0: Evaluating archive   log 4 thread 1 sequence 30712
ARC0: Beginning to archive log 4 thread 1 sequence 30712
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30712.001'
Fri Nov 28 17:02:25 2008
ARC0: Completed archiving log 4 thread 1 sequence 30712
Fri Nov 28 17:32:17 2008
Completed checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
Fri Nov 28 18:00:19 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
Fri Nov 28 18:00:19 2008
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
Fri Nov 28 18:00:19 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
Fri Nov 28 18:00:19 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
Fri Nov 28 18:00:20 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
Fri Nov 28 18:00:21 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
Fri Nov 28 18:00:23 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
Fri Nov 28 18:00:23 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
Fri Nov 28 18:00:23 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
Fri Nov 28 18:00:24 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
Fri Nov 28 18:00:26 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Fri Nov 28 20:00:17 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Fri Nov 28 20:00:36 2008
Beginning log switch checkpoint up to RBA [0x77fa.2.10], SCN: 0x0000.3c0cf8e3
Thread 1 advanced to log sequence 30714
Current log# 2 seq# 30714 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
Current log# 2 seq# 30714 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
Fri Nov 28 20:00:36 2008
ARC0: Evaluating archive   log 3 thread 1 sequence 30713
ARC0: Beginning to archive log 3 thread 1 sequence 30713
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30713.001'
Fri Nov 28 20:00:44 2008
ARC1: Evaluating archive   log 3 thread 1 sequence 30713
ARC1: Unable to archive log 3 thread 1 sequence 30713
      Log actively being archived by another process
Fri Nov 28 20:00:47 2008
ARC0: Completed archiving log 3 thread 1 sequence 30713
Dump file f:\oracle\prd\saptrace\background\alert_prd.log
Mon Dec 01 09:03:16 2008
ORACLE V9.2.0.8.0 - Production vsnsta=0
vsnsql=12 vsnxtr=3
Windows 2000 Version 5.2 Service Pack 2, CPU type 586
Mon Dec 01 09:03:16 2008
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
processes                = 120
sessions                 = 150
timed_statistics         = TRUE
shared_pool_size         = 822083584
large_pool_size          = 8388608
java_pool_size           = 8388608
enqueue_resources        = 8000
filesystemio_options     = SETALL
control_files            = C:\oracle\PRD\origlogA\ctrlPRD.ctl, F:\oracle\PRD\saparch\ctrlPRD.ctl, E:\oracle\PRD\sapdata1\ctrlPRD.ctl
db_block_checksum        = FALSE
db_block_size            = 8192
db_cache_size            = 1073741824
compatible               = 9.2.0
log_archive_start        = TRUE
log_archive_dest         = F:\oracle\PRD\saparch
log_buffer               = 4194304
log_checkpoint_interval = 0
db_files                 = 254
db_file_multiblock_read_count= 8
log_checkpoints_to_alert = TRUE
control_file_record_keep_time= 30
dml_locks                = 2500
row_locking              = always
replication_dependency_tracking= FALSE
transactions_per_rollback_segment= 20
transaction_auditing     = FALSE
undo_management          = AUTO
undo_tablespace          = UNDOTBS1
undo_retention           = 43200
remote_os_authent        = TRUE
db_domain                = world
service_names            = PRD.world, PRD.WORLD
hash_join_enabled        = FALSE
background_dump_dest     = F:\Oracle\PRD\saptrace\background
user_dump_dest           = F:\Oracle\PRD\saptrace\usertrace
max_dump_file_size       = 1240
core_dump_dest           = F:\Oracle\PRD\saptrace\background
optimizer_features_enable= 9.2.0
sort_area_size           = 0
sort_area_retained_size = 0
db_name                  = PRD
open_cursors             = 800
optimizer_mode           = CHOOSE
sortelimination_cost_ratio= 10
btree_bitmap_plans     = FALSE
eliminatecommon_subexpr= FALSE
pushjoin_predicate     = FALSE
optimizerjoin_sel_sanity_check= TRUE
parallel_threads_per_cpu = 4
optimizer_index_cost_adj = 10
optimizer_index_caching = 50
optimpeek_user_binds   = FALSE
pga_aggregate_target     = 536870912
optimizeror_expansion = DEPTH
PMON started with pid=2, OS id=17336
DBW0 started with pid=3, OS id=16972
LGWR started with pid=4, OS id=16900
CKPT started with pid=5, OS id=17236
SMON started with pid=6, OS id=17436
Regards
Pieter

Querying the toplink cache under high-load

We've had some interesting experiences with "querying" the TopLink Cache lately.
It was recently discovered that our "read a single object" method was incorrectly
setting query.checkCacheThenDB() for all ReadObjectQueries. This was brought to light
when we upgraded our production servers from 4 cores to 8. We immediatly started
experiencing very long response times under load.
We traced this down to the following stack: (TopLink version 10.1.3.1.0)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
at java.lang.Object.wait(Object.java:474)
at oracle.toplink.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:179)
- locked <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
at oracle.toplink.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:167)
at oracle.toplink.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:122)
at oracle.toplink.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:31)
at oracle.toplink.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:530)
at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:412)
at oracle.toplink.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:223)
at oracle.toplink.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:504)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:564)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
We moved the query back to the default, query.checkByPrimaryKey() and this issue went away.
The bottleneck seemed to stem from the read lock on the CacheKey from IdenityMapKeyEnumeration
public Object nextElement() {
if (this.nextKey == null) {
throw new NoSuchElementException("IdentityMapKeyEnumeration nextElement");
// CR#... Must check the read lock to avoid
// returning half built objects.
this.nextKey.checkReadLock();
return this.nextKey;
We had many threads that were having contention while searching the cache for a particular query.
From the stack we know that the contention was limited to one class. We've since refactored that code
not to use a query in that code path.
Question:
Armed with this better knowledge of how TopLink queries the cache, we do have a few objects that we
frequently read by something other than the primary key. A natural key, but not the oid.
We have some other caching mechanisms in place (JBoss TreeCache) to help eliminate queries to the DB
for these objects. But the TreeCache also tries to acquire a read lock when accessing the cache.
Presumably a read lock over the network to the cluster.
Is there anything that can be done about the read lock on CacheKey when querying the cache in a high load
situation?

CheckCacheThenDatabase will check the entire cache for a match using a linear search. This can be efficient if the cache is very large. Typically it is more efficient to access the database if your cache is large and the field you are querying in the database is indexed in the table.
The cache concurrency was greatly improved in TopLink 11g/EclipseLink, so you may wish to try it out.
Supporting indexes in the TopLink/EclipseLink cache is something desirable (feel free to log the enhancement request on EclipseLink). You can simulate this to some degree using a named query and a query cache.
-- James : http://www.eclipselink.org

High-load sql statement using v$sql

Hi,
Can any one please tell me, how do we find high load sql statement and it's user from v$SQL view.
what is the query to find it.
Thank you!

Hello,
You can run ADDM report and check its findings it will tell you tome stuff like the following:
     Finding
67      SQL statements consuming significant database time were found.
40.7      Time spent on the CPU by the instance was responsible for a substantial part of database time.
20.7      Individual SQL statements responsible for significant user I/O wait were found.
13.7      Individual database segments responsible for significant user I/O wait were found.Kind regards
Mohamed Elazab

Performance Bottleneck at HIgh Loads

Hi,
We are currently load testing our web application and have been facing an issue that has ben bothering us for sometime now.
If we load test the application at high load for sustained amounts of time, (around 2 hours), the performance suddenly dips to Zero hits/sec, Nothing is processed , no output generated. It seems as though all the processes some how are waiting for a resource to get free.After sometime the performance goes up again and again there is a race condition because of which the performance goes down again. Here on the dipping becomes highly erratic.
We are trying to find out if this is a known bug with the oracle jdbc driver. "oracle.jdbc.driver.OracleDriver" .
Would appreciate if anybody could provide any inputs as to why this is happening and how we could alleviate the situation.
Thanks in advance.
Deepanjan

Hi I vaguely remember something like this.... but I thought it typically occurred during lots of heavy insert/update activity
which version of the jdbc drivers are you using.... are you using thin/thick
Have you checked the SQL or the database to determine whether any locks are blocking activity or whether you have large database wait issues....
Dom

Struts not good for high-load requests?

I need a small advice from somebody with experience writing high-load web applications. I worked in Java servlets and was doing a bit in Struts but this is my first bigger project without working in the team so now need to decide the technology.
I have to do the J2EE project that has simple business logic but will take some time for database and I/O operations and the maximum number of users is expected to be big, a few hundreds of them in the same time. Struts + Hibernate combination is very comfortable for working but am now confused may I use it for this purpose because I heard that Struts' action is single-threaded. If action is single-threaded, does it mean that hundred requests will make a queue and wait for each other's I/O processing completely finishes before getting next request? If so, then I shouldn't use Struts but ordinary servlets, right?
Any hint will be very useful, thanks in advance!

Same problem here (I also do panoramas).
Actually I can't even import my files into Aperture.
Other, slighly smaller, files (PSD or TIFF) show similar problems than yours just that my Previews or Thumbnails decide to turn blue, displaying the "Unsupported format" message - after sometimes weeks of working fine.
But anyway, most of my files are usually too large to save out from Photoshop as PSD or even Tiff. PSB is unfortunately not supported by Aperture (according to the tech specs) and Aperture cannot indeed see the .psb files …
I shall hope that larger files in general and the large file format will be reliably supported by v3 (likely out right after Snow Leopard).
My current workaround:
For processing large size files I avoid Aperture altogether anyway, using Bridge/Photoshop (+ Photomatix and/or PTgui) instead.
Frankly, for me this combination works pretty well.
The final image will be saved out as HiQuality JPEG for Aperture, solely for the purpose of file management.
That JPEG will be stored in the same folder as the .psb and the other files (braketing and/or pano images). So by using "Show in Finder" I can quickly find the Photoshop original.

Very high "load average" in top

Hi,
our OES11SP1 two-server-cluster (fully patched) shows a very high "load
average" (>50, up to 110) in top in some circumstances. There are no
problems in normal operation, but administrator actions like shutdown or
cluster migrate might trigger the problem.
For example when I enter 'halt', then there is the following line in
/var/log/messages:
Sep 12 20:27:18 srv1 shutdown[14675]: shutting down for system halt
more than 20 minutes later:
Sep 12 20:51:19 srv1 init: Switching to runlevel: 0
Within thes 20 minutes nothing happens, but "average load" goes up to at
least 50, with ndsd at top. Access to storage related tools and commands is
not possible, for example 'nss /pool' hangs without any output.
This happens on nearly every shutdown, but from time to time it doesn't. The
same will sometimes be triggered by a cluster migrate.
This only happens with our OES11SP1 cluster, it does not happen with OES11
and OES2SP3; the only other difference I'm aware of: Novell CIFS is only
running on the OES11SP1 cluster.
Any ideas?
Thanks,
Mirko

Sorry for the delay, it seems it's a bad habit of me to ask questions
immediately before holidays...
Yes, these servers have replicas, all of them... Cache size is set to 195328
KB, which is about twice the DIB size. IIRC this was a recommendation I read
somewhere at Novell. But I'll check that information again.
Thanks,
Mirko
kjhurni wrote:
>
> Mirko Guldner;2283539 Wrote:
>> top shows ndsd on top - but it's there in normal operation too, so I
>> don't
>> know if this means something.. (?) And it's not always the CPU which is
>> at
>> 100% - I have an example screenshot with: load average 50.20, 51.61,
>> 41.0
>> 3.2%us, 1.0%sy, 0.0%ni, 77.0%id 18%wa 0.0%hi 0.3%si 0.0%st. But this is
>> only
>> an example - this differs.
>>
>> Thanks,
>> Mirko
>>
>> kjhurni wrote:
>>
>> >
>> > Mirko Guldner;2283448 Wrote:
>> >> Hi,
>> >>
>> >> our OES11SP1 two-server-cluster (fully patched) shows a very high
>> "load
>> >> average" (>50, up to 110) in top in some circumstances. There are no
>> >> problems in normal operation, but administrator actions like
>> shutdown
>> >> or
>> >> cluster migrate might trigger the problem.
>> >>
>> >> For example when I enter 'halt', then there is the following line in
>> >> /var/log/messages:
>> >>
>> >> Sep 12 20:27:18 srv1 shutdown[14675]: shutting down for system halt
>> >>
>> >> more than 20 minutes later:
>> >>
>> >> Sep 12 20:51:19 srv1 init: Switching to runlevel: 0
>> >>
>> >> Within thes 20 minutes nothing happens, but "average load" goes up
>> to
>> >> at
>> >> least 50, with ndsd at top. Access to storage related tools and
>> commands
>> >> is
>> >> not possible, for example 'nss /pool' hangs without any output.
>> >>
>> >> This happens on nearly every shutdown, but from time to time it
>> doesn't.
>> >> The
>> >> same will sometimes be triggered by a cluster migrate.
>> >>
>> >> This only happens with our OES11SP1 cluster, it does not happen with
>> >> OES11
>> >> and OES2SP3; the only other difference I'm aware of: Novell CIFS is
>> >> only
>> >> running on the OES11SP1 cluster.
>> >>
>> >> Any ideas?
>> >>
>> >> Thanks,
>> >> Mirko
>> >
>> > Which process(es) does top show as being the culprit?
>> >
>> > In the past (on OES2 SP3) we had issues with CIFS causing ncp to
>> cause
>> > high utilization, but that was fixed a while ago.
>> >
>> > --Kevin
>> >
>> >
>
> I have seen ncp issues cause high ndsd utilization, but we've not yet
> upgraded our cluster or DS servers to OES11 yet (waiting for new
> hardware to go in place first).
>
> Out of curiosity, are the servers with high utilization also replica
> servers? For some reason, during one of our upgrades on a replica
> server (we have a server that contains all R/W copies of everything),
> the cache size got set down really low and that caused all sorts of
> issues.
>
> Maybe one of my collegues will wander by and offer additional insight,
> as this may be eDir related and/or NCP related. Not sure if triggering
> a core manually would help (but you'd have to send that to Novell and
> open an SR to get it read).
>
> IF you suspect CIFS, do you have the ability to temporarily shut off
> CIFS for like a few days to see if that's the culprit?
>
>

Querying high load sql quies and users.

Hi all,
how do we find which query is performing high load and taking very long time on the database. and after that how we can find which user is giving that query and his session id.
Please give me the details and query if you have.
i tried with awr report and some queries with google search, but always i am getting result like no rows select.
thank you.

You've posted enough "101 (28 unresolved)" to know that you need to mark questions ANSWERED when they have been and to provide your 4 digit Oracle version (result of SELECT * FROM V$VERSION).
>
how do we find which query is performing high load and taking very long time on the database. and after that how we can find which user is giving that query and his session id.
>
If it ain't broke, don't fix it!
What makes you think you have any queries that are 'performing high load and taking very long time'? Post the information that you are basing this on.
>
i tried with awr report and some queries with google search, but always i am getting result like no rows select.
>
How can anyone comment on your awr report, your queries and your results if you don't post them?
Provide the details about what you have tried and what the results were.

What are considerations for Highly Transactional Database

Hi,
Can anyone please tell me about considerations for highly transactional database? Is Oracle 10g RAC better or Oracle DB with dataguard?
Thanks.
Regards,
RJiv.

I'm still not understanding what your question is... Load characteristics are quite irrelevant when discussing the necessity of DataGuard, though the amount of redo generated obviously impacts how much bandwidth is required between the primary and standby site. Bare transaction numbers are somewhat irrelevant when discussing the necessity or advisability of RAC since the amount of work a "transaction" does depends wildly on the application, the number of "transactions" a server can handle depends wildly on the hardware, and the business's need for scalability/ load balancing/ surviving node failure are independent of the transaction load.
Justin

High load and high CPU caused by perl process

Using: Oracle 10.2.0.1.0
OS: Red Hat ES 4
Since this morning I have high load on one of my prod database server, this server has 3 databases of similar size.
High load is caused by perl process, one process has been running for over 75 min’s. process info
14302 ?        R     74:50 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.pl
26530 ?        R     23:36 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.pl
31152 ?        R      7:58 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.plSearching segAdv.pl on google, Metalink and this forum gives me ZERO results.
Any idea why this is running and cuasing this issue now (for last 8 months, I had not simlar perl issue), can I stop it without kill –9, maybe with emctl stop dbconsole.
Thanks in advance for any help.

Hi Lee,
You say this is a core router, but as people use this terms to mean different things, are there any directly connected hosts? If there are one thing to check for is hosts sending multicast traffic with TTL=1. If you run the command show ip traffic | in bad hop and look at the rate the "bad hop count" is increasing. If I had a £ for every time I've seen that problem.... I couldn't retire just yet, but at least I could go buy myself a beer or two.
If you're seeing the hop count increasing at a rapid rate there are a couple of options:
Take a read of Troubleshooting high CPU on a 6500 with sup720 starting from the section Tools used to determine the source of the traffic and identify the hosts sending multicast with TTL=1.
If the Catalyst 6500 use PFC 3B add the command mls rate-limit all ttl-failure 100 10 to drop the multicast traffic in hardware.
The first option will take you longer, but at least you identify the cause and can advise the application owner why they should change their applications behaviour.
Regards

High load, database hangs

Similar Messages

Maybe you are looking for