OSB: ArrayIndexOutOfBoundsException: 4080 on high load

Hello,
I have a problem with the OSB 11.1.1.6. When I call services in the OSB with a certain load, I get the exception below. If I call the same services one by one (no high load), everything works. I didn't have this problem in the OSB 11.1.1.5.
Any idea?
[OSB Tracing] Outbound request was sent.
Service Ref = sampleBS
URI = https://someaddress
Request metadata =
<xml-fragment>
<tran:headers xsi:type="http:HttpRequestHeaders" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<tran:user-header..../>
<http:Accept>text/xml</http:Accept>
<http:Connection>Keep-Alive</http:Connection>
<http:Content-Type>text/xml; charset=UTF-8</http:Content-Type>
<http:User-Agent>Jakarta Commons-HttpClient/3.0</http:User-Agent>
</tran:headers>
<tran:encoding xmlns:tran="http://www.bea.com/wli/sb/transports">UTF-8</tran:encoding>
<http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http">sample</http:relative-URI>
<http:query-string xmlns:http="http://www.bea.com/wli/sb/transports/http">sample=sample</http:query-string>
<http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method>
</xml-fragment>
Payload =
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025394> <BEA-000000> <com.bea.wli.sb.transports.http.wls.HttpOutboundMessageContextWls: Request MetaData: <xml-fragment xmlns:tran="http://www.bea.com/wli/sb/transports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:http="http://www.bea.com/wli/sb/transports/http">
<tran:headers xsi:type="http:HttpRequestHeaders">
<tran:user-header.../>
<http:Accept>text/xml</http:Accept>
<http:Connection>Keep-Alive</http:Connection>
<http:Content-Type>text/xml; charset=UTF-8</http:Content-Type>
<http:User-Agent>Jakarta Commons-HttpClient/3.0</http:User-Agent>
</tran:headers>
<tran:encoding>UTF-8</tran:encoding>
<http:relative-URI>sample</http:relative-URI>
<http:query-string>sample=sample</http:query-string>
<http:http-method>GET</http:http-method>
</xml-fragment>>
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025395> <BEA-000000> <Sending outbound request to https://someaddresshere with QOS: best effort only...>
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025395> <BEA-000000> <*** Connecting via proxy *** HTTP @ someotheraddress/xx.xx.xx.xx:8080>
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025396> <BEA-000000> <-TransportManagerImpl.sendMessageAsync>
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbPipeline> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025396> <BEA-000000> <+RouterCallback.enableCallback>
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025396> <BEA-000000> <-TransportManagerImpl.receiveMessage>
####<Aug 7, 2012 1:13:45 PM CEST> <Debug> <AlsbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000097> <1344338025396> <BEA-000000> <-HttpTransportServlet.service>
####<Aug 7, 2012 1:13:45 PM CEST> <Error> <WliSbTransports> <xxx> <AdminServerDEV1> <[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <11d1def534ea1be0:-4539430:13900751647:-8000-0000000000000098> <1344338025510> <BEA-381304> <Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.lang.ArrayIndexOutOfBoundsException: 4080
java.lang.ArrayIndexOutOfBoundsException: 4080
     at weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:145)
     at java.io.SequenceInputStream.read(SequenceInputStream.java:149)
     at weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:162)
     at weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)
     at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:401)
     at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
     at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1005)
     at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.getResponse(HttpOutboundMessageContext.java:679)
     at com.bea.wli.sb.transports.http.wls.HttpOutboundMessageContextWls.access$100(HttpOutboundMessageContextWls.java:26)
     at com.bea.wli.sb.transports.http.wls.HttpOutboundMessageContextWls$RetrieveHttpResponseWork.handleResponse(HttpOutboundMessageContextWls.java:96)
     at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse$RunnableCallback.run(AsyncResponseHandler.java:535)
     at weblogic.work.ContextWrap.run(ContextWrap.java:41)
     at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
     at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Hi all,
I found the area to support Oracle reported a bug for this problem. I applied the patch and the problem was solved. The problem occurred when the BS called an external url in https.
The bug id is 14747231. Below is information about the bug.
*** Symptoms ***
With Certicom, the communication between OSB and external service is working fine. When they enable JSSE, they observe errors like the below and communication is failing:
####<Oct 2, 2012 10:30:13 AM EST> <Error> <WliSbTransports> <aubdc00-ofm01t> <osb_server1> <[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <2121cf169d8bc427:-4a412af5:13a00b4a777:-7fef-0000000000074ab8> <1349137813077> <BEA-381304> <Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.lang.ArrayIndexOutOfBoundsException
java.lang.ArrayIndexOutOfBoundsException
at weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:145)
at java.io.SequenceInputStream.read(SequenceInputStream.java:149)
at weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:162)
at weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:401)
at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1005)
at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.getResponse(HttpOutboundMessageContext.java:679)
at com.bea.wli.sb.transports.http.wls.HttpOutboundMessageContextWls.access$100(HttpOutboundMessageContextWls.java:26)
at com.bea.wli.sb.transports.http.wls.HttpOutboundMessageContextWls$RetrieveHttpResponseWork.handleResponse(HttpOutboundMessageContextWls.java:96)
at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse$RunnableCallback.run(AsyncResponseHandler.java:535)
at weblogic.work.ContextWrap.run(ContextWrap.java:41)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:531)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
*** Cause ***
If the ChunkedInputStream contains a Chunk chain having a non-null chunk and the end size is 0, ChunkedInputStream does not handle the chunk correctly. This issue is addressed by Bug 14747231, which introduces new logic to handle this condition.
*** Solution ***
Apply the patch. https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?patchId=14747231
Regards,

Similar Messages

  • OEM High Load Widget -- does it work with Windows 7?

    The README page for the OEM High Load Widget ( http://www.oracle.com/technetwork/oem/grid-control/emgridcontrol-10-101399.html ) does not list Windows 7 as a supported O/S. Has anyone been able to get it to work on Windows 7 (64-bit)?
    My new PC at work is running Windows 7 64-bit O/S. I've had good luck in getting everything I was running on my previous Win XP SP3 machine but the OEM High Load Widget is not connecting.
    I have found that widget to be quite useful and really would like to be able to use it.

    Yes,http://manuals.info.apple.com/en_US/time_capsule_4th_gen_setup.pdf

  • Weblogic Server hanging at high load!!!!!!!!!!!

    Folks,
    We have a weblogic server 9.2 which is running fine but after quiet a while at high load we observe that the server goes down and following exception is displayed in the server logs:
    ####<Nov 13, 2009 2:42:48 AM EST> <Error> <WebLogicServer> <ibmrxcptpnp-wb1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1258098168215> <BEA-000337> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "617" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@1cec1cec", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
         java.net.SocketOutputStream.socketWrite0(Native Method)
         java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105)
         java.net.SocketOutputStream.write(SocketOutputStream.java:149)
         oracle.net.ns.DataPacket.send(Unknown Source)
         oracle.net.ns.NetOutputStream.flush(Unknown Source)
         oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         oracle.net.ns.NetInputStream.read(Unknown Source)
         oracle.net.ns.NetInputStream.read(Unknown Source)
         oracle.net.ns.NetInputStream.read(Unknown Source)
         oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
         oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
         oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
         oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
         oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
         oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1060)
         oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839)
         oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132)
         oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
         oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3422)
         weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:718)
         weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:460)
         weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1455)
         weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1372)
         weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1362)
         weblogic.common.resourcepool.ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:1767)
         weblogic.common.resourcepool.ResourcePoolImpl.access$700(ResourcePoolImpl.java:37)
         weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:1935)
         weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
         weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
         weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    >
    Any answers?????

    Hi,
    I have a similar issue on ODSI 10gR3 (WLS 10.3). I am having stuck threads - but, in my case it is clear that the database is down (not at the startup)
    Here is some more info:
    1. Test Connections On Reserve - Enabled
    2. Default values for other parameters
    Using Oracle thin driver for the connection pool "MyPool" -
    ./wlserver_10.3/server/lib/ojdbc6.jar
    My question is:
    1. How do I avoid the stuck thread caused when the database is down? During peak load, the stuck threads are causing some service requests to fail.
    I did my homework and the closest option I could find is set the statementTimeout (which is set currently to the default value of -1 i.e. never timeout). Oracle documentation (http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/111070_readme.html) says that the ojdbc6.jar supports setQueryTimeout() method.
    Is there any better way to attack this problem?
    <Dec 21, 2009 12:11:18 AM EST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "MyPool" failed with exception: "java.sql.SQLRecoverableException: Io exception: Read failed: Connection timed out".>
    <Dec 21, 2009 12:14:27 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:18:38 AM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@297ac2b", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-52 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: javax.management.remote.rmi.RMIConnector$RMINotifClient@271498f[fat lock]
    java.lang.Object.wait(Object.java:485)
    com.sun.jmx.remote.internal.ClientNotifForwarder.postReconnection(ClientNotifForwarder.java:304)
    javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.reconnectNotificationListeners(RMIConnector.java:1488)
    javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.doStart(RMIConnector.java:1568)
    com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:72)
    com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
    javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1420)
    javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:857)
    weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.getAttribute(ManagedMBeanServerConnection.java:288)
    javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:235)
    weblogic.management.jmx.MBeanServerInvocationHandler.doInvoke(MBeanServerInvocationHandler.java:477)
    weblogic.management.jmx.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:294)
    $Proxy69.getServerRuntime(Unknown Source)
    weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.lookupServerRuntime(DomainRuntimeServiceMBeanImpl.java:242)
    sun.reflect.GeneratedMethodAccessor1880.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:355)
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
    com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
    weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:255)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:441)
    weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:55)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:441)
    weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:437)
    weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10$1.run(AuthenticatedSubjectInterceptor.java:582)
    weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10.run(AuthenticatedSubjectInterceptor.java:576)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor.invoke(AuthenticatedSubjectInterceptor.java:570)
    weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:305)
    javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1378)
    javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
    javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
    javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1338)
    javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:761)
    javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:136)
    weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:211)
    javax.management.remote.rmi.RMIConnectionImpl_1030_WLStub.invoke(Unknown Source)
    javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:969)
    com.bea.diagnostics.server.MBeanServerUtil.processRecordsFromArchive(MBeanServerUtil.java:177)
    com.bea.diagnostics.server.MetricEarliestTimestampLocator$RecordProcessorDesc.identifyEarliestTimestamps(MetricEarliestTimestampLocator.java:261)
    com.bea.diagnostics.server.MetricEarliestTimestampLocator.identifyEarliestTimestamps(MetricEarliestTimestampLocator.java:84)
    com.bea.diagnostics.server.MetricEarliestTimestampLocator.timerExpired(MetricEarliestTimestampLocator.java:79)
    weblogic.timers.internal.TimerImpl.run(TimerImpl.java:253)
    weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    <Dec 21, 2009 12:24:04 AM EST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "MyPool" failed with exception: "java.sql.SQLRecoverableException: Io exception: Connection reset".>
    <Dec 21, 2009 12:24:04 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:04 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:05 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:07 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:12 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:17 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:22 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>

  • 2008 R2/ Win7 Offline Files Sync causing High Load on server

    Hi Everyone,
    I have recently been investigating extremely high CPU Usage from the System Process on my company's main File Cluster.
    We managed to track SRV2.sys Threads causing high CPU load within the system process, but was having issues identifying why this was the case.
    As per Microsoft's direction via support call, we have installed the latest SRV2.sys Hotfixes, but this does not appear to have allivated the issue we are experiencing. We have also added more CPU and Memory into both nodes, which has not helped either.
    We have since managed to create a system dump and is being sent to MS Support for analysis.
    I have noticed the following that appears to happen on our cluster:
    Whenever our CAD/Design department run certain functions within their apps running on a windows 7 client (apps include MicroStation, Revit, AutoCAD etc) we see a massive spike and flatline of the system process.
    We found several users with Windows 7 clients that have Configured Offline File to sync an entire Network Volume (some volumes are 2TB Plus, so would never fit on a users computer anyway, i was quite shocked when I found this). How we spotted this was through
    Resource Monitor, the System Process was trolling through all the folders and files in a given volume (it was "reading every single folder). Now, while this was the system process, we could identify the user by using the Open Files view in Server Manager's
    Share and Storage Management tool.
    I have done a fair bit of research and found that a lot of CAD/Drawing applications in the market have issues with using SMB2 (srv2.sys). When reviewing the products that we use, I noticed that a lot of these products actually recommended disabling SMB2
    and reverting to SMB1 on File Server and/or the clients.
    I have gone down the path of disabling SMB2 on all Windows 7 clients that have these CAD Applications installed to assist with lowering the load (our other options are to potentially shift the CAD Volumes off our main file cluster to further isolation these
    performance issues we have been experiencing.) We will be testing this again tomorrow to confirm that the issue is resolved when mass amounts of CAD users access data on these volumes via their CAD Application.
    We only noticed the issue with Offline Files today with trying to sync an ENTIRE Volume. My questions are:
    Should Offline File sync's really cause this much load on a File Server?
    Would the the size of the volume the sync is trying to complete create this additional load within the system process?
    What is the industry considered "Best Practice" regarding Offline Files setup for Volumes which could have up to 1000+ users attached? (My personal opinion is that Offline Files should only be sync of user "Personal/Home" Folders
    as they users themselves have a 1 to 1 relationship with this data.)
    Is there an easier way to identify what users have Offline Files enabled and "actually being used" (from my understanding, Sync Center and Offline Files are enabled by default, but you obviously have to add the folders/drives you wish to sync)?
    If I disable the ability for Offline Files on the volumes, what will the user experience be when/if they try to sync their offline files config after this has been disabled on the volume?
    Hoping for some guidance regarding this setup with Offline Files.
    Thanks in Advance!
    Simon

    Hi Everyone,
    Just thought I would give an update on this.
    While we're still deploying http://support.microsoft.com/kb/2733363/en-us to
    the remainder of our Windows 7 SP1 fleet, according to some Network Traces and XPerf that were sent to MS Support, it looks as though the issu with the Offline Files is now resolved.
    However, we are still having some issues with High CPU with the System Process in particular. Upon review of the traces, they have found a lot of ABE related traffic, leading them to believe that the issue may also be caused by ABE Access on our File Cluster.
    We already have the required hotfix for ABE on 2008 R2 installed (please see
    http://support.microsoft.com/kb/2732618/en-us), although we have it set to a certain value that MS believe may be too high. Our current value is set to "5", as that is the lowest level of any type of permission is set (i.e. where the lowest level of inheritance
    is broken).
    I think I will mark this as resolved regarding the Offline Files issue (as it did resolve the problem with the Offline Files)...
    Fingers crossed we can identify the rest of the high load within the System Process!!!

  • Strange Oracle Behaviour - only month end or high load periods

    Hi SAP Gurus
    In one of our systems I am expreriencing a strange behaviour from Oracle, in that it seems to be in reconnect, but yet it is not, it resets and carries on but do produce Dump and SM21 log messages, the dump is not really helpfull as it is the same dump but with different program everytime.
    I have looked in oracle alert log and found no event tying back to the SM21 log message or dump.
    The following is what I found in the developer trace of one of these insident, but there are many, about 125 - 200 for the day during high load:
    Fri Nov 28 10:05:55 2008
    C  *** ERROR => Failed to register application info: orc=3113
    [dboci.c      3042]
    C  *** ERROR => Error 3114 in stmt_fetch()
    [dbsloci.c    13001]
    C  *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
    [dbsloci.c    13002]
    C  sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB                       
    C  SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
    C  NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
    C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
    C  "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
    C  GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
    C  INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
    C  ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
    C  TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
    C  DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
    C  PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
    C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
    C  SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
    C  FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
    C  sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB                       
    C  prep=0,lit=0,nsql=0,lobret=0,#exec=1,dbcnt=0,upsh_p=00000000,ocistmth_p=345E20EC
    C  IN : cols=4,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=17,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=3470A8D8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  OUT: cols=105,rmax=58,xcnt=58,rpc=1,rowi=0,rtot=1,upto=-1,rsize=895,vmax=128,bound=1,iobuf_p=0E0A4EC8,vda_p=34F8F7E8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
    C  NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
    C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
    C  "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
    C  GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
    C  INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
    C  ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
    C  TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
    C  DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
    C  PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
    C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
    C  SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
    C  FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  ***LOG BY4=> sql error 3114   performing FET on table CATSDB     [dbtran#10 @ 7292] [dbtran  7292 ]
    B  ***LOG BY0=> ORA-03114: not connected to ORACLE [dbtran#10 @ 7292] [dbtran  7292 ]
    B  dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
    B   RSLT: {dbsl=99, tran=1}
    B   FHDR: {tab='CATSDB', fcode=225, mode=2, bpb=0, dbcnt=0, crsr=1,
    B          hold=0, keep=1, xfer=0, pkg=0, upto=0, init:b=0,
    B          init:p=00000000, init:#=871, wa:p=0XA000DC88, wa:#=871}
    B  dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
    B   STMT:
    B   CRSR: {tab='CATSDB', id=1, hold=0, prop=0x1000, max.in@0=1, fae:blk=1,
    B          con:id=0, con:vndr=5, val=2,
    B          key:#=4, xfer=0, xin:#=0, row:#=0, upto=0, wa:p=0XA000DC88}
    C  *** ERROR => Error 3114 in stmt_fetch()
    [dbsloci.c    13001]
    C  *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
    [dbsloci.c    13002]
    C  sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD                     
    C  SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
    C  R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
    C  sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD                     
    C  prep=0,lit=0,nsql=0,lobret=0,#exec=29,dbcnt=0,upsh_p=00000000,ocistmth_p=34B9ADD4
    C  IN : cols=3,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=45,vmax=32,bound=1,iobuf_p=0E0A4EC8,vda_p=31361B18
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  OUT: cols=9,rmax=1,xcnt=1,rpc=1,rowi=0,rtot=1,upto=-1,rsize=56,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=345C39B8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
    C  R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  ***LOG BY4=> sql error 3114   performing SEL on table REPOLOAD   [dbrepo#6 @ 2657] [dbrepo  2657 ]
    B  ***LOG BY0=> ORA-03114: not connected to ORACLE [dbrepo#6 @ 2657] [dbrepo  2657 ]
    A  TH VERBOSE LEVEL FULL
    A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.
    A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.
    A  ** RABAX: level LEV_RX_VMIT_CLEANUP entered.
    A  ** RABAX: level LEV_RX_VMIT_CLEANUP completed.
    A  ** RABAX: level LEV_RX_ROLLBACK entered.
    C  *** ERROR => OCI-call 'OCITransRollback' failed: rc = 3114
    [dboci.c      3127]
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  Reconnect state is entered by connection:
    B  0: name = R/3, con_id = 000000000 state = ACTIVE      , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    B  hdl_error_on_commit_rollback: DB-ROLLBACK detected RECONNECT state
    M  ThShortCommit: db unusable
    B  db_con_reconnect disconnecting connection:
    B  0: name = R/3, con_id = 000000000 state = INACTIVE    , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    C  Disconnecting from connection 0 ...
    C  Close user session (con_hdl=0,svchp=045D7FA4,usrhp=345E67A0)
    C  Detaching from DB Server (con_hdl=0,svchp=045D7FA4,srvhp=045D0174)
    C  Now I'm disconnected from ORACLE
    B  db_con_reconnect performing the reconnect for con:
    B  0: name = R/3, con_id = 000000000 state = DISCONNECTED, perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    C  got NLS_LANG='AMERICAN_AMERICA.US7ASCII' from environment
    C  Client NLS settings:
    C  Logon as OPS$-user to get SAPR3's password
    C  Connecting as /@PRD on connection 0 (nls_hdl 0) ... (dbsl 640 250407)
    C  Nls CharacterSet                 NationalCharSet              C      EnvHp      ErrHp ErrHpBatch
    C    0 US7ASCII                                                  1   045CBB38   0E0A4648   0E0A40B0
    C  Attaching to DB Server PRD (con_hdl=0,svchp=045
    SM51 log entries:
         09:00:32     MS                         Q0I     Operating system call recv failed (error no. 10054)
         09:00:32     DP                         Q0K     Connection to message server (on FCMSPROD) established
         09:00:32     DIA     004     040     181JOLENES     VA02     BYM     SQL error 3114 . Work processes in reconnect status
         09:00:32     DP                         Q1C     MsgServer Hardware ID Was Determined
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     BYY     Work process has left reconnect status
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:32     DIA     014     040     161ADRIV     VA02     BYY     Work process has left reconnect status
         09:00:32     DIA     013     040     140ELECIA          BYY     Work process has left reconnect status
         09:00:32     DIA     003     040     133MIRANDAV     KO88     BYY     Work process has left reconnect status
         09:00:32     DIA     003     040     133MIRANDAV     KO88     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB1     > Short dump "081128 090032 FCMSPROD 181ANNELINEJ" generated
         09:00:32     DIA     003     040     133MIRANDAV     KO88     AB1     > Short dump "081128 090032 FCMSPROD 133MIRANDAV " generated
         09:00:32     DIA     009     040     120ELIZEP     FB01     BYY     Work process has left reconnect status
         09:00:33     DIA     009     040     120ELIZEP     FB01     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     004     040     181JOLENES     VA02     BYY     Work process has left reconnect status
         09:00:33     DIA     002     040     133KOEKIEJ          BYY     Work process has left reconnect status
         09:00:33     DIA     002     040     133KOEKIEJ          AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     004     040     181JOLENES     VA02     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     009     040     120ELIZEP     FB01     AB1     > Short dump "081128 090033 FCMSPROD 120ELIZEP " generated
         09:00:33     DIA     005     040     181CLIVEB     VA02     BYY     Work process has left reconnect status
         09:00:33     DIA     003                    R0Z     The update dispatch info was reset
         09:00:33     DIA     003                    R0Z     The update dispatch info was reset
    Please can you assist in maybe you have seen this before.
    Regards
    Pieter

    Hi
    Yes I have gone through the Oracle Alert log several time, It is huge, but I am pasting an extract for the entire day this was happening into this reply.  Oracle version is 9.2.0.8.0 with the only intrim path I can see being installed : 6130293.
    Following is the extract from Alert Log for the day this occurred, again I must stress that this happens only on high system load days like month end:
    Fri Nov 28 03:43:33 2008
    ALTER TABLESPACE "PSAPBTABI" END BACKUP
    Fri Nov 28 03:43:33 2008
    Completed: ALTER TABLESPACE "PSAPBTABI" END BACKUP
    Fri Nov 28 03:43:37 2008
    ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
    Fri Nov 28 03:53:38 2008
    ALTER TABLESPACE "PSAPCLUD" END BACKUP
    Fri Nov 28 03:53:38 2008
    Completed: ALTER TABLESPACE "PSAPCLUD" END BACKUP
    Fri Nov 28 03:53:42 2008
    ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
    Fri Nov 28 03:54:36 2008
    ALTER TABLESPACE "PSAPCLUI" END BACKUP
    Fri Nov 28 03:54:36 2008
    Completed: ALTER TABLESPACE "PSAPCLUI" END BACKUP
    Fri Nov 28 03:54:38 2008
    ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
    Fri Nov 28 03:55:47 2008
    ALTER TABLESPACE "PSAPDDICD" END BACKUP
    Fri Nov 28 03:55:47 2008
    Completed: ALTER TABLESPACE "PSAPDDICD" END BACKUP
    Fri Nov 28 03:55:50 2008
    ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
    Fri Nov 28 03:56:41 2008
    ALTER TABLESPACE "PSAPDDICI" END BACKUP
    Fri Nov 28 03:56:41 2008
    Completed: ALTER TABLESPACE "PSAPDDICI" END BACKUP
    Fri Nov 28 03:56:45 2008
    ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
    Fri Nov 28 03:56:49 2008
    ALTER TABLESPACE "PSAPDOCUD" END BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUD" END BACKUP
    Fri Nov 28 03:56:52 2008
    ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
    Fri Nov 28 03:56:55 2008
    ALTER TABLESPACE "PSAPDOCUI" END BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUI" END BACKUP
    Fri Nov 28 03:56:59 2008
    ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
    Fri Nov 28 04:03:46 2008
    ALTER TABLESPACE "PSAPEL640D" END BACKUP
    Fri Nov 28 04:03:46 2008
    Completed: ALTER TABLESPACE "PSAPEL640D" END BACKUP
    Fri Nov 28 04:03:50 2008
    ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
    Fri Nov 28 04:04:37 2008
    ALTER TABLESPACE "PSAPEL640I" END BACKUP
    Fri Nov 28 04:04:37 2008
    Completed: ALTER TABLESPACE "PSAPEL640I" END BACKUP
    Fri Nov 28 04:04:40 2008
    ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
    Fri Nov 28 04:20:17 2008
    ALTER TABLESPACE "PSAPES640D" END BACKUP
    Fri Nov 28 04:20:17 2008
    Completed: ALTER TABLESPACE "PSAPES640D" END BACKUP
    Fri Nov 28 04:20:20 2008
    ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
    Fri Nov 28 04:36:26 2008
    ALTER TABLESPACE "PSAPES640I" END BACKUP
    Fri Nov 28 04:36:26 2008
    Completed: ALTER TABLESPACE "PSAPES640I" END BACKUP
    Fri Nov 28 04:36:29 2008
    ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
    Fri Nov 28 04:36:40 2008
    ALTER TABLESPACE "PSAPLOADD" END BACKUP
    Fri Nov 28 04:36:40 2008
    Completed: ALTER TABLESPACE "PSAPLOADD" END BACKUP
    Fri Nov 28 04:36:43 2008
    ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
    Fri Nov 28 04:36:53 2008
    ALTER TABLESPACE "PSAPLOADI" END BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADI" END BACKUP
    Fri Nov 28 04:36:56 2008
    ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
    Fri Nov 28 04:41:04 2008
    ALTER TABLESPACE "PSAPPOOLD" END BACKUP
    Fri Nov 28 04:41:04 2008
    Completed: ALTER TABLESPACE "PSAPPOOLD" END BACKUP
    Fri Nov 28 04:41:08 2008
    ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
    Fri Nov 28 04:45:17 2008
    ALTER TABLESPACE "PSAPPOOLI" END BACKUP
    Fri Nov 28 04:45:17 2008
    Completed: ALTER TABLESPACE "PSAPPOOLI" END BACKUP
    Fri Nov 28 04:45:21 2008
    ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
    Fri Nov 28 04:47:25 2008
    ALTER TABLESPACE "PSAPPROTD" END BACKUP
    Fri Nov 28 04:47:25 2008
    Completed: ALTER TABLESPACE "PSAPPROTD" END BACKUP
    Fri Nov 28 04:47:29 2008
    ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
    Fri Nov 28 04:48:09 2008
    ALTER TABLESPACE "PSAPPROTI" END BACKUP
    Fri Nov 28 04:48:09 2008
    Completed: ALTER TABLESPACE "PSAPPROTI" END BACKUP
    Fri Nov 28 04:48:13 2008
    ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
    Fri Nov 28 04:48:46 2008
    ALTER TABLESPACE "PSAPSOURCED" END BACKUP
    Fri Nov 28 04:48:46 2008
    Completed: ALTER TABLESPACE "PSAPSOURCED" END BACKUP
    Fri Nov 28 04:48:49 2008
    ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
    Fri Nov 28 04:49:29 2008
    ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
    Fri Nov 28 04:49:29 2008
    Completed: ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
    Fri Nov 28 04:49:34 2008
    ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
    Fri Nov 28 05:05:16 2008
    ALTER TABLESPACE "PSAPSTABD" END BACKUP
    Fri Nov 28 05:05:16 2008
    Completed: ALTER TABLESPACE "PSAPSTABD" END BACKUP
    Fri Nov 28 05:05:23 2008
    ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
    Fri Nov 28 05:21:15 2008
    ALTER TABLESPACE "PSAPSTABI" END BACKUP
    Fri Nov 28 05:21:15 2008
    Completed: ALTER TABLESPACE "PSAPSTABI" END BACKUP
    Fri Nov 28 05:21:19 2008
    ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
    Fri Nov 28 05:21:42 2008
    ALTER TABLESPACE "PSAPUSER1D" END BACKUP
    Fri Nov 28 05:21:42 2008
    Completed: ALTER TABLESPACE "PSAPUSER1D" END BACKUP
    Fri Nov 28 05:21:46 2008
    ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
    Fri Nov 28 05:23:03 2008
    ALTER TABLESPACE "PSAPUSER1I" END BACKUP
    Fri Nov 28 05:23:03 2008
    Completed: ALTER TABLESPACE "PSAPUSER1I" END BACKUP
    Fri Nov 28 05:23:06 2008
    ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
    Completed: ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
    Fri Nov 28 05:24:03 2008
    ALTER TABLESPACE "SYSTEM" END BACKUP
    Fri Nov 28 05:24:03 2008
    Completed: ALTER TABLESPACE "SYSTEM" END BACKUP
    Fri Nov 28 05:24:07 2008
    ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
    Completed: ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
    Fri Nov 28 05:42:06 2008
    ALTER TABLESPACE "UNDOTBS1" END BACKUP
    Fri Nov 28 05:42:06 2008
    Completed: ALTER TABLESPACE "UNDOTBS1" END BACKUP
    Fri Nov 28 05:42:07 2008
    ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program Files\CA\BrightStor ARCserve Backup Enterprise Option for SAP R3 for Oracle\CONTROL.PRD' REUSE
    Completed: ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program F
    Fri Nov 28 05:42:14 2008
    Beginning log switch checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
    Thread 1 advanced to log sequence 30706
      Current log# 2 seq# 30706 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30706 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 05:42:14 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30705
    ARC0: Beginning to archive log 3 thread 1 sequence 30705
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30705.001'
    Fri Nov 28 05:42:27 2008
    ARC1: Evaluating archive   log 3 thread 1 sequence 30705
    ARC1: Unable to archive log 3 thread 1 sequence 30705
          Log actively being archived by another process
    Fri Nov 28 05:42:29 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30705
    Fri Nov 28 06:11:26 2008
    Completed checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
    Fri Nov 28 09:20:59 2008
    Beginning log switch checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
    Thread 1 advanced to log sequence 30707
      Current log# 1 seq# 30707 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
      Current log# 1 seq# 30707 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
    Fri Nov 28 09:20:59 2008
    ARC0: Evaluating archive   log 2 thread 1 sequence 30706
    ARC0: Beginning to archive log 2 thread 1 sequence 30706
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30706.001'
    Fri Nov 28 09:21:18 2008
    ARC0: Completed archiving  log 2 thread 1 sequence 30706
    Fri Nov 28 09:51:06 2008
    Completed checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
    Fri Nov 28 10:44:43 2008
    Beginning log switch checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
    Thread 1 advanced to log sequence 30708
      Current log# 4 seq# 30708 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
      Current log# 4 seq# 30708 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
    Fri Nov 28 10:44:43 2008
    ARC0: Evaluating archive   log 1 thread 1 sequence 30707
    ARC0: Beginning to archive log 1 thread 1 sequence 30707
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30707.001'
    Fri Nov 28 10:45:07 2008
    ARC1: Evaluating archive   log 1 thread 1 sequence 30707
    ARC1: Unable to archive log 1 thread 1 sequence 30707
          Log actively being archived by another process
    Fri Nov 28 10:45:09 2008
    ARC0: Completed archiving  log 1 thread 1 sequence 30707
    Fri Nov 28 11:13:20 2008
    Completed checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
    Fri Nov 28 11:45:09 2008
    Beginning log switch checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
    Thread 1 advanced to log sequence 30709
      Current log# 3 seq# 30709 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
      Current log# 3 seq# 30709 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
    Fri Nov 28 11:45:09 2008
    ARC0: Evaluating archive   log 4 thread 1 sequence 30708
    ARC0: Beginning to archive log 4 thread 1 sequence 30708
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30708.001'
    Fri Nov 28 11:45:13 2008
    ARC1: Evaluating archive   log 4 thread 1 sequence 30708
    ARC1: Unable to archive log 4 thread 1 sequence 30708
          Log actively being archived by another process
    Fri Nov 28 11:45:29 2008
    ARC0: Completed archiving  log 4 thread 1 sequence 30708
    Fri Nov 28 12:15:17 2008
    Completed checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
    Fri Nov 28 12:50:23 2008
    Beginning log switch checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
    Thread 1 advanced to log sequence 30710
      Current log# 2 seq# 30710 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30710 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 12:50:24 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30709
    ARC0: Beginning to archive log 3 thread 1 sequence 30709
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30709.001'
    Fri Nov 28 12:50:40 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30709
    Fri Nov 28 13:20:29 2008
    Completed checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
    Fri Nov 28 14:27:01 2008
    Beginning log switch checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
    Thread 1 advanced to log sequence 30711
      Current log# 1 seq# 30711 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
      Current log# 1 seq# 30711 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
    Fri Nov 28 14:27:01 2008
    ARC0: Evaluating archive   log 2 thread 1 sequence 30710
    ARC0: Beginning to archive log 2 thread 1 sequence 30710
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30710.001'
    Fri Nov 28 14:27:25 2008
    ARC0: Completed archiving  log 2 thread 1 sequence 30710
    Fri Nov 28 14:57:08 2008
    Completed checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
    Fri Nov 28 15:57:26 2008
    Beginning log switch checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
    Thread 1 advanced to log sequence 30712
      Current log# 4 seq# 30712 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
      Current log# 4 seq# 30712 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
    Fri Nov 28 15:57:26 2008
    ARC0: Evaluating archive   log 1 thread 1 sequence 30711
    ARC0: Beginning to archive log 1 thread 1 sequence 30711
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30711.001'
    Fri Nov 28 15:57:40 2008
    ARC1: Evaluating archive   log 1 thread 1 sequence 30711
    ARC1: Unable to archive log 1 thread 1 sequence 30711
          Log actively being archived by another process
    Fri Nov 28 15:57:40 2008
    ARC0: Completed archiving  log 1 thread 1 sequence 30711
    Fri Nov 28 16:27:31 2008
    Completed checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
    Fri Nov 28 17:02:12 2008
    Beginning log switch checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
    Thread 1 advanced to log sequence 30713
      Current log# 3 seq# 30713 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
      Current log# 3 seq# 30713 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
    Fri Nov 28 17:02:12 2008
    ARC0: Evaluating archive   log 4 thread 1 sequence 30712
    ARC0: Beginning to archive log 4 thread 1 sequence 30712
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30712.001'
    Fri Nov 28 17:02:25 2008
    ARC0: Completed archiving  log 4 thread 1 sequence 30712
    Fri Nov 28 17:32:17 2008
    Completed checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
    Fri Nov 28 18:00:19 2008
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
    Fri Nov 28 18:00:20 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
    Fri Nov 28 18:00:21 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
    Fri Nov 28 18:00:24 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
    Fri Nov 28 18:00:26 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Fri Nov 28 20:00:17 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Fri Nov 28 20:00:36 2008
    Beginning log switch checkpoint up to RBA [0x77fa.2.10], SCN: 0x0000.3c0cf8e3
    Thread 1 advanced to log sequence 30714
      Current log# 2 seq# 30714 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30714 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 20:00:36 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30713
    ARC0: Beginning to archive log 3 thread 1 sequence 30713
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30713.001'
    Fri Nov 28 20:00:44 2008
    ARC1: Evaluating archive   log 3 thread 1 sequence 30713
    ARC1: Unable to archive log 3 thread 1 sequence 30713
          Log actively being archived by another process
    Fri Nov 28 20:00:47 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30713
    Dump file f:\oracle\prd\saptrace\background\alert_prd.log
    Mon Dec 01 09:03:16 2008
    ORACLE V9.2.0.8.0 - Production vsnsta=0
    vsnsql=12 vsnxtr=3
    Windows 2000 Version 5.2 Service Pack 2, CPU type 586
    Mon Dec 01 09:03:16 2008
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    SCN scheme 2
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up ORACLE RDBMS Version: 9.2.0.8.0.
    System parameters with non-default values:
      processes                = 120
      sessions                 = 150
      timed_statistics         = TRUE
      shared_pool_size         = 822083584
      large_pool_size          = 8388608
      java_pool_size           = 8388608
      enqueue_resources        = 8000
      filesystemio_options     = SETALL
      control_files            = C:\oracle\PRD\origlogA\ctrlPRD.ctl, F:\oracle\PRD\saparch\ctrlPRD.ctl, E:\oracle\PRD\sapdata1\ctrlPRD.ctl
      db_block_checksum        = FALSE
      db_block_size            = 8192
      db_cache_size            = 1073741824
      compatible               = 9.2.0
      log_archive_start        = TRUE
      log_archive_dest         = F:\oracle\PRD\saparch
      log_buffer               = 4194304
      log_checkpoint_interval  = 0
      db_files                 = 254
      db_file_multiblock_read_count= 8
      log_checkpoints_to_alert = TRUE
      control_file_record_keep_time= 30
      dml_locks                = 2500
      row_locking              = always
      replication_dependency_tracking= FALSE
      transactions_per_rollback_segment= 20
      transaction_auditing     = FALSE
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      undo_retention           = 43200
      remote_os_authent        = TRUE
      db_domain                = world
      service_names            = PRD.world, PRD.WORLD
      hash_join_enabled        = FALSE
      background_dump_dest     = F:\Oracle\PRD\saptrace\background
      user_dump_dest           = F:\Oracle\PRD\saptrace\usertrace
      max_dump_file_size       = 1240
      core_dump_dest           = F:\Oracle\PRD\saptrace\background
      optimizer_features_enable= 9.2.0
      sort_area_size           = 0
      sort_area_retained_size  = 0
      db_name                  = PRD
      open_cursors             = 800
      optimizer_mode           = CHOOSE
      sortelimination_cost_ratio= 10
      btree_bitmap_plans     = FALSE
      eliminatecommon_subexpr= FALSE
      pushjoin_predicate     = FALSE
      optimizerjoin_sel_sanity_check= TRUE
      parallel_threads_per_cpu = 4
      optimizer_index_cost_adj = 10
      optimizer_index_caching  = 50
      optimpeek_user_binds   = FALSE
      pga_aggregate_target     = 536870912
      optimizeror_expansion  = DEPTH
    PMON started with pid=2, OS id=17336
    DBW0 started with pid=3, OS id=16972
    LGWR started with pid=4, OS id=16900
    CKPT started with pid=5, OS id=17236
    SMON started with pid=6, OS id=17436
    Regards
    Pieter

  • Querying the toplink cache under high-load

    We've had some interesting experiences with "querying" the TopLink Cache lately.
    It was recently discovered that our "read a single object" method was incorrectly
    setting query.checkCacheThenDB() for all ReadObjectQueries. This was brought to light
    when we upgraded our production servers from 4 cores to 8. We immediatly started
    experiencing very long response times under load.
    We traced this down to the following stack: (TopLink version 10.1.3.1.0)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
    at java.lang.Object.wait(Object.java:474)
    at oracle.toplink.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:179)
    - locked <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
    at oracle.toplink.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:167)
    at oracle.toplink.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:122)
    at oracle.toplink.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:31)
    at oracle.toplink.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:530)
    at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:412)
    at oracle.toplink.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:223)
    at oracle.toplink.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:504)
    at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:564)
    at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
    at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
    We moved the query back to the default, query.checkByPrimaryKey() and this issue went away.
    The bottleneck seemed to stem from the read lock on the CacheKey from IdenityMapKeyEnumeration
    public Object nextElement() {
    if (this.nextKey == null) {
    throw new NoSuchElementException("IdentityMapKeyEnumeration nextElement");
    // CR#... Must check the read lock to avoid
    // returning half built objects.
    this.nextKey.checkReadLock();
    return this.nextKey;
    We had many threads that were having contention while searching the cache for a particular query.
    From the stack we know that the contention was limited to one class. We've since refactored that code
    not to use a query in that code path.
    Question:
    Armed with this better knowledge of how TopLink queries the cache, we do have a few objects that we
    frequently read by something other than the primary key. A natural key, but not the oid.
    We have some other caching mechanisms in place (JBoss TreeCache) to help eliminate queries to the DB
    for these objects. But the TreeCache also tries to acquire a read lock when accessing the cache.
    Presumably a read lock over the network to the cluster.
    Is there anything that can be done about the read lock on CacheKey when querying the cache in a high load
    situation?

    CheckCacheThenDatabase will check the entire cache for a match using a linear search. This can be efficient if the cache is very large. Typically it is more efficient to access the database if your cache is large and the field you are querying in the database is indexed in the table.
    The cache concurrency was greatly improved in TopLink 11g/EclipseLink, so you may wish to try it out.
    Supporting indexes in the TopLink/EclipseLink cache is something desirable (feel free to log the enhancement request on EclipseLink). You can simulate this to some degree using a named query and a query cache.
    -- James : http://www.eclipselink.org

  • Replacing / Compiling Database Triggers in a HA/High Load system

    Hi there,
    My collegue has just asked me if downtime needs to be scheduled to replace an After IUD Database trigger where I've made a minor change.
    If this were a package or procedure, the answer would be an obvious yes, as I'd be wary of my users getting the dreaded: ORA-04068... State of Package has been disguarded.
    What about when working with database triggers?
    If the trigger is firing at the same time that I perform my Create or Replace Trigger DDL, will everything fall into a screaming heap?
    Notes:
    Oracle 10g R2 db.
    High DML rates on the table that the After IUD Trigger is 'attached' to.
    It's a near certainty that when the Create or Replace Trigger DDL runs, the trigger will be active at the time.
    The Trigger is 'simple' in that it's use: is to call a DB package with some of the :new values as params.
    +(I'm happy to rtfm - if someone can tell me where!!)+
    Cheers -
    Ron Marks

    If this were a package or procedure, the answer would be an obvious yes, as I'd be wary of my users getting the dreaded: ORA-04068... State of Package has been disguarded.Not true. This exception is thrown only if sessions have some state represented in package variables. If there is no associated session state, then no exception would be thrown after recompile.
    If the trigger is firing at the same time that I perform my Create or Replace Trigger DDL, will everything fall into a screaming heap?You'll wait - to lock an object in library cache which represents your trigger. Not sure if this would require a library cache lock on underlying table, but I would guess that it is required. BTW, locking a table would make no any sense: CREATE TRIGGER is a DDL, which means before it began it issues a COMMIT => all locks are released, and it is quite possible that a TM lock for CREATE TRIGGER won't be acquired since resource would be busy.
    Oracle 11gR2 claims to address problems of application upgrades on a high-load environments with a feature called "Editions" (not a good name - try to find something about it via Google...)

  • High load CPU X process

    Hi,
    I'm using ArchLinux 64bit, with KDE 4.3.4 on a sony vaio, everything work fine, but after some normal work I see that process X takes more then 90% of CPU and never slow down. Even if I closed all my application it stays at 90%. I don't know how to see which application could cause this.
    As first impression it seems when I use samba adn move some file from my machine to a windows machine, but today it takes high load cpu without using samba, so I don't know what else do, any good advice?
    thanks a lot
    cesare

    Hi thank you very much for your reply, I'm sorry to reply just now.
    Anyway I have a Sony Vaio VGN.SR21M, using opensource driver xf86-video-radeonhd with hardware acc on. My graphics card is a Radeon3400HD
    I used the closed driver till I switch to Xorg 1.7 and so I can't using it anymore so I switched to the free one.
    Should I disable desktop effects?
    Thanks again
    Cesare
    Last edited by cesare (2010-01-25 08:48:01)

  • JAXB:  NPE on high load in Coordinator.java:112

    Hello,
    I'm using JAXB 2.0 and when running high load tests I get NPE.
    In my code I have a static marshaller (initialized in a static initializer) and the method which makes use of it is synchronized.
    Once every few (~10) invocations of my method I get:
    java.lang.NullPointerException
    at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:112)
    at com.sun.xml.bind.v2.runtime.XMLSerializer.close(XMLSerializer.java:752)
    at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:272)
    at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:200)
    at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:70)
    Any help...?
    Thanks,
    Zohar.

    Hello,
    I'm using JAXB 2.0 and when running high load tests I get NPE.
    In my code I have a static marshaller (initialized in a static initializer) and the method which makes use of it is synchronized.
    Once every few (~10) invocations of my method I get:
    java.lang.NullPointerException
    at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:112)
    at com.sun.xml.bind.v2.runtime.XMLSerializer.close(XMLSerializer.java:752)
    at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:272)
    at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:200)
    at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:70)
    Any help...?
    Thanks,
    Zohar.

  • Sync messages failed due to high load

    Hi!
    On our test system we had a very high load of async messages for 3 - 4 hours. The async messages were processed successfully but all sync messages failed during this time frame! As soon as the high load of async messages stopped, the sync messages started executing successfully again.
    Has anybody also experienced this kind of problem and what can be done to avoid this issue? Thanks for any input and suggestions.
    Regards, Tanja

    hey Tanja,
    i had a similar issue when testing the box with Load runner, that a simple sync scenario fails when the size of payload or attachment is high, the thing that i did is just increased the virtual memory ,
    You can also check the trace files to figure out problem ?
    This might not be a definitive answer , hope this helps,
    Vara
    [PLEASE REWARD FOR POSTING]

  • High-load sql statement using v$sql

    Hi,
    Can any one please tell me, how do we find high load sql statement and it's user from v$SQL view.
    what is the query to find it.
    Thank you!

    Hello,
    You can run ADDM report and check its findings it will tell you tome stuff like the following:
         Finding
    67      SQL statements consuming significant database time were found.
    40.7      Time spent on the CPU by the instance was responsible for a substantial part of database time.
    20.7      Individual SQL statements responsible for significant user I/O wait were found.
    13.7      Individual database segments responsible for significant user I/O wait were found.Kind regards
    Mohamed Elazab

  • One of 4 node RAC always have higher load averages and higher than others

    Hello,
    We have a 4 node rac, 9208 on linux 4. When viewing top, we noticed the same one node always have a higher load average than the other 3 nodes. Is this normal. Loan balance is working fine but this one node always have higher load average. This is the node where we do the rac installation. Thank you.

    I do not remember what is the default for clb_goal (client load balancing) for 9i but 10g is LONG.
    check it
    select clb_goal from dba_services where name = <service name>
    you may have to change from LONG to SHORT OR SHORT to LONG depending your connection types.
    dbms_service.MODIFY_SERVICE(‘<service>’,clb_goal=> dbms_service.CLB_GOAL_LONG);
    Read the following article.
    http://www.databasejournal.com/features/oracle/article.php/3659411/Oracle-RAC-Administration---Part-15-Connection-Load-Balancing-and-FAN.htm

  • I am planning to buy an ipad mini wifi 16gb because i really want a tablet with high graphics and high loaded games support so is ipad mini wifi 16gb best for gaming ? because i own a samsung galaxy tab 3 and it is worst even it lag and stucks !

    is ipad mini is capable to run high graphics and high loaded games with no lag and will not stuck while playing high graphics / loaded game ?
    tell me i want your suggestions !

    is ipad mini is capable to run high graphics and high loaded games with no lag and will not stuck while playing high graphics / loaded game ?
    tell me i want your suggestions !

  • Performance Bottleneck at HIgh Loads

    Hi,
    We are currently load testing our web application and have been facing an issue that has ben bothering us for sometime now.
    If we load test the application at high load for sustained amounts of time, (around 2 hours), the performance suddenly dips to Zero hits/sec, Nothing is processed , no output generated. It seems as though all the processes some how are waiting for a resource to get free.After sometime the performance goes up again and again there is a race condition because of which the performance goes down again. Here on the dipping becomes highly erratic.
    We are trying to find out if this is a known bug with the oracle jdbc driver. "oracle.jdbc.driver.OracleDriver" .
    Would appreciate if anybody could provide any inputs as to why this is happening and how we could alleviate the situation.
    Thanks in advance.
    Deepanjan

    Hi I vaguely remember something like this.... but I thought it typically occurred during lots of heavy insert/update activity
    which version of the jdbc drivers are you using.... are you using thin/thick
    Have you checked the SQL or the database to determine whether any locks are blocking activity or whether you have large database wait issues....
    Dom

  • Toomany files open on high load

    We have a app with MDB's and stateles EJB's
    Under lower load it works fine. When the load increases, we get the following exception
    on the weblogic trace.
    java.io.FileNotFoundException: ./config.xml (Too many open files)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(Unknown Source)
    at weblogic.management.internal.AdminMBeanHomeImpl.getDigest(AdminMBeanHomeImpl.java:846)
    at weblogic.management.internal.AdminMBeanHomeImpl.updateManagedServerMSIFile(AdminMBeanHomeImpl.java:754)
    at weblogic.management.internal.SaveDomainTrigger.trigger(SaveDomainTrigger.java:55)
    at weblogic.time.common.internal.ScheduledTrigger$1.run(ScheduledTrigger.java:174)
    at weblogic.security.service.SecurityServiceManager.runAs(SecurityServiceManager.java:721)
    at weblogic.time.common.internal.ScheduledTrigger.executeLocally(ScheduledTrigger.java:171)
    at weblogic.time.common.internal.ScheduledTrigger.execute(ScheduledTrigger.java:161)
    at weblogic.time.server.ScheduledTrigger.execute(ScheduledTrigger.java:38)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:152)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:133)
    <Jul 12, 2002 1:26:27 PM PDT> <Error> <kernel> <000802> <ExecuteRequest failed
    java.lang.NullPointerException
    java.lang.NullPointerException
    at weblogic.security.MessageDigestUtils.isEqual(MessageDigestUtils.java:42)
    at weblogic.management.internal.AdminMBeanHomeImpl.updateManagedServerMSIFile(AdminMBeanHomeImpl.java:755)
    at weblogic.management.internal.SaveDomainTrigger.trigger(SaveDomainTrigger.java:55)
    at weblogic.time.common.internal.ScheduledTrigger$1.run(ScheduledTrigger.java:174)
    at weblogic.security.service.SecurityServiceManager.runAs(SecurityServiceManager.java:721)
    at weblogic.time.common.internal.ScheduledTrigger.executeLocally(ScheduledTrigger.java:171)
    at weblogic.time.common.internal.ScheduledTrigger.execute(ScheduledTrigger.java:161)
    at weblogic.time.server.ScheduledTrigger.execute(ScheduledTrigger.java:38)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:152)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:133)
    We are using HP-UX 11.0 and the value for ulimit is unlimited

    Hi Everyone,
    Just thought I would give an update on this.
    While we're still deploying http://support.microsoft.com/kb/2733363/en-us to
    the remainder of our Windows 7 SP1 fleet, according to some Network Traces and XPerf that were sent to MS Support, it looks as though the issu with the Offline Files is now resolved.
    However, we are still having some issues with High CPU with the System Process in particular. Upon review of the traces, they have found a lot of ABE related traffic, leading them to believe that the issue may also be caused by ABE Access on our File Cluster.
    We already have the required hotfix for ABE on 2008 R2 installed (please see
    http://support.microsoft.com/kb/2732618/en-us), although we have it set to a certain value that MS believe may be too high. Our current value is set to "5", as that is the lowest level of any type of permission is set (i.e. where the lowest level of inheritance
    is broken).
    I think I will mark this as resolved regarding the Offline Files issue (as it did resolve the problem with the Offline Files)...
    Fingers crossed we can identify the rest of the high load within the System Process!!!

Maybe you are looking for

  • AE CS6 crashes while trying to open CS5 project

    First issue that comes upon opening a particular CS5 project into CS6 is that DE_FreeForm plug-in is notified as missing. Ok. Then tries to load a comp and crashes. Project opens fine in CS5. I removed DE_FreeFormAE, will not open in CS6. Help please

  • Issues in using LSMW IDOC method for Article Hierarchy

    Hi Experts, I am working on a scenario for uploading article hierarchy using LSMW idoc method. Details:           Message type WMATGRP           IDoc type WMATGRP01 But as per my understanig, for processing, there should be a inbound Function module/

  • How to insert current_user to report?

    Hi, how to insert current_user to report (current_user is user who login to BI ) Edited by: user10236405 on Oct 24, 2008 3:11 AM

  • Sold to Party and Ship to Party Function module or BAPI

    Hi Experts,    1) I need to create customer data,So i need to Function module or BAPI.    2) Address create data Function module or BAPI.    3) Sold to Party and Ship to Party Function module or BAPI relation ship. Thank you , Dinesh Reddy Moderator

  • Discovery Warnings grid control 11g

    Hi All I have installed a new Windows 2008R2 server, installed 11gR1 agent + psu 4 Plus grid infrastructure. When trying to discover the target I get Discovery Warnings - IOException in sending Request :: Connection establishment timed out The emoms.