Performance Bottleneck at HIgh Loads

Hi,
We are currently load testing our web application and have been facing an issue that has ben bothering us for sometime now.
If we load test the application at high load for sustained amounts of time, (around 2 hours), the performance suddenly dips to Zero hits/sec, Nothing is processed , no output generated. It seems as though all the processes some how are waiting for a resource to get free.After sometime the performance goes up again and again there is a race condition because of which the performance goes down again. Here on the dipping becomes highly erratic.
We are trying to find out if this is a known bug with the oracle jdbc driver. "oracle.jdbc.driver.OracleDriver" .
Would appreciate if anybody could provide any inputs as to why this is happening and how we could alleviate the situation.
Thanks in advance.
Deepanjan

Hi I vaguely remember something like this.... but I thought it typically occurred during lots of heavy insert/update activity
which version of the jdbc drivers are you using.... are you using thin/thick
Have you checked the SQL or the database to determine whether any locks are blocking activity or whether you have large database wait issues....
Dom

Similar Messages

  • 2008 R2/ Win7 Offline Files Sync causing High Load on server

    Hi Everyone,
    I have recently been investigating extremely high CPU Usage from the System Process on my company's main File Cluster.
    We managed to track SRV2.sys Threads causing high CPU load within the system process, but was having issues identifying why this was the case.
    As per Microsoft's direction via support call, we have installed the latest SRV2.sys Hotfixes, but this does not appear to have allivated the issue we are experiencing. We have also added more CPU and Memory into both nodes, which has not helped either.
    We have since managed to create a system dump and is being sent to MS Support for analysis.
    I have noticed the following that appears to happen on our cluster:
    Whenever our CAD/Design department run certain functions within their apps running on a windows 7 client (apps include MicroStation, Revit, AutoCAD etc) we see a massive spike and flatline of the system process.
    We found several users with Windows 7 clients that have Configured Offline File to sync an entire Network Volume (some volumes are 2TB Plus, so would never fit on a users computer anyway, i was quite shocked when I found this). How we spotted this was through
    Resource Monitor, the System Process was trolling through all the folders and files in a given volume (it was "reading every single folder). Now, while this was the system process, we could identify the user by using the Open Files view in Server Manager's
    Share and Storage Management tool.
    I have done a fair bit of research and found that a lot of CAD/Drawing applications in the market have issues with using SMB2 (srv2.sys). When reviewing the products that we use, I noticed that a lot of these products actually recommended disabling SMB2
    and reverting to SMB1 on File Server and/or the clients.
    I have gone down the path of disabling SMB2 on all Windows 7 clients that have these CAD Applications installed to assist with lowering the load (our other options are to potentially shift the CAD Volumes off our main file cluster to further isolation these
    performance issues we have been experiencing.) We will be testing this again tomorrow to confirm that the issue is resolved when mass amounts of CAD users access data on these volumes via their CAD Application.
    We only noticed the issue with Offline Files today with trying to sync an ENTIRE Volume. My questions are:
    Should Offline File sync's really cause this much load on a File Server?
    Would the the size of the volume the sync is trying to complete create this additional load within the system process?
    What is the industry considered "Best Practice" regarding Offline Files setup for Volumes which could have up to 1000+ users attached? (My personal opinion is that Offline Files should only be sync of user "Personal/Home" Folders
    as they users themselves have a 1 to 1 relationship with this data.)
    Is there an easier way to identify what users have Offline Files enabled and "actually being used" (from my understanding, Sync Center and Offline Files are enabled by default, but you obviously have to add the folders/drives you wish to sync)?
    If I disable the ability for Offline Files on the volumes, what will the user experience be when/if they try to sync their offline files config after this has been disabled on the volume?
    Hoping for some guidance regarding this setup with Offline Files.
    Thanks in Advance!
    Simon

    Hi Everyone,
    Just thought I would give an update on this.
    While we're still deploying http://support.microsoft.com/kb/2733363/en-us to
    the remainder of our Windows 7 SP1 fleet, according to some Network Traces and XPerf that were sent to MS Support, it looks as though the issu with the Offline Files is now resolved.
    However, we are still having some issues with High CPU with the System Process in particular. Upon review of the traces, they have found a lot of ABE related traffic, leading them to believe that the issue may also be caused by ABE Access on our File Cluster.
    We already have the required hotfix for ABE on 2008 R2 installed (please see
    http://support.microsoft.com/kb/2732618/en-us), although we have it set to a certain value that MS believe may be too high. Our current value is set to "5", as that is the lowest level of any type of permission is set (i.e. where the lowest level of inheritance
    is broken).
    I think I will mark this as resolved regarding the Offline Files issue (as it did resolve the problem with the Offline Files)...
    Fingers crossed we can identify the rest of the high load within the System Process!!!

  • Strange Oracle Behaviour - only month end or high load periods

    Hi SAP Gurus
    In one of our systems I am expreriencing a strange behaviour from Oracle, in that it seems to be in reconnect, but yet it is not, it resets and carries on but do produce Dump and SM21 log messages, the dump is not really helpfull as it is the same dump but with different program everytime.
    I have looked in oracle alert log and found no event tying back to the SM21 log message or dump.
    The following is what I found in the developer trace of one of these insident, but there are many, about 125 - 200 for the day during high load:
    Fri Nov 28 10:05:55 2008
    C  *** ERROR => Failed to register application info: orc=3113
    [dboci.c      3042]
    C  *** ERROR => Error 3114 in stmt_fetch()
    [dbsloci.c    13001]
    C  *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
    [dbsloci.c    13002]
    C  sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB                       
    C  SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
    C  NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
    C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
    C  "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
    C  GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
    C  INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
    C  ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
    C  TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
    C  DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
    C  PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
    C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
    C  SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
    C  FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
    C  sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB                       
    C  prep=0,lit=0,nsql=0,lobret=0,#exec=1,dbcnt=0,upsh_p=00000000,ocistmth_p=345E20EC
    C  IN : cols=4,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=17,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=3470A8D8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  OUT: cols=105,rmax=58,xcnt=58,rpc=1,rowi=0,rtot=1,upto=-1,rsize=895,vmax=128,bound=1,iobuf_p=0E0A4EC8,vda_p=34F8F7E8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
    C  NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
    C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
    C  "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
    C  GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
    C  INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
    C  ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
    C  TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
    C  DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
    C  PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
    C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
    C  SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
    C  FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  ***LOG BY4=> sql error 3114   performing FET on table CATSDB     [dbtran#10 @ 7292] [dbtran  7292 ]
    B  ***LOG BY0=> ORA-03114: not connected to ORACLE [dbtran#10 @ 7292] [dbtran  7292 ]
    B  dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
    B   RSLT: {dbsl=99, tran=1}
    B   FHDR: {tab='CATSDB', fcode=225, mode=2, bpb=0, dbcnt=0, crsr=1,
    B          hold=0, keep=1, xfer=0, pkg=0, upto=0, init:b=0,
    B          init:p=00000000, init:#=871, wa:p=0XA000DC88, wa:#=871}
    B  dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
    B   STMT:
    B   CRSR: {tab='CATSDB', id=1, hold=0, prop=0x1000, max.in@0=1, fae:blk=1,
    B          con:id=0, con:vndr=5, val=2,
    B          key:#=4, xfer=0, xin:#=0, row:#=0, upto=0, wa:p=0XA000DC88}
    C  *** ERROR => Error 3114 in stmt_fetch()
    [dbsloci.c    13001]
    C  *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
    [dbsloci.c    13002]
    C  sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD                     
    C  SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
    C  R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
    C  sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD                     
    C  prep=0,lit=0,nsql=0,lobret=0,#exec=29,dbcnt=0,upsh_p=00000000,ocistmth_p=34B9ADD4
    C  IN : cols=3,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=45,vmax=32,bound=1,iobuf_p=0E0A4EC8,vda_p=31361B18
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  OUT: cols=9,rmax=1,xcnt=1,rpc=1,rowi=0,rtot=1,upto=-1,rsize=56,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=345C39B8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
    C  R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  ***LOG BY4=> sql error 3114   performing SEL on table REPOLOAD   [dbrepo#6 @ 2657] [dbrepo  2657 ]
    B  ***LOG BY0=> ORA-03114: not connected to ORACLE [dbrepo#6 @ 2657] [dbrepo  2657 ]
    A  TH VERBOSE LEVEL FULL
    A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.
    A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.
    A  ** RABAX: level LEV_RX_VMIT_CLEANUP entered.
    A  ** RABAX: level LEV_RX_VMIT_CLEANUP completed.
    A  ** RABAX: level LEV_RX_ROLLBACK entered.
    C  *** ERROR => OCI-call 'OCITransRollback' failed: rc = 3114
    [dboci.c      3127]
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  Reconnect state is entered by connection:
    B  0: name = R/3, con_id = 000000000 state = ACTIVE      , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    B  hdl_error_on_commit_rollback: DB-ROLLBACK detected RECONNECT state
    M  ThShortCommit: db unusable
    B  db_con_reconnect disconnecting connection:
    B  0: name = R/3, con_id = 000000000 state = INACTIVE    , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    C  Disconnecting from connection 0 ...
    C  Close user session (con_hdl=0,svchp=045D7FA4,usrhp=345E67A0)
    C  Detaching from DB Server (con_hdl=0,svchp=045D7FA4,srvhp=045D0174)
    C  Now I'm disconnected from ORACLE
    B  db_con_reconnect performing the reconnect for con:
    B  0: name = R/3, con_id = 000000000 state = DISCONNECTED, perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    C  got NLS_LANG='AMERICAN_AMERICA.US7ASCII' from environment
    C  Client NLS settings:
    C  Logon as OPS$-user to get SAPR3's password
    C  Connecting as /@PRD on connection 0 (nls_hdl 0) ... (dbsl 640 250407)
    C  Nls CharacterSet                 NationalCharSet              C      EnvHp      ErrHp ErrHpBatch
    C    0 US7ASCII                                                  1   045CBB38   0E0A4648   0E0A40B0
    C  Attaching to DB Server PRD (con_hdl=0,svchp=045
    SM51 log entries:
         09:00:32     MS                         Q0I     Operating system call recv failed (error no. 10054)
         09:00:32     DP                         Q0K     Connection to message server (on FCMSPROD) established
         09:00:32     DIA     004     040     181JOLENES     VA02     BYM     SQL error 3114 . Work processes in reconnect status
         09:00:32     DP                         Q1C     MsgServer Hardware ID Was Determined
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     BYY     Work process has left reconnect status
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:32     DIA     014     040     161ADRIV     VA02     BYY     Work process has left reconnect status
         09:00:32     DIA     013     040     140ELECIA          BYY     Work process has left reconnect status
         09:00:32     DIA     003     040     133MIRANDAV     KO88     BYY     Work process has left reconnect status
         09:00:32     DIA     003     040     133MIRANDAV     KO88     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB1     > Short dump "081128 090032 FCMSPROD 181ANNELINEJ" generated
         09:00:32     DIA     003     040     133MIRANDAV     KO88     AB1     > Short dump "081128 090032 FCMSPROD 133MIRANDAV " generated
         09:00:32     DIA     009     040     120ELIZEP     FB01     BYY     Work process has left reconnect status
         09:00:33     DIA     009     040     120ELIZEP     FB01     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     004     040     181JOLENES     VA02     BYY     Work process has left reconnect status
         09:00:33     DIA     002     040     133KOEKIEJ          BYY     Work process has left reconnect status
         09:00:33     DIA     002     040     133KOEKIEJ          AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     004     040     181JOLENES     VA02     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     009     040     120ELIZEP     FB01     AB1     > Short dump "081128 090033 FCMSPROD 120ELIZEP " generated
         09:00:33     DIA     005     040     181CLIVEB     VA02     BYY     Work process has left reconnect status
         09:00:33     DIA     003                    R0Z     The update dispatch info was reset
         09:00:33     DIA     003                    R0Z     The update dispatch info was reset
    Please can you assist in maybe you have seen this before.
    Regards
    Pieter

    Hi
    Yes I have gone through the Oracle Alert log several time, It is huge, but I am pasting an extract for the entire day this was happening into this reply.  Oracle version is 9.2.0.8.0 with the only intrim path I can see being installed : 6130293.
    Following is the extract from Alert Log for the day this occurred, again I must stress that this happens only on high system load days like month end:
    Fri Nov 28 03:43:33 2008
    ALTER TABLESPACE "PSAPBTABI" END BACKUP
    Fri Nov 28 03:43:33 2008
    Completed: ALTER TABLESPACE "PSAPBTABI" END BACKUP
    Fri Nov 28 03:43:37 2008
    ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
    Fri Nov 28 03:53:38 2008
    ALTER TABLESPACE "PSAPCLUD" END BACKUP
    Fri Nov 28 03:53:38 2008
    Completed: ALTER TABLESPACE "PSAPCLUD" END BACKUP
    Fri Nov 28 03:53:42 2008
    ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
    Fri Nov 28 03:54:36 2008
    ALTER TABLESPACE "PSAPCLUI" END BACKUP
    Fri Nov 28 03:54:36 2008
    Completed: ALTER TABLESPACE "PSAPCLUI" END BACKUP
    Fri Nov 28 03:54:38 2008
    ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
    Fri Nov 28 03:55:47 2008
    ALTER TABLESPACE "PSAPDDICD" END BACKUP
    Fri Nov 28 03:55:47 2008
    Completed: ALTER TABLESPACE "PSAPDDICD" END BACKUP
    Fri Nov 28 03:55:50 2008
    ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
    Fri Nov 28 03:56:41 2008
    ALTER TABLESPACE "PSAPDDICI" END BACKUP
    Fri Nov 28 03:56:41 2008
    Completed: ALTER TABLESPACE "PSAPDDICI" END BACKUP
    Fri Nov 28 03:56:45 2008
    ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
    Fri Nov 28 03:56:49 2008
    ALTER TABLESPACE "PSAPDOCUD" END BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUD" END BACKUP
    Fri Nov 28 03:56:52 2008
    ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
    Fri Nov 28 03:56:55 2008
    ALTER TABLESPACE "PSAPDOCUI" END BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUI" END BACKUP
    Fri Nov 28 03:56:59 2008
    ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
    Fri Nov 28 04:03:46 2008
    ALTER TABLESPACE "PSAPEL640D" END BACKUP
    Fri Nov 28 04:03:46 2008
    Completed: ALTER TABLESPACE "PSAPEL640D" END BACKUP
    Fri Nov 28 04:03:50 2008
    ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
    Fri Nov 28 04:04:37 2008
    ALTER TABLESPACE "PSAPEL640I" END BACKUP
    Fri Nov 28 04:04:37 2008
    Completed: ALTER TABLESPACE "PSAPEL640I" END BACKUP
    Fri Nov 28 04:04:40 2008
    ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
    Fri Nov 28 04:20:17 2008
    ALTER TABLESPACE "PSAPES640D" END BACKUP
    Fri Nov 28 04:20:17 2008
    Completed: ALTER TABLESPACE "PSAPES640D" END BACKUP
    Fri Nov 28 04:20:20 2008
    ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
    Fri Nov 28 04:36:26 2008
    ALTER TABLESPACE "PSAPES640I" END BACKUP
    Fri Nov 28 04:36:26 2008
    Completed: ALTER TABLESPACE "PSAPES640I" END BACKUP
    Fri Nov 28 04:36:29 2008
    ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
    Fri Nov 28 04:36:40 2008
    ALTER TABLESPACE "PSAPLOADD" END BACKUP
    Fri Nov 28 04:36:40 2008
    Completed: ALTER TABLESPACE "PSAPLOADD" END BACKUP
    Fri Nov 28 04:36:43 2008
    ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
    Fri Nov 28 04:36:53 2008
    ALTER TABLESPACE "PSAPLOADI" END BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADI" END BACKUP
    Fri Nov 28 04:36:56 2008
    ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
    Fri Nov 28 04:41:04 2008
    ALTER TABLESPACE "PSAPPOOLD" END BACKUP
    Fri Nov 28 04:41:04 2008
    Completed: ALTER TABLESPACE "PSAPPOOLD" END BACKUP
    Fri Nov 28 04:41:08 2008
    ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
    Fri Nov 28 04:45:17 2008
    ALTER TABLESPACE "PSAPPOOLI" END BACKUP
    Fri Nov 28 04:45:17 2008
    Completed: ALTER TABLESPACE "PSAPPOOLI" END BACKUP
    Fri Nov 28 04:45:21 2008
    ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
    Fri Nov 28 04:47:25 2008
    ALTER TABLESPACE "PSAPPROTD" END BACKUP
    Fri Nov 28 04:47:25 2008
    Completed: ALTER TABLESPACE "PSAPPROTD" END BACKUP
    Fri Nov 28 04:47:29 2008
    ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
    Fri Nov 28 04:48:09 2008
    ALTER TABLESPACE "PSAPPROTI" END BACKUP
    Fri Nov 28 04:48:09 2008
    Completed: ALTER TABLESPACE "PSAPPROTI" END BACKUP
    Fri Nov 28 04:48:13 2008
    ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
    Fri Nov 28 04:48:46 2008
    ALTER TABLESPACE "PSAPSOURCED" END BACKUP
    Fri Nov 28 04:48:46 2008
    Completed: ALTER TABLESPACE "PSAPSOURCED" END BACKUP
    Fri Nov 28 04:48:49 2008
    ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
    Fri Nov 28 04:49:29 2008
    ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
    Fri Nov 28 04:49:29 2008
    Completed: ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
    Fri Nov 28 04:49:34 2008
    ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
    Fri Nov 28 05:05:16 2008
    ALTER TABLESPACE "PSAPSTABD" END BACKUP
    Fri Nov 28 05:05:16 2008
    Completed: ALTER TABLESPACE "PSAPSTABD" END BACKUP
    Fri Nov 28 05:05:23 2008
    ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
    Fri Nov 28 05:21:15 2008
    ALTER TABLESPACE "PSAPSTABI" END BACKUP
    Fri Nov 28 05:21:15 2008
    Completed: ALTER TABLESPACE "PSAPSTABI" END BACKUP
    Fri Nov 28 05:21:19 2008
    ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
    Fri Nov 28 05:21:42 2008
    ALTER TABLESPACE "PSAPUSER1D" END BACKUP
    Fri Nov 28 05:21:42 2008
    Completed: ALTER TABLESPACE "PSAPUSER1D" END BACKUP
    Fri Nov 28 05:21:46 2008
    ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
    Fri Nov 28 05:23:03 2008
    ALTER TABLESPACE "PSAPUSER1I" END BACKUP
    Fri Nov 28 05:23:03 2008
    Completed: ALTER TABLESPACE "PSAPUSER1I" END BACKUP
    Fri Nov 28 05:23:06 2008
    ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
    Completed: ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
    Fri Nov 28 05:24:03 2008
    ALTER TABLESPACE "SYSTEM" END BACKUP
    Fri Nov 28 05:24:03 2008
    Completed: ALTER TABLESPACE "SYSTEM" END BACKUP
    Fri Nov 28 05:24:07 2008
    ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
    Completed: ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
    Fri Nov 28 05:42:06 2008
    ALTER TABLESPACE "UNDOTBS1" END BACKUP
    Fri Nov 28 05:42:06 2008
    Completed: ALTER TABLESPACE "UNDOTBS1" END BACKUP
    Fri Nov 28 05:42:07 2008
    ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program Files\CA\BrightStor ARCserve Backup Enterprise Option for SAP R3 for Oracle\CONTROL.PRD' REUSE
    Completed: ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program F
    Fri Nov 28 05:42:14 2008
    Beginning log switch checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
    Thread 1 advanced to log sequence 30706
      Current log# 2 seq# 30706 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30706 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 05:42:14 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30705
    ARC0: Beginning to archive log 3 thread 1 sequence 30705
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30705.001'
    Fri Nov 28 05:42:27 2008
    ARC1: Evaluating archive   log 3 thread 1 sequence 30705
    ARC1: Unable to archive log 3 thread 1 sequence 30705
          Log actively being archived by another process
    Fri Nov 28 05:42:29 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30705
    Fri Nov 28 06:11:26 2008
    Completed checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
    Fri Nov 28 09:20:59 2008
    Beginning log switch checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
    Thread 1 advanced to log sequence 30707
      Current log# 1 seq# 30707 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
      Current log# 1 seq# 30707 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
    Fri Nov 28 09:20:59 2008
    ARC0: Evaluating archive   log 2 thread 1 sequence 30706
    ARC0: Beginning to archive log 2 thread 1 sequence 30706
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30706.001'
    Fri Nov 28 09:21:18 2008
    ARC0: Completed archiving  log 2 thread 1 sequence 30706
    Fri Nov 28 09:51:06 2008
    Completed checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
    Fri Nov 28 10:44:43 2008
    Beginning log switch checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
    Thread 1 advanced to log sequence 30708
      Current log# 4 seq# 30708 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
      Current log# 4 seq# 30708 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
    Fri Nov 28 10:44:43 2008
    ARC0: Evaluating archive   log 1 thread 1 sequence 30707
    ARC0: Beginning to archive log 1 thread 1 sequence 30707
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30707.001'
    Fri Nov 28 10:45:07 2008
    ARC1: Evaluating archive   log 1 thread 1 sequence 30707
    ARC1: Unable to archive log 1 thread 1 sequence 30707
          Log actively being archived by another process
    Fri Nov 28 10:45:09 2008
    ARC0: Completed archiving  log 1 thread 1 sequence 30707
    Fri Nov 28 11:13:20 2008
    Completed checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
    Fri Nov 28 11:45:09 2008
    Beginning log switch checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
    Thread 1 advanced to log sequence 30709
      Current log# 3 seq# 30709 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
      Current log# 3 seq# 30709 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
    Fri Nov 28 11:45:09 2008
    ARC0: Evaluating archive   log 4 thread 1 sequence 30708
    ARC0: Beginning to archive log 4 thread 1 sequence 30708
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30708.001'
    Fri Nov 28 11:45:13 2008
    ARC1: Evaluating archive   log 4 thread 1 sequence 30708
    ARC1: Unable to archive log 4 thread 1 sequence 30708
          Log actively being archived by another process
    Fri Nov 28 11:45:29 2008
    ARC0: Completed archiving  log 4 thread 1 sequence 30708
    Fri Nov 28 12:15:17 2008
    Completed checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
    Fri Nov 28 12:50:23 2008
    Beginning log switch checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
    Thread 1 advanced to log sequence 30710
      Current log# 2 seq# 30710 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30710 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 12:50:24 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30709
    ARC0: Beginning to archive log 3 thread 1 sequence 30709
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30709.001'
    Fri Nov 28 12:50:40 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30709
    Fri Nov 28 13:20:29 2008
    Completed checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
    Fri Nov 28 14:27:01 2008
    Beginning log switch checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
    Thread 1 advanced to log sequence 30711
      Current log# 1 seq# 30711 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
      Current log# 1 seq# 30711 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
    Fri Nov 28 14:27:01 2008
    ARC0: Evaluating archive   log 2 thread 1 sequence 30710
    ARC0: Beginning to archive log 2 thread 1 sequence 30710
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30710.001'
    Fri Nov 28 14:27:25 2008
    ARC0: Completed archiving  log 2 thread 1 sequence 30710
    Fri Nov 28 14:57:08 2008
    Completed checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
    Fri Nov 28 15:57:26 2008
    Beginning log switch checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
    Thread 1 advanced to log sequence 30712
      Current log# 4 seq# 30712 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
      Current log# 4 seq# 30712 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
    Fri Nov 28 15:57:26 2008
    ARC0: Evaluating archive   log 1 thread 1 sequence 30711
    ARC0: Beginning to archive log 1 thread 1 sequence 30711
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30711.001'
    Fri Nov 28 15:57:40 2008
    ARC1: Evaluating archive   log 1 thread 1 sequence 30711
    ARC1: Unable to archive log 1 thread 1 sequence 30711
          Log actively being archived by another process
    Fri Nov 28 15:57:40 2008
    ARC0: Completed archiving  log 1 thread 1 sequence 30711
    Fri Nov 28 16:27:31 2008
    Completed checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
    Fri Nov 28 17:02:12 2008
    Beginning log switch checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
    Thread 1 advanced to log sequence 30713
      Current log# 3 seq# 30713 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
      Current log# 3 seq# 30713 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
    Fri Nov 28 17:02:12 2008
    ARC0: Evaluating archive   log 4 thread 1 sequence 30712
    ARC0: Beginning to archive log 4 thread 1 sequence 30712
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30712.001'
    Fri Nov 28 17:02:25 2008
    ARC0: Completed archiving  log 4 thread 1 sequence 30712
    Fri Nov 28 17:32:17 2008
    Completed checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
    Fri Nov 28 18:00:19 2008
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
    Fri Nov 28 18:00:20 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
    Fri Nov 28 18:00:21 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
    Fri Nov 28 18:00:24 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
    Fri Nov 28 18:00:26 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Fri Nov 28 20:00:17 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Fri Nov 28 20:00:36 2008
    Beginning log switch checkpoint up to RBA [0x77fa.2.10], SCN: 0x0000.3c0cf8e3
    Thread 1 advanced to log sequence 30714
      Current log# 2 seq# 30714 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30714 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 20:00:36 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30713
    ARC0: Beginning to archive log 3 thread 1 sequence 30713
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30713.001'
    Fri Nov 28 20:00:44 2008
    ARC1: Evaluating archive   log 3 thread 1 sequence 30713
    ARC1: Unable to archive log 3 thread 1 sequence 30713
          Log actively being archived by another process
    Fri Nov 28 20:00:47 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30713
    Dump file f:\oracle\prd\saptrace\background\alert_prd.log
    Mon Dec 01 09:03:16 2008
    ORACLE V9.2.0.8.0 - Production vsnsta=0
    vsnsql=12 vsnxtr=3
    Windows 2000 Version 5.2 Service Pack 2, CPU type 586
    Mon Dec 01 09:03:16 2008
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    SCN scheme 2
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up ORACLE RDBMS Version: 9.2.0.8.0.
    System parameters with non-default values:
      processes                = 120
      sessions                 = 150
      timed_statistics         = TRUE
      shared_pool_size         = 822083584
      large_pool_size          = 8388608
      java_pool_size           = 8388608
      enqueue_resources        = 8000
      filesystemio_options     = SETALL
      control_files            = C:\oracle\PRD\origlogA\ctrlPRD.ctl, F:\oracle\PRD\saparch\ctrlPRD.ctl, E:\oracle\PRD\sapdata1\ctrlPRD.ctl
      db_block_checksum        = FALSE
      db_block_size            = 8192
      db_cache_size            = 1073741824
      compatible               = 9.2.0
      log_archive_start        = TRUE
      log_archive_dest         = F:\oracle\PRD\saparch
      log_buffer               = 4194304
      log_checkpoint_interval  = 0
      db_files                 = 254
      db_file_multiblock_read_count= 8
      log_checkpoints_to_alert = TRUE
      control_file_record_keep_time= 30
      dml_locks                = 2500
      row_locking              = always
      replication_dependency_tracking= FALSE
      transactions_per_rollback_segment= 20
      transaction_auditing     = FALSE
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      undo_retention           = 43200
      remote_os_authent        = TRUE
      db_domain                = world
      service_names            = PRD.world, PRD.WORLD
      hash_join_enabled        = FALSE
      background_dump_dest     = F:\Oracle\PRD\saptrace\background
      user_dump_dest           = F:\Oracle\PRD\saptrace\usertrace
      max_dump_file_size       = 1240
      core_dump_dest           = F:\Oracle\PRD\saptrace\background
      optimizer_features_enable= 9.2.0
      sort_area_size           = 0
      sort_area_retained_size  = 0
      db_name                  = PRD
      open_cursors             = 800
      optimizer_mode           = CHOOSE
      sortelimination_cost_ratio= 10
      btree_bitmap_plans     = FALSE
      eliminatecommon_subexpr= FALSE
      pushjoin_predicate     = FALSE
      optimizerjoin_sel_sanity_check= TRUE
      parallel_threads_per_cpu = 4
      optimizer_index_cost_adj = 10
      optimizer_index_caching  = 50
      optimpeek_user_binds   = FALSE
      pga_aggregate_target     = 536870912
      optimizeror_expansion  = DEPTH
    PMON started with pid=2, OS id=17336
    DBW0 started with pid=3, OS id=16972
    LGWR started with pid=4, OS id=16900
    CKPT started with pid=5, OS id=17236
    SMON started with pid=6, OS id=17436
    Regards
    Pieter

  • Querying the toplink cache under high-load

    We've had some interesting experiences with "querying" the TopLink Cache lately.
    It was recently discovered that our "read a single object" method was incorrectly
    setting query.checkCacheThenDB() for all ReadObjectQueries. This was brought to light
    when we upgraded our production servers from 4 cores to 8. We immediatly started
    experiencing very long response times under load.
    We traced this down to the following stack: (TopLink version 10.1.3.1.0)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
    at java.lang.Object.wait(Object.java:474)
    at oracle.toplink.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:179)
    - locked <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
    at oracle.toplink.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:167)
    at oracle.toplink.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:122)
    at oracle.toplink.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:31)
    at oracle.toplink.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:530)
    at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:412)
    at oracle.toplink.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:223)
    at oracle.toplink.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:504)
    at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:564)
    at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
    at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
    We moved the query back to the default, query.checkByPrimaryKey() and this issue went away.
    The bottleneck seemed to stem from the read lock on the CacheKey from IdenityMapKeyEnumeration
    public Object nextElement() {
    if (this.nextKey == null) {
    throw new NoSuchElementException("IdentityMapKeyEnumeration nextElement");
    // CR#... Must check the read lock to avoid
    // returning half built objects.
    this.nextKey.checkReadLock();
    return this.nextKey;
    We had many threads that were having contention while searching the cache for a particular query.
    From the stack we know that the contention was limited to one class. We've since refactored that code
    not to use a query in that code path.
    Question:
    Armed with this better knowledge of how TopLink queries the cache, we do have a few objects that we
    frequently read by something other than the primary key. A natural key, but not the oid.
    We have some other caching mechanisms in place (JBoss TreeCache) to help eliminate queries to the DB
    for these objects. But the TreeCache also tries to acquire a read lock when accessing the cache.
    Presumably a read lock over the network to the cluster.
    Is there anything that can be done about the read lock on CacheKey when querying the cache in a high load
    situation?

    CheckCacheThenDatabase will check the entire cache for a match using a linear search. This can be efficient if the cache is very large. Typically it is more efficient to access the database if your cache is large and the field you are querying in the database is indexed in the table.
    The cache concurrency was greatly improved in TopLink 11g/EclipseLink, so you may wish to try it out.
    Supporting indexes in the TopLink/EclipseLink cache is something desirable (feel free to log the enhancement request on EclipseLink). You can simulate this to some degree using a named query and a query cache.
    -- James : http://www.eclipselink.org

  • OWB Performance Bottleneck

    Is there any session log that is produced by the OWB mapping execution other than seeing the results in OWB Runtime Audit Browser.
    Suppose that the mapping is doing some hash join which is consuming too much amount of time and I would like to see which are the tables that are being joined at that instant. This would help me in identifying the exact area of the problem in a mapping. Does OWB provide a session log which can help me get that information, or from any other place where I can get some information regarding the operation which is causing a performance bottleneck
    regards
    -AP

    Thanks for all your suggestions. The mapping was using a join between some 4 - 5 tables and I think this was the place the mapping was getting stuck during execution in Set Based Mode. Moreover the mapping loads some 70 million records into the target table. Perhaps, loading such huge volume of data and that too in a set based mode and also with a massive join in the beginning, mapping should have got stuck somwhere.
    The solution that came up was to create a table with the join condition and use the table as input to the mapping. This helps us to get rid of the joiner in the very beginning and also the mapping be run in Row Based Target Only mode. The data (70 million) got loaded in some 4 hours.
    regards
    -AP

  • Will RAC's performance bottleneck be the shared disk storage ?

    Hi All
    I'm studying RAC and I'm concerned about RAC's I/O performance bottleneck.
    If I have 10 nodes and they use the same storage disk to hold database, then
    they will do I/Os to the disk simultaneously.
    Maybe we got more latency ...
    Will that be a performance problem?
    How does RAC solve this kind of problem?
    Thanks.

    J.Laurence wrote:
    I see FC can solve the problem with bandwidth(throughput),There are a couple of layers in the I/O subsystem for RAC.
    There is CacheFusion as already mentioned. Why read a data block from disk when another node has it in is buffer cache and can provide that instead (over the Interconnect communication layer).
    Then there is the actual pipes between the server nodes and the storage system. Fibre is slow and not what the latest RAC architecture (such as Exadata) uses.
    Traditionally, you pop a HBA card into the server that provides you with 2 fibre channel pipes to the storage switch. These usually run at 2Gb/s and the I/O driver can load balance and fail over. So it in theory can scale to 4Gb/s and provide redundancy should one one fail.
    Exadata and more "+modern+" RAC systems use HCA cards running Infiniband (IB). This provides scalability of up to 40Gb/s. Also dual port, which means that you have 2 cables running into the storage switch.
    IB supports a protocol called RDMA (Remote Direct Memory Access). This essentially allow memory to be "+shared+" across the IB fabric layer - and is used to read data blocks from the storage array's buffer cache into the local Oracle RAC instance's buffer cache.
    Port to port latency for a properly configured IB layer running QDR (4 speed) can be lower than 70ns.
    And this does not stop there. You can of course add a huge memory cache in the storage array (which is essentially a server with a bunch of disks). Current x86-64 motherboard technology supports up to 512GB RAM.
    Exadata takes it even further as special ASM software on the storage node reconstructs data blocks on the fly to supply the RAC instance with only relevant data. This reduces the data volume to push from the storage node to the database node.
    So fibre channels in this sense is a bit dated. As is GigE.
    But what about the hard drive's reading & writing I/O? Not a problem as the storage array deals with that. A RAC instance that writes a data block, writes it into storage buffer cache.. where the storage array s/w manages that cache and will do the physical write to disk.
    Of course, it will stripe heavily and will have 24+ disk controllers available to write that data block.. so do not think of I/O latency ito of the actual speed of a single disk.

  • Replacing / Compiling Database Triggers in a HA/High Load system

    Hi there,
    My collegue has just asked me if downtime needs to be scheduled to replace an After IUD Database trigger where I've made a minor change.
    If this were a package or procedure, the answer would be an obvious yes, as I'd be wary of my users getting the dreaded: ORA-04068... State of Package has been disguarded.
    What about when working with database triggers?
    If the trigger is firing at the same time that I perform my Create or Replace Trigger DDL, will everything fall into a screaming heap?
    Notes:
    Oracle 10g R2 db.
    High DML rates on the table that the After IUD Trigger is 'attached' to.
    It's a near certainty that when the Create or Replace Trigger DDL runs, the trigger will be active at the time.
    The Trigger is 'simple' in that it's use: is to call a DB package with some of the :new values as params.
    +(I'm happy to rtfm - if someone can tell me where!!)+
    Cheers -
    Ron Marks

    If this were a package or procedure, the answer would be an obvious yes, as I'd be wary of my users getting the dreaded: ORA-04068... State of Package has been disguarded.Not true. This exception is thrown only if sessions have some state represented in package variables. If there is no associated session state, then no exception would be thrown after recompile.
    If the trigger is firing at the same time that I perform my Create or Replace Trigger DDL, will everything fall into a screaming heap?You'll wait - to lock an object in library cache which represents your trigger. Not sure if this would require a library cache lock on underlying table, but I would guess that it is required. BTW, locking a table would make no any sense: CREATE TRIGGER is a DDL, which means before it began it issues a COMMIT => all locks are released, and it is quite possible that a TM lock for CREATE TRIGGER won't be acquired since resource would be busy.
    Oracle 11gR2 claims to address problems of application upgrades on a high-load environments with a feature called "Editions" (not a good name - try to find something about it via Google...)

  • Major performance bottleneck in JSF RI 1.0

    We've been doing some load testing this week, and have come up with what I believe is a major performance bottleneck in the reference implementation.
    Our test suite was conducted two different application servers (JBoss and Oracle) and we found that in both cases response time degraded dramatically when hitting about 25-30 concurrent users.
    On analyzing a thread dump when the application server was in this state we noticed that close to twenty threads were waiting on the same locked resource.
    The resource is the 'descriptors' static field in the javax.faces.component.UIComponentBase class. It is a WeakHashMap. The contention occurs in the getPropertyDescriptors method, which has a large synchronized block.

    Well not the answer I was hoping for. But at least that's clear.
    Jayashri, I'm using JSF RI for an application that will be delivered to testing in august. Can you give advice wether I can expect an update for this bottleneck problem within that timeframe?
    Sincerely,
    Joost de Vries
    ps hi netbug. Saw you at theserverside! :-)

  • Weblogic Thread hogging at  java.io.WinNTFileSystem.getBooleanAttributes at high loads

    Hi,
    At high loads, we are seeing that one WebLogic thread is active on a windows native call for quite some time.. I can see the following stack statements in thread dumps taken at 10 minutes interval. This has led to slow server performance. 
    This happens on WebLogic 10.3.2 that is hosted in Windows Server 64 bit 2008 Enterprise Edition SP1.
    The JDK used is JRockit jdk1.6.0_20-R28.3.2 (64 Bit)
    Also this happens at high user loads only and also irregularly.
    Our eclipse link versions are (Implementation-Version: 1.0.1-20080905, Specification-Version: 1.0.1)
    Any recommendation or pointer is appreciated.
    Now the thread dump details
    "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE
                java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
                java.io.File.isDirectory(File.java:754)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive$1.accept(DirectoryArchive.java:91)
                java.io.File.listFiles(File.java:1134)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:89)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.init(DirectoryArchive.java:95)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.<init>(DirectoryArchive.java:73)
                org.eclipse.persistence.internal.jpa.deployment.DirectoryArchive.<init>(DirectoryArchive.java:55)
                org.eclipse.persistence.internal.jpa.deployment.ArchiveFactoryImpl.createArchive(ArchiveFactoryImpl.java:76)
                org.eclipse.persistence.internal.jpa.deployment.PersistenceUnitProcessor.findPersistenceArchives(PersistenceUnitProcessor.java:184)
                org.eclipse.persistence.internal.jpa.deployment.JPAInitializer.initialize(JPAInitializer.java:132)
                org.eclipse.persistence.jpa.PersistenceProvider.createEntityManagerFactory(PersistenceProvider.java:104)
                org.eclipse.persistence.jpa.PersistenceProvider.createEntityManagerFactory(PersistenceProvider.java:64)
                javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:78)
                javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:54)
                -- Our internal class calls
                org.jbpm.graph.def.Action.execute(Action.java:123)
                org.jbpm.graph.def.GraphElement.executeAction(GraphElement.java:235)
                org.jbpm.graph.def.GraphElement.executeActions(GraphElement.java:212)
                org.jbpm.graph.def.GraphElement.fireAndPropagateEvent(GraphElement.java:182)
                org.jbpm.graph.def.GraphElement.fireEvent(GraphElement.java:166)
                org.jbpm.graph.exe.ProcessInstance.<init>(ProcessInstance.java:120)
                org.jbpm.JbpmContext.newProcessInstance(JbpmContext.java:372)
                -- Our internal class calls
                sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                java.lang.reflect.Method.invoke(Method.java:597)
                org.apache.myfaces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:129)
                org.apache.myfaces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:63)
                javax.faces.component.UICommand.broadcast(UICommand.java:106)
                org.ajax4jsf.framework.ajax.AjaxViewRoot.processEvents(AjaxViewRoot.java:180)
                org.ajax4jsf.framework.ajax.AjaxViewRoot.broadcastEvents(AjaxViewRoot.java:158)
                org.ajax4jsf.framework.ajax.AjaxViewRoot.processApplication(AjaxViewRoot.java:346)
                org.apache.myfaces.lifecycle.LifecycleImpl.invokeApplication(LifecycleImpl.java:343)
                org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:86)
                -- Our internal class calls
                javax.faces.webapp.FacesServlet.service(FacesServlet.java:137)
                weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
                weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
                weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
                weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:144)
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                org.ajax4jsf.framework.ajax.xmlfilter.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:127)
                org.ajax4jsf.framework.ajax.xmlfilter.BaseFilter.doFilter(BaseFilter.java:277)
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                com.anychart.servlet.ResourceFilter.doFilter(ResourceFilter.java:53)
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                -- Our internal class calls
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                -- Our internal class calls
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
                weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
                weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)
                weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
                weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
                weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
                weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
                weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

    Hi,
    Maybe this thread could help, if it addresses your problem correctly:
    Performance issues with class loader on Windows server
    Regards,
    Mohab

  • Querying high load sql quies and users.

    Hi all,
    how do we find which query is performing high load and taking very long time on the database. and after that how we can find which user is giving that query and his session id.
    Please give me the details and query if you have.
    i tried with awr report and some queries with google search, but always i am getting result like no rows select.
    thank you.

    You've posted enough "101 (28 unresolved)" to know that you need to mark questions ANSWERED when they have been and to provide your 4 digit Oracle version (result of SELECT * FROM V$VERSION).
    >
    how do we find which query is performing high load and taking very long time on the database. and after that how we can find which user is giving that query and his session id.
    >
    If it ain't broke, don't fix it!
    What makes you think you have any queries that are 'performing high load and taking very long time'? Post the information that you are basing this on.
    >
    i tried with awr report and some queries with google search, but always i am getting result like no rows select.
    >
    How can anyone comment on your awr report, your queries and your results if you don't post them?
    Provide the details about what you have tried and what the results were.

  • Slow KVM and extremely high load

    I've just installed Archlinux on a i7 920 with 24 GB of ram. Not the fastest machine in the world but it performs well.
    But Virtual machines in KVM is quite slow and it shows a high load for the easiest of tasks. For instance, when installing CentOS 5 (or 6) (and no other virtual machines running) the load on my machine is ~25. Installing the OS takes about 15 minutes.
    The setup is this:
    3 1,5TB disk in raid5 (software)
    lvm
    /storage is ext4 (and a part of lvm) and this is the place where I have the disk images for the Virtual Machines.
    When the Virtual machine (I only have one) top tells me that qemu-kvm is using 20% cpu.
    Any hints?

    A common reason of high CPU load is qemu-only mode instead of kvm, as qemu emulates every instruction.
    First of all, check if your kvm module is loaded:
    $ lsmod|grep kvm
    kvm_intel 125165 0
    kvm 361546 1 kvm_intel
    Then, check if your VM is actually using KVM.
    Enter the VM's monitor and type "info kvm":
    QEMU 0.13.0 monitor - type 'help' for more information
    (qemu) info kvm
    info kvm
    kvm support: enabled

  • FX8320 with MSI 760GA-P43 (FX) MS-7699 drop speed on high load

    I had an FX8320 with watercooling on this MS-7699 motherboard, when with high load the CPU (all 8 cores) drop to 1.4Ghz then come back to the normal frequency within a few seconds, is there anyway to maintain the cpu speed at all time with high load?

    No there isn't. The board is not able to cope with the power draw of an FX83xx cpu. Sadly MSI for some time listed FX 8-core cpus supported on all 760G product sites despite these are only able to run 95W TDP cpus at the very maximum. The behavior you are experiencing is a cheap trick to make the cpu seem supported while the only way to keep the board from burning up is to throttle the cpu under load. To run the cpu at full performance you need a new board meant for supporting 125w+ cpus.

  • OEM High Load Widget -- does it work with Windows 7?

    The README page for the OEM High Load Widget ( http://www.oracle.com/technetwork/oem/grid-control/emgridcontrol-10-101399.html ) does not list Windows 7 as a supported O/S. Has anyone been able to get it to work on Windows 7 (64-bit)?
    My new PC at work is running Windows 7 64-bit O/S. I've had good luck in getting everything I was running on my previous Win XP SP3 machine but the OEM High Load Widget is not connecting.
    I have found that widget to be quite useful and really would like to be able to use it.

    Yes,http://manuals.info.apple.com/en_US/time_capsule_4th_gen_setup.pdf

  • Weblogic Server hanging at high load!!!!!!!!!!!

    Folks,
    We have a weblogic server 9.2 which is running fine but after quiet a while at high load we observe that the server goes down and following exception is displayed in the server logs:
    ####<Nov 13, 2009 2:42:48 AM EST> <Error> <WebLogicServer> <ibmrxcptpnp-wb1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1258098168215> <BEA-000337> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "617" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@1cec1cec", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
         java.net.SocketOutputStream.socketWrite0(Native Method)
         java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105)
         java.net.SocketOutputStream.write(SocketOutputStream.java:149)
         oracle.net.ns.DataPacket.send(Unknown Source)
         oracle.net.ns.NetOutputStream.flush(Unknown Source)
         oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         oracle.net.ns.NetInputStream.read(Unknown Source)
         oracle.net.ns.NetInputStream.read(Unknown Source)
         oracle.net.ns.NetInputStream.read(Unknown Source)
         oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
         oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
         oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
         oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
         oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
         oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1060)
         oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839)
         oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132)
         oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
         oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3422)
         weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:718)
         weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:460)
         weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1455)
         weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1372)
         weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1362)
         weblogic.common.resourcepool.ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:1767)
         weblogic.common.resourcepool.ResourcePoolImpl.access$700(ResourcePoolImpl.java:37)
         weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:1935)
         weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
         weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
         weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    >
    Any answers?????

    Hi,
    I have a similar issue on ODSI 10gR3 (WLS 10.3). I am having stuck threads - but, in my case it is clear that the database is down (not at the startup)
    Here is some more info:
    1. Test Connections On Reserve - Enabled
    2. Default values for other parameters
    Using Oracle thin driver for the connection pool "MyPool" -
    ./wlserver_10.3/server/lib/ojdbc6.jar
    My question is:
    1. How do I avoid the stuck thread caused when the database is down? During peak load, the stuck threads are causing some service requests to fail.
    I did my homework and the closest option I could find is set the statementTimeout (which is set currently to the default value of -1 i.e. never timeout). Oracle documentation (http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/111070_readme.html) says that the ojdbc6.jar supports setQueryTimeout() method.
    Is there any better way to attack this problem?
    <Dec 21, 2009 12:11:18 AM EST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "MyPool" failed with exception: "java.sql.SQLRecoverableException: Io exception: Read failed: Connection timed out".>
    <Dec 21, 2009 12:14:27 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:18:38 AM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@297ac2b", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-52 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: javax.management.remote.rmi.RMIConnector$RMINotifClient@271498f[fat lock]
    java.lang.Object.wait(Object.java:485)
    com.sun.jmx.remote.internal.ClientNotifForwarder.postReconnection(ClientNotifForwarder.java:304)
    javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.reconnectNotificationListeners(RMIConnector.java:1488)
    javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.doStart(RMIConnector.java:1568)
    com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:72)
    com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
    javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1420)
    javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:857)
    weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.getAttribute(ManagedMBeanServerConnection.java:288)
    javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:235)
    weblogic.management.jmx.MBeanServerInvocationHandler.doInvoke(MBeanServerInvocationHandler.java:477)
    weblogic.management.jmx.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:294)
    $Proxy69.getServerRuntime(Unknown Source)
    weblogic.management.mbeanservers.domainruntime.internal.DomainRuntimeServiceMBeanImpl.lookupServerRuntime(DomainRuntimeServiceMBeanImpl.java:242)
    sun.reflect.GeneratedMethodAccessor1880.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:355)
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
    com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
    weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:255)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:441)
    weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:55)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
    weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:441)
    weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:437)
    weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10$1.run(AuthenticatedSubjectInterceptor.java:582)
    weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10.run(AuthenticatedSubjectInterceptor.java:576)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor.invoke(AuthenticatedSubjectInterceptor.java:570)
    weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:305)
    javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1378)
    javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
    javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
    javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1338)
    javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:761)
    javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:136)
    weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:211)
    javax.management.remote.rmi.RMIConnectionImpl_1030_WLStub.invoke(Unknown Source)
    javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:969)
    com.bea.diagnostics.server.MBeanServerUtil.processRecordsFromArchive(MBeanServerUtil.java:177)
    com.bea.diagnostics.server.MetricEarliestTimestampLocator$RecordProcessorDesc.identifyEarliestTimestamps(MetricEarliestTimestampLocator.java:261)
    com.bea.diagnostics.server.MetricEarliestTimestampLocator.identifyEarliestTimestamps(MetricEarliestTimestampLocator.java:84)
    com.bea.diagnostics.server.MetricEarliestTimestampLocator.timerExpired(MetricEarliestTimestampLocator.java:79)
    weblogic.timers.internal.TimerImpl.run(TimerImpl.java:253)
    weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    <Dec 21, 2009 12:24:04 AM EST> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "MyPool" failed with exception: "java.sql.SQLRecoverableException: Io exception: Connection reset".>
    <Dec 21, 2009 12:24:04 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:04 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:05 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:07 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:12 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:17 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>
    <Dec 21, 2009 12:24:22 AM EST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "MyPool": Io exception: The Network Adapter could not establish the connection>

  • DATABASE performance is very high

    Hi,
      I'm having the following code. its performance is very high. Is it possible to reduce the processing time?
    SELECT  objnr gjahr kstar beknz twaer
                wtg001 wtg002 wtg003 wtg004 wtg005 wtg006
                wtg007 wtg008 wtg009 wtg010 wtg011 wtg012
                wkg001 wkg002 wkg003 wkg004 wkg005 wkg006
                wkg007 wkg008 wkg009 wkg010 wkg011 wkg012
                wog001 wog002 wog003 wog004 wog005 wog006
                wog007 wog008 wog009 wog010 wog011 wog012
        FROM cosp APPENDING TABLE i_cosp_t PACKAGE SIZE 2000
       FOR ALL ENTRIES IN p_i_vbakp
        WHERE lednr EQ c_lednr_0 AND
              versn EQ c_versn_0 AND
              gjahr EQ p_gjahr AND
              wrttp EQ c_wrttp AND
              kstar IN p_r_kstar_cosp.
        ENDSELECT.
      READ TABLE COSS AND APPEND TO P_I_COSP FILLED ABOVE
        SELECT  objnr gjahr kstar  uspob beknz twaer
                wtg001 wtg002 wtg003 wtg004 wtg005 wtg006
                wtg007 wtg008 wtg009 wtg010 wtg011 wtg012
                wkg001 wkg002 wkg003 wkg004 wkg005 wkg006
                wkg007 wkg008 wkg009 wkg010 wkg011 wkg012
                wog001 wog002 wog003 wog004 wog005 wog006
                wog007 wog008 wog009 wog010 wog011 wog012
        FROM coss APPENDING TABLE i_coss_t PACKAGE SIZE 2000
       FOR ALL ENTRIES IN p_i_vbakp
        WHERE lednr EQ c_lednr_0 AND
              versn EQ c_versn_0 AND
              gjahr EQ p_gjahr AND
              wrttp EQ c_wrttp AND
              kstar IN p_r_kstar_coss.
        ENDSELECT.
    Thanks & Regards,
    Vallamuthu.M

    Hi,
    First of all create an index on table COSP - in the order
    MANDT
    LEDNR
    GJAHR
    WRTTP
    VERSN
    KSTAR
    then use the following -
    SELECT objnr gjahr kstar beknz twaer
    wtg001 wtg002 wtg003 wtg004 wtg005 wtg006
    wtg007 wtg008 wtg009 wtg010 wtg011 wtg012
    wkg001 wkg002 wkg003 wkg004 wkg005 wkg006
    wkg007 wkg008 wkg009 wkg010 wkg011 wkg012
    wog001 wog002 wog003 wog004 wog005 wog006
    wog007 wog008 wog009 wog010 wog011 wog012
    FROM cosp Client specified APPENDING TABLE i_cosp_t PACKAGE SIZE 2000
    FOR ALL ENTRIES IN p_i_vbakp
    WHERE MANDT = sy_mandt AND
    lednr EQ c_lednr_0 AND
    gjahr EQ p_gjahr AND wrttp EQ c_wrttp AND
    versn EQ c_versn_0 AND kstar IN p_r_kstar_cosp.
    ENDSELECT.
    Then create an index on table COSS - in the order
    MANDT
    LEDNR
    GJAHR
    WRTTP
    VERSN
    KSTAR
    READ TABLE COSS AND APPEND TO P_I_COSP FILLED ABOVE
    SELECT objnr gjahr kstar uspob beknz twaer
    wtg001 wtg002 wtg003 wtg004 wtg005 wtg006
    wtg007 wtg008 wtg009 wtg010 wtg011 wtg012
    wkg001 wkg002 wkg003 wkg004 wkg005 wkg006
    wkg007 wkg008 wkg009 wkg010 wkg011 wkg012
    wog001 wog002 wog003 wog004 wog005 wog006
    wog007 wog008 wog009 wog010 wog011 wog012
    FROM coss APPENDING TABLE i_coss_t PACKAGE SIZE 2000
    FOR ALL ENTRIES IN p_i_vbakp
    WHERE MANDT = sy-MANDT AND
    lednr EQ c_lednr_0 AND gjahr EQ p_gjahr AND
    wrttp EQ c_wrttp AND versn EQ c_versn_0 AND
    kstar IN p_r_kstar_coss.
    ENDSELECT.
    And also if possible avoid
    SELECT--ENDSELECT.  Statement
    Hope this will help you.
    Regards
    MADAN NAMDEO

Maybe you are looking for