High Load on Concurrent Processing node

Hello :
version :11.5.10.CU2 / Linux -86
Recently I have started to notice that my concurrent processing node is loaded heavily between 3.00 AM to 6.00 AM and the load is normal. I need to find out the root cause of this spike. Is there a log that I can look for in concurrent processing node that will give me a clue.
Regards,
Bala

Look at the longest running requests during that time, then enable sql trace so you can start to tune them, or work with Oracle Support if they are standard programs running with suboptimal execution plans.
SELECT CR.REQUEST_ID "Request", U.USER_NAME "User", CP.USER_CONCURRENT_PROGRAM_NAME "Program",
TO_CHAR(CR.REQUESTED_START_DATE,'DD-MON-YY HH24:MI:SS') "Requested Start",
TO_CHAR(CR.ACTUAL_START_DATE,'DD-MON-YY HH24:MI:SS') "Actual Start",
ROUND((CR.ACTUAL_COMPLETION_DATE-ACTUAL_START_DATE)*24*60*60,0) "Duration (sec)",
ROUND((CR.ACTUAL_COMPLETION_DATE-ACTUAL_START_DATE)*24*60,2) "Duration (min)",
CR.PHASE_CODE "Phase", CR.STATUS_CODE "Status", CR.ARGUMENT_TEXT "Arguments"
FROM FND_CONCURRENT_REQUESTS CR,
FND_USER U,
FND_CONCURRENT_PROGRAMS_TL CP,
FND_RESPONSIBILITY_TL R
WHERE CR.REQUESTED_BY = U.USER_ID
AND CR.PROGRAM_APPLICATION_ID = CP.APPLICATION_ID
AND CR.CONCURRENT_PROGRAM_ID = CP.CONCURRENT_PROGRAM_ID
AND CP.LANGUAGE='US'
AND CR.RESPONSIBILITY_APPLICATION_ID = R.APPLICATION_ID
AND CR.RESPONSIBILITY_ID = R.RESPONSIBILITY_ID AND R.LANGUAGE='US'
AND (TO_CHAR(CR.ACTUAL_START_DATE,'DD-MON-YYYY HH24:MI') BETWEEN '30-JUN-2011 03:00' and '30-JUN-2011 06:00')
ORDER BY ROUND((CR.ACTUAL_COMPLETION_DATE-ACTUAL_START_DATE)*24*60*60,0) DESC

Similar Messages

  • High load CPU X process

    Hi,
    I'm using ArchLinux 64bit, with KDE 4.3.4 on a sony vaio, everything work fine, but after some normal work I see that process X takes more then 90% of CPU and never slow down. Even if I closed all my application it stays at 90%. I don't know how to see which application could cause this.
    As first impression it seems when I use samba adn move some file from my machine to a windows machine, but today it takes high load cpu without using samba, so I don't know what else do, any good advice?
    thanks a lot
    cesare

    Hi thank you very much for your reply, I'm sorry to reply just now.
    Anyway I have a Sony Vaio VGN.SR21M, using opensource driver xf86-video-radeonhd with hardware acc on. My graphics card is a Radeon3400HD
    I used the closed driver till I switch to Xorg 1.7 and so I can't using it anymore so I switched to the free one.
    Should I disable desktop effects?
    Thanks again
    Cesare
    Last edited by cesare (2010-01-25 08:48:01)

  • Better utilization of cpus and memory - Concurrent processing tier

    I have a three node 11.5.10.2 setup:
    Node1 - Database only (10.1.0.5)
    Node2 - Admin, Forms, Web
    Node3 - Concurrent processing
    My concurrent processing node has lot of free cpus and memory
    Cpus: Average utilization 20%
    Memory : Average utilization 15%
    The highest spikes I've had are:
    Cpus- 24% utilization
    Memory - 20% utilization
    What I can do on the concurrent processing tier that would make better use of the cpus/memory resources and improve performance?
    I have 2 dual core Intel(R) Xeon(R) processors and 16 GB Physical memory on the concurrent processing server.
    Thanks & regards,

    Hi Asif,
    You are seeing a manifestation of the old myth from the earlier part of this decade that you need many middle tier servers to effectively interact with users in a busy ERP environment. In reality, the middle tier processes which are normally web, forms, and concurrent managers typically consume only memory, and maybe10-20 percent CPU utilization of a modern quad core Xeon/Opteron Linux server. I have seen several 200 plus user single node 11i installations that handled the load just fine. In my experience, 95-97% of the ERP CPU utilization comes from database sessions, not from middle tier processes. Occasionally, a forms session goes to a runaway, or a ar60run process runs at 100% CPU while the request is running, but if you follow the Oracle recommendations of limiting fnd_concurrent_requests to 50k rows or less, you will not see much CPU utilization from FNDLIBR processes.
    Most 32 GB 2 socket quad core servers can support 200 plus forms user connections plus host the concurrent manager as well. I would only add additional web/forms nodes after you start paging on the middle tier box. Having concurrent managers separated on a 10-20 percent utilized server is sort of a waste of a server.
    Regards,
    -Michael

  • One of 4 node RAC always have higher load averages and higher than others

    Hello,
    We have a 4 node rac, 9208 on linux 4. When viewing top, we noticed the same one node always have a higher load average than the other 3 nodes. Is this normal. Loan balance is working fine but this one node always have higher load average. This is the node where we do the rac installation. Thank you.

    I do not remember what is the default for clb_goal (client load balancing) for 9i but 10g is LONG.
    check it
    select clb_goal from dba_services where name = <service name>
    you may have to change from LONG to SHORT OR SHORT to LONG depending your connection types.
    dbms_service.MODIFY_SERVICE(‘<service>’,clb_goal=> dbms_service.CLB_GOAL_LONG);
    Read the following article.
    http://www.databasejournal.com/features/oracle/article.php/3659411/Oracle-RAC-Administration---Part-15-Connection-Load-Balancing-and-FAN.htm

  • High load and high CPU caused by perl process

    Using: Oracle 10.2.0.1.0
    OS: Red Hat ES 4
    Since this morning I have high load on one of my prod database server, this server has 3 databases of similar size.
    High load is caused by perl process, one process has been running for over 75 min’s. process info
    14302 ?        R     74:50 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.pl
    26530 ?        R     23:36 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.pl
    31152 ?        R      7:58 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.plSearching segAdv.pl on google, Metalink and this forum gives me ZERO results.
    Any idea why this is running and cuasing this issue now (for last 8 months, I had not simlar perl issue), can I stop it without kill –9, maybe with emctl stop dbconsole.
    Thanks in advance for any help.

    Hi Lee,
    You say this is a core router, but as people use this terms to mean different things, are there any directly connected hosts? If there are one thing to check for is hosts sending multicast traffic with TTL=1. If you run the command show ip traffic | in bad hop and look at the rate the "bad hop count" is increasing. If I had a £ for every time I've seen that problem.... I couldn't retire just yet, but at least I could go buy myself a beer or two.
    If you're seeing the hop count increasing at a rapid rate there are a couple of options:
    Take a read of Troubleshooting high CPU on a 6500 with sup720 starting from the section Tools used to determine the source of the traffic and identify the hosts sending multicast with TTL=1.
    If the Catalyst 6500 use PFC 3B add the command mls rate-limit all ttl-failure 100 10 to drop the multicast traffic in hardware.
    The first option will take you longer, but at least you identify the cause and can advise the application owner why they should change their applications behaviour.
    Regards

  • 2008 R2/ Win7 Offline Files Sync causing High Load on server

    Hi Everyone,
    I have recently been investigating extremely high CPU Usage from the System Process on my company's main File Cluster.
    We managed to track SRV2.sys Threads causing high CPU load within the system process, but was having issues identifying why this was the case.
    As per Microsoft's direction via support call, we have installed the latest SRV2.sys Hotfixes, but this does not appear to have allivated the issue we are experiencing. We have also added more CPU and Memory into both nodes, which has not helped either.
    We have since managed to create a system dump and is being sent to MS Support for analysis.
    I have noticed the following that appears to happen on our cluster:
    Whenever our CAD/Design department run certain functions within their apps running on a windows 7 client (apps include MicroStation, Revit, AutoCAD etc) we see a massive spike and flatline of the system process.
    We found several users with Windows 7 clients that have Configured Offline File to sync an entire Network Volume (some volumes are 2TB Plus, so would never fit on a users computer anyway, i was quite shocked when I found this). How we spotted this was through
    Resource Monitor, the System Process was trolling through all the folders and files in a given volume (it was "reading every single folder). Now, while this was the system process, we could identify the user by using the Open Files view in Server Manager's
    Share and Storage Management tool.
    I have done a fair bit of research and found that a lot of CAD/Drawing applications in the market have issues with using SMB2 (srv2.sys). When reviewing the products that we use, I noticed that a lot of these products actually recommended disabling SMB2
    and reverting to SMB1 on File Server and/or the clients.
    I have gone down the path of disabling SMB2 on all Windows 7 clients that have these CAD Applications installed to assist with lowering the load (our other options are to potentially shift the CAD Volumes off our main file cluster to further isolation these
    performance issues we have been experiencing.) We will be testing this again tomorrow to confirm that the issue is resolved when mass amounts of CAD users access data on these volumes via their CAD Application.
    We only noticed the issue with Offline Files today with trying to sync an ENTIRE Volume. My questions are:
    Should Offline File sync's really cause this much load on a File Server?
    Would the the size of the volume the sync is trying to complete create this additional load within the system process?
    What is the industry considered "Best Practice" regarding Offline Files setup for Volumes which could have up to 1000+ users attached? (My personal opinion is that Offline Files should only be sync of user "Personal/Home" Folders
    as they users themselves have a 1 to 1 relationship with this data.)
    Is there an easier way to identify what users have Offline Files enabled and "actually being used" (from my understanding, Sync Center and Offline Files are enabled by default, but you obviously have to add the folders/drives you wish to sync)?
    If I disable the ability for Offline Files on the volumes, what will the user experience be when/if they try to sync their offline files config after this has been disabled on the volume?
    Hoping for some guidance regarding this setup with Offline Files.
    Thanks in Advance!
    Simon

    Hi Everyone,
    Just thought I would give an update on this.
    While we're still deploying http://support.microsoft.com/kb/2733363/en-us to
    the remainder of our Windows 7 SP1 fleet, according to some Network Traces and XPerf that were sent to MS Support, it looks as though the issu with the Offline Files is now resolved.
    However, we are still having some issues with High CPU with the System Process in particular. Upon review of the traces, they have found a lot of ABE related traffic, leading them to believe that the issue may also be caused by ABE Access on our File Cluster.
    We already have the required hotfix for ABE on 2008 R2 installed (please see
    http://support.microsoft.com/kb/2732618/en-us), although we have it set to a certain value that MS believe may be too high. Our current value is set to "5", as that is the lowest level of any type of permission is set (i.e. where the lowest level of inheritance
    is broken).
    I think I will mark this as resolved regarding the Offline Files issue (as it did resolve the problem with the Offline Files)...
    Fingers crossed we can identify the rest of the high load within the System Process!!!

  • COncurrent process fails with warning for OPP

    Concurrent process generates xml output (version 5.6.2) .
    When it tries to publish it , the log suggests checking out the OPP log.
    The OPP log has a series of java entries for the request, including one of the type
    Caused by: oracle.xdo.parser.v2.XPathException: Expected 'QNAME' instead of ''.
    header of the OPP log says
    Resolution History
    18-OCT-06 20:36:05 GMT
    Can you easily recover from, bypass or work around the problem? = NO
    Does your system or application continue normally after the problem occurs? =
    YES
    Are the standard features of the system or application still available; is the
    loss of service minor? = YES
    ### Informational Only:Warning Presented to Customer ###
    You chose 'Other technical issues with this product' from the Type of Problem
    list. This is a generic Service Request form which will require additional time
    to be routed appropriately. You may be able to expedite the resolution of your
    Service Request by doing the following: 1. Use your browser’s back button to
    return to the Create a SR – Brief Description page. 2. Select an option from
    the Type of Problem list that more closely describes the problem.
    ### Requested files:ACT(Note 183274.1),RDA (Note 175853.1). Files to be loaded.
    None
    ### Detailed Problem Statement ###
    Concurrent process for XML Publisher fails with warning. Cannot see Diagnostics
    --> XML output completely. Only first page visible. Buttons and Menu items
    disabled not allowing to view the whole XML file to troubleshoot for
    correct/incorrect tags.
    Attached are OPP Manager log file, Screen shot of 'Preview' output jjst before
    submitting request. Also log file of failed concurrent request which contains
    standard error " See OPP Service log file for more".
    ### Steps to Reproduce Problem ###
    Registered a concurrent process with XML template , and running the process
    from Receivables Manager responsibility.
    Ran concurrent process for 1) Standard RAXINV report , with output method as
    'XML'. 2) Customised RAXINV report, with output method as 'XML'
    ### Instance Name(s) and Type of System(s) Where Error Occurs ###
    Lunix, 11.5.10 on 10g
    ### Recent Changes to this Environment ###
    None
    ### Workarounds Used ###
    None
    ### How is this Issue Impacting Your Business ###
    Business is readying for CRP3, so would like to have reports ready by then.
    Contact me via : MetaLink
    18-OCT-06 20:48:07 GMT
    The customer : [email protected] : has uploaded the following file via MetaLink:
    C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\XML_Pub1.bmp
    18-OCT-06 20:49:14 GMT
    The customer : [email protected] : has uploaded the following file via MetaLink:
    C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\XMLPublisher_java.t
    xt
    18-OCT-06 21:17:26 GMT
    Hi Bashobi,
    Thank you for using MetaLink. We are currently reviewing/researching the situation and will update the SR as soon as we have relevant information
    Best Regards,
    Bill
    Oracle Support Services
    STATUS
    =======
    @WIP -- Work In Progress
    18-OCT-06 21:18:04 GMT
    Email Update button has been pressed: Sending email to [email protected].
    18-OCT-06 21:21:45 GMT
    ISSUE CLARIFICATION
    ====================
    On 11.5.10.2 in Production:
    Find concurrent process for XML Publisher fails with warning. Cannot see Diagnostics
    --> XML output completely. Only first page visible. Buttons and Menu items
    disabled not allowing to view the whole XML file to troubleshoot for
    correct/incorrect tags.
    EXPECTED BEHAVIOR
    Expect the processes run successfully
    STEPS
    The issue can be reproduced at will with the following steps:
    1. Registered a concurrent process with XML template .
    2. Run the process from Receivables Manager responsibility.
    3. Ran concurrent process for
    1) Standard RAXINV report , with output method as 'XML'
    2) Customised RAXINV report, with output method as 'XML'
    BUSINESS IMPACT
    The issue has the following business impact:
    Business is readying for CRP3, so would like to have reports ready by then.
    ACTION PLAN
    ============
    Hi Bashobi,
    1. Can you reproduce the issue from sysadmin responsibility with Active users report? if yes
    please upload OPP log file after reproduce it and upload the request log file.
    Thanks.
    18-OCT-06 23:09:02 GMT
    New info : [email protected] : Did the same as suggested, Request
    completes successfully.
    But what is the relation between Active Users report and a report sent via XML
    Publisher?
    19-OCT-06 15:54:26 GMT
    ACTION PLAN
    ============
    Hi Bashobi,
    1. When you reproduce the issue from sysadmin responsibility with Active users report change the format out put to XML,
    Save your change:
    2. Upload OPP manager log file and upload the request log file.
    Thanks,
    Bill.
    19-OCT-06 15:54:33 GMT
    Email Update button has been pressed: Sending email to [email protected].
    19-OCT-06 16:42:23 GMT
    New info : [email protected] : Bill,
    No, I could not reproduce the problem with Active Users program.
    Attaching OPP manager log in the next update. Bashobi
    19-OCT-06 17:05:07 GMT
    New info : [email protected] : Bill, Attaching internal log and manager
    log
    Bashobi
    19-OCT-06 17:06:08 GMT
    The customer : [email protected] : has uploaded the following file via MetaLink:
    C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\OPP_internal_log.tx
    t
    19-OCT-06 17:07:10 GMT
    The customer : [email protected] : has uploaded the following file via MetaLink:
    C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\OPP_manager_log.txt
    19-OCT-06 18:14:32 GMT
    UPDATE
    =======
    Hi Bashobi,
    Thank you for providing the requested information.
    Best Regards,
    Bill
    STATUS
    =======
    @WIP -- Work In Progress
    19-OCT-06 18:14:36 GMT
    Email Update button has been pressed: Sending email to [email protected].
    19-OCT-06 18:59:25 GMT
    DATA COLLECTED
    ===============
    SCREEN SHOTS
    XML_Pub1.bmp:
    Error:
    Jave.sql SQL Exception No corresponding LOB data found: select File_data_dbms_lob...........
    LOG FILE
    1. XMLPublisher_java.txt:
    [10/18/06 11:16:57 AM] [5670:RT277479] Completed post-processing actions for request 277479.
    [10/18/06 11:29:06 AM] [OPPServiceThread1] Post-processing request 277488.
    [10/18/06 11:29:06 AM] [5670:RT277488] Executing post-processing actions for request 2
    77488.
    [10/18/06 11:29:06 AM] [5670:RT277488] Starting XML Publisher post-processing action.
    [10/18/06 11:29:06 AM] [5670:RT277488]
    Template code: XXBE_TEST_INV
    Template app: AR
    Language: en
    Territory: US
    Output type: null
    [10/18/06 11:29:07 AM] [UNEXPECTED] [5670:RT277488] java.lang.reflect.InvocationTarg
    etException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    10/18/06 11:29:07 AM] [5670:RT277488] Completed post-processing actions for reque
    st 277488.
    2. OPP_manager_log.txt:
    [10/19/06 5:22:36 AM] [main] Starting GSF service with concurrent process id = 5688.
    [10/19/06 5:22:36 AM] [main] Initialization Parameters: oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5
    [10/19/06 5:22:37 AM] [Thread-12] Service thread starting up.
    [10/19/06 5:22:37 AM] [Thread-13] Service thread starting up.
    ISSUE VERIFICATION
    =================
    Issue veryfied by the screen shots: XML_Pub1.bmp which shows:
    Jave.sql SQL Exception No corresponding LOB data found: select File_data_dbms_lob...........
    ACTION PLAN
    ============
    Hi Bashobi,
    1. Since you can not reproduce the issue from sysadmin responsibility can we say the issue specific to Receivables Manager
    responsibility<INV xml report>?
    2. You said before: "Concurrent process for XML Publisher fails with warning. Cannot see Diagnostics
    --> XML output completely. Only first page visible. Buttons and Menu items
    disabled not allowing to view the whole XML file to troubleshoot for
    correct/incorrect tags.":
    Is the above mean :
    -The xml publisher report completed but you can not view the whole XML file
    -Or the xml publisher report error out with "Jave.sql SQL Exception No corresponding LOB data found: select File_data_dbms_lob.."
    Thanks.
    19-OCT-06 18:59:32 GMT
    Email Update button has been pressed: Sending email to [email protected].
    20-OCT-06 21:36:13 GMT
    Email Update button has been pressed: Sending email to [email protected].
    24-OCT-06 21:50:06 GMT
    New info : [email protected] : Hi,
    We are in the process of upgrading XML Publisher at the moment. I would like to
    wait a day or two to test the report with the upgrade.
    Thanks,Bashobi
    26-OCT-06 20:15:07 GMT
    New info : [email protected] : Hi,
    Our DBA has applied the 5.6.2 patch on our dev instance. My concurrent process
    runs to a warning.
    The warnings in the OPP manager log file say
    Template code: BE_RAXINV
    Template app: XXBE
    Language: en
    Territory: US
    Output type: PDF
    [10/26/06 12:50:13 PM] [UNEXPECTED] [5849:RT281617]
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    Any idea?

    Hi All,
    I have ran into the same issue.
    "The Output Post-processor is running but has not picked up this request.
    Any ideas.
    Thanks
    Ravi

  • How to configure number of concurrent processes in XI?

    Hi experts,
    I have a simple interface:
    FILE -> XI -> FILE
    The input file is a huge file containing thousands of records.
    XI will split the huge input files, so that each record gets written to one file each.
    Owing to the huge number of files being generated, I have many messages in status = "To Be Delivered" in the RWB.
    When I do a filter for status = "Delivering", I see 4 messages in that status. (Of course the message IDs will differ from time to time, but at any point in time, I can only see 4 messages with the status = "Delivering").
    How can I increase the number of concurrent processes that produces the output files?
    Please advise.
    Thanks.
    Ron

    Ron,
    From this perfromance tuning guide , page 12 ,
    https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/70ada5ef-0201-0010-1f8b-c935e444b0ad
    <i>The Adapter Framework requires a set of fixed application threads within the J2EE Engine for proc-essing different internal messaging queues (AFW queue, BC Adapter, CIDX Adapter, and so on). Ad-ditionally, certain channel configurations also require a fixed set of application threads (for example, RFC sender channel, File adapter channels). This may require a higher number of application threads. Therefore, within the J2EE Engine configuration (configtool or Visual Admin), go to:
    Global Configuration -> Server -> Kernel -> ApplicationThreadManager
    Set the MaxThreadCount property to 300</i>
    Not sure if this is the exact property but this is sure worth giving a shot.
    Regards
    Bhavesh

  • Strange Oracle Behaviour - only month end or high load periods

    Hi SAP Gurus
    In one of our systems I am expreriencing a strange behaviour from Oracle, in that it seems to be in reconnect, but yet it is not, it resets and carries on but do produce Dump and SM21 log messages, the dump is not really helpfull as it is the same dump but with different program everytime.
    I have looked in oracle alert log and found no event tying back to the SM21 log message or dump.
    The following is what I found in the developer trace of one of these insident, but there are many, about 125 - 200 for the day during high load:
    Fri Nov 28 10:05:55 2008
    C  *** ERROR => Failed to register application info: orc=3113
    [dboci.c      3042]
    C  *** ERROR => Error 3114 in stmt_fetch()
    [dbsloci.c    13001]
    C  *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
    [dbsloci.c    13002]
    C  sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB                       
    C  SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
    C  NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
    C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
    C  "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
    C  GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
    C  INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
    C  ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
    C  TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
    C  DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
    C  PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
    C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
    C  SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
    C  FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
    C  sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB                       
    C  prep=0,lit=0,nsql=0,lobret=0,#exec=1,dbcnt=0,upsh_p=00000000,ocistmth_p=345E20EC
    C  IN : cols=4,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=17,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=3470A8D8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  OUT: cols=105,rmax=58,xcnt=58,rpc=1,rowi=0,rtot=1,upto=-1,rsize=895,vmax=128,bound=1,iobuf_p=0E0A4EC8,vda_p=34F8F7E8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
    C  NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
    C   , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
    C  "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
    C  GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
    C  INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
    C  ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
    C  TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
    C  DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
    C  PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
    C   "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
    C  SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
    C  FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  ***LOG BY4=> sql error 3114   performing FET on table CATSDB     [dbtran#10 @ 7292] [dbtran  7292 ]
    B  ***LOG BY0=> ORA-03114: not connected to ORACLE [dbtran#10 @ 7292] [dbtran  7292 ]
    B  dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
    B   RSLT: {dbsl=99, tran=1}
    B   FHDR: {tab='CATSDB', fcode=225, mode=2, bpb=0, dbcnt=0, crsr=1,
    B          hold=0, keep=1, xfer=0, pkg=0, upto=0, init:b=0,
    B          init:p=00000000, init:#=871, wa:p=0XA000DC88, wa:#=871}
    B  dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
    B   STMT:
    B   CRSR: {tab='CATSDB', id=1, hold=0, prop=0x1000, max.in@0=1, fae:blk=1,
    B          con:id=0, con:vndr=5, val=2,
    B          key:#=4, xfer=0, xin:#=0, row:#=0, upto=0, wa:p=0XA000DC88}
    C  *** ERROR => Error 3114 in stmt_fetch()
    [dbsloci.c    13001]
    C  *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
    [dbsloci.c    13002]
    C  sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD                     
    C  SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
    C  R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
    C  sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD                     
    C  prep=0,lit=0,nsql=0,lobret=0,#exec=29,dbcnt=0,upsh_p=00000000,ocistmth_p=34B9ADD4
    C  IN : cols=3,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=45,vmax=32,bound=1,iobuf_p=0E0A4EC8,vda_p=31361B18
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  OUT: cols=9,rmax=1,xcnt=1,rpc=1,rowi=0,rtot=1,upto=-1,rsize=56,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=345C39B8
    C       lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
    C  SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
    C  R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  ***LOG BY4=> sql error 3114   performing SEL on table REPOLOAD   [dbrepo#6 @ 2657] [dbrepo  2657 ]
    B  ***LOG BY0=> ORA-03114: not connected to ORACLE [dbrepo#6 @ 2657] [dbrepo  2657 ]
    A  TH VERBOSE LEVEL FULL
    A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.
    A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.
    A  ** RABAX: level LEV_RX_VMIT_CLEANUP entered.
    A  ** RABAX: level LEV_RX_VMIT_CLEANUP completed.
    A  ** RABAX: level LEV_RX_ROLLBACK entered.
    C  *** ERROR => OCI-call 'OCITransRollback' failed: rc = 3114
    [dboci.c      3127]
    B  ***LOG BYM=> severe DB error 3114      ; work process in reconnect status [dbsh#3 @ 1123] [dbsh    1123 ]
    B  Reconnect state is entered by connection:
    B  0: name = R/3, con_id = 000000000 state = ACTIVE      , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    B  hdl_error_on_commit_rollback: DB-ROLLBACK detected RECONNECT state
    M  ThShortCommit: db unusable
    B  db_con_reconnect disconnecting connection:
    B  0: name = R/3, con_id = 000000000 state = INACTIVE    , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    C  Disconnecting from connection 0 ...
    C  Close user session (con_hdl=0,svchp=045D7FA4,usrhp=345E67A0)
    C  Detaching from DB Server (con_hdl=0,svchp=045D7FA4,srvhp=045D0174)
    C  Now I'm disconnected from ORACLE
    B  db_con_reconnect performing the reconnect for con:
    B  0: name = R/3, con_id = 000000000 state = DISCONNECTED, perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
    C  got NLS_LANG='AMERICAN_AMERICA.US7ASCII' from environment
    C  Client NLS settings:
    C  Logon as OPS$-user to get SAPR3's password
    C  Connecting as /@PRD on connection 0 (nls_hdl 0) ... (dbsl 640 250407)
    C  Nls CharacterSet                 NationalCharSet              C      EnvHp      ErrHp ErrHpBatch
    C    0 US7ASCII                                                  1   045CBB38   0E0A4648   0E0A40B0
    C  Attaching to DB Server PRD (con_hdl=0,svchp=045
    SM51 log entries:
         09:00:32     MS                         Q0I     Operating system call recv failed (error no. 10054)
         09:00:32     DP                         Q0K     Connection to message server (on FCMSPROD) established
         09:00:32     DIA     004     040     181JOLENES     VA02     BYM     SQL error 3114 . Work processes in reconnect status
         09:00:32     DP                         Q1C     MsgServer Hardware ID Was Determined
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     BYY     Work process has left reconnect status
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:32     DIA     014     040     161ADRIV     VA02     BYY     Work process has left reconnect status
         09:00:32     DIA     013     040     140ELECIA          BYY     Work process has left reconnect status
         09:00:32     DIA     003     040     133MIRANDAV     KO88     BYY     Work process has left reconnect status
         09:00:32     DIA     003     040     133MIRANDAV     KO88     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:32     DIA     001     040     181ANNELINEJ     VA42     AB1     > Short dump "081128 090032 FCMSPROD 181ANNELINEJ" generated
         09:00:32     DIA     003     040     133MIRANDAV     KO88     AB1     > Short dump "081128 090032 FCMSPROD 133MIRANDAV " generated
         09:00:32     DIA     009     040     120ELIZEP     FB01     BYY     Work process has left reconnect status
         09:00:33     DIA     009     040     120ELIZEP     FB01     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     004     040     181JOLENES     VA02     BYY     Work process has left reconnect status
         09:00:33     DIA     002     040     133KOEKIEJ          BYY     Work process has left reconnect status
         09:00:33     DIA     002     040     133KOEKIEJ          AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     004     040     181JOLENES     VA02     AB0     Run-time error "DBIF_REPO_SQL_ERROR" occurred
         09:00:33     DIA     009     040     120ELIZEP     FB01     AB1     > Short dump "081128 090033 FCMSPROD 120ELIZEP " generated
         09:00:33     DIA     005     040     181CLIVEB     VA02     BYY     Work process has left reconnect status
         09:00:33     DIA     003                    R0Z     The update dispatch info was reset
         09:00:33     DIA     003                    R0Z     The update dispatch info was reset
    Please can you assist in maybe you have seen this before.
    Regards
    Pieter

    Hi
    Yes I have gone through the Oracle Alert log several time, It is huge, but I am pasting an extract for the entire day this was happening into this reply.  Oracle version is 9.2.0.8.0 with the only intrim path I can see being installed : 6130293.
    Following is the extract from Alert Log for the day this occurred, again I must stress that this happens only on high system load days like month end:
    Fri Nov 28 03:43:33 2008
    ALTER TABLESPACE "PSAPBTABI" END BACKUP
    Fri Nov 28 03:43:33 2008
    Completed: ALTER TABLESPACE "PSAPBTABI" END BACKUP
    Fri Nov 28 03:43:37 2008
    ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
    Fri Nov 28 03:53:38 2008
    ALTER TABLESPACE "PSAPCLUD" END BACKUP
    Fri Nov 28 03:53:38 2008
    Completed: ALTER TABLESPACE "PSAPCLUD" END BACKUP
    Fri Nov 28 03:53:42 2008
    ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
    Fri Nov 28 03:54:36 2008
    ALTER TABLESPACE "PSAPCLUI" END BACKUP
    Fri Nov 28 03:54:36 2008
    Completed: ALTER TABLESPACE "PSAPCLUI" END BACKUP
    Fri Nov 28 03:54:38 2008
    ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
    Fri Nov 28 03:55:47 2008
    ALTER TABLESPACE "PSAPDDICD" END BACKUP
    Fri Nov 28 03:55:47 2008
    Completed: ALTER TABLESPACE "PSAPDDICD" END BACKUP
    Fri Nov 28 03:55:50 2008
    ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
    Fri Nov 28 03:56:41 2008
    ALTER TABLESPACE "PSAPDDICI" END BACKUP
    Fri Nov 28 03:56:41 2008
    Completed: ALTER TABLESPACE "PSAPDDICI" END BACKUP
    Fri Nov 28 03:56:45 2008
    ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
    Fri Nov 28 03:56:49 2008
    ALTER TABLESPACE "PSAPDOCUD" END BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUD" END BACKUP
    Fri Nov 28 03:56:52 2008
    ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
    Fri Nov 28 03:56:55 2008
    ALTER TABLESPACE "PSAPDOCUI" END BACKUP
    Completed: ALTER TABLESPACE "PSAPDOCUI" END BACKUP
    Fri Nov 28 03:56:59 2008
    ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
    Fri Nov 28 04:03:46 2008
    ALTER TABLESPACE "PSAPEL640D" END BACKUP
    Fri Nov 28 04:03:46 2008
    Completed: ALTER TABLESPACE "PSAPEL640D" END BACKUP
    Fri Nov 28 04:03:50 2008
    ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
    Fri Nov 28 04:04:37 2008
    ALTER TABLESPACE "PSAPEL640I" END BACKUP
    Fri Nov 28 04:04:37 2008
    Completed: ALTER TABLESPACE "PSAPEL640I" END BACKUP
    Fri Nov 28 04:04:40 2008
    ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
    Fri Nov 28 04:20:17 2008
    ALTER TABLESPACE "PSAPES640D" END BACKUP
    Fri Nov 28 04:20:17 2008
    Completed: ALTER TABLESPACE "PSAPES640D" END BACKUP
    Fri Nov 28 04:20:20 2008
    ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
    Fri Nov 28 04:36:26 2008
    ALTER TABLESPACE "PSAPES640I" END BACKUP
    Fri Nov 28 04:36:26 2008
    Completed: ALTER TABLESPACE "PSAPES640I" END BACKUP
    Fri Nov 28 04:36:29 2008
    ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
    Fri Nov 28 04:36:40 2008
    ALTER TABLESPACE "PSAPLOADD" END BACKUP
    Fri Nov 28 04:36:40 2008
    Completed: ALTER TABLESPACE "PSAPLOADD" END BACKUP
    Fri Nov 28 04:36:43 2008
    ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
    Fri Nov 28 04:36:53 2008
    ALTER TABLESPACE "PSAPLOADI" END BACKUP
    Completed: ALTER TABLESPACE "PSAPLOADI" END BACKUP
    Fri Nov 28 04:36:56 2008
    ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
    Fri Nov 28 04:41:04 2008
    ALTER TABLESPACE "PSAPPOOLD" END BACKUP
    Fri Nov 28 04:41:04 2008
    Completed: ALTER TABLESPACE "PSAPPOOLD" END BACKUP
    Fri Nov 28 04:41:08 2008
    ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
    Fri Nov 28 04:45:17 2008
    ALTER TABLESPACE "PSAPPOOLI" END BACKUP
    Fri Nov 28 04:45:17 2008
    Completed: ALTER TABLESPACE "PSAPPOOLI" END BACKUP
    Fri Nov 28 04:45:21 2008
    ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
    Fri Nov 28 04:47:25 2008
    ALTER TABLESPACE "PSAPPROTD" END BACKUP
    Fri Nov 28 04:47:25 2008
    Completed: ALTER TABLESPACE "PSAPPROTD" END BACKUP
    Fri Nov 28 04:47:29 2008
    ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
    Fri Nov 28 04:48:09 2008
    ALTER TABLESPACE "PSAPPROTI" END BACKUP
    Fri Nov 28 04:48:09 2008
    Completed: ALTER TABLESPACE "PSAPPROTI" END BACKUP
    Fri Nov 28 04:48:13 2008
    ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
    Fri Nov 28 04:48:46 2008
    ALTER TABLESPACE "PSAPSOURCED" END BACKUP
    Fri Nov 28 04:48:46 2008
    Completed: ALTER TABLESPACE "PSAPSOURCED" END BACKUP
    Fri Nov 28 04:48:49 2008
    ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
    Fri Nov 28 04:49:29 2008
    ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
    Fri Nov 28 04:49:29 2008
    Completed: ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
    Fri Nov 28 04:49:34 2008
    ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
    Fri Nov 28 05:05:16 2008
    ALTER TABLESPACE "PSAPSTABD" END BACKUP
    Fri Nov 28 05:05:16 2008
    Completed: ALTER TABLESPACE "PSAPSTABD" END BACKUP
    Fri Nov 28 05:05:23 2008
    ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
    Fri Nov 28 05:21:15 2008
    ALTER TABLESPACE "PSAPSTABI" END BACKUP
    Fri Nov 28 05:21:15 2008
    Completed: ALTER TABLESPACE "PSAPSTABI" END BACKUP
    Fri Nov 28 05:21:19 2008
    ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
    Fri Nov 28 05:21:42 2008
    ALTER TABLESPACE "PSAPUSER1D" END BACKUP
    Fri Nov 28 05:21:42 2008
    Completed: ALTER TABLESPACE "PSAPUSER1D" END BACKUP
    Fri Nov 28 05:21:46 2008
    ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
    Completed: ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
    Fri Nov 28 05:23:03 2008
    ALTER TABLESPACE "PSAPUSER1I" END BACKUP
    Fri Nov 28 05:23:03 2008
    Completed: ALTER TABLESPACE "PSAPUSER1I" END BACKUP
    Fri Nov 28 05:23:06 2008
    ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
    Completed: ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
    Fri Nov 28 05:24:03 2008
    ALTER TABLESPACE "SYSTEM" END BACKUP
    Fri Nov 28 05:24:03 2008
    Completed: ALTER TABLESPACE "SYSTEM" END BACKUP
    Fri Nov 28 05:24:07 2008
    ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
    Completed: ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
    Fri Nov 28 05:42:06 2008
    ALTER TABLESPACE "UNDOTBS1" END BACKUP
    Fri Nov 28 05:42:06 2008
    Completed: ALTER TABLESPACE "UNDOTBS1" END BACKUP
    Fri Nov 28 05:42:07 2008
    ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program Files\CA\BrightStor ARCserve Backup Enterprise Option for SAP R3 for Oracle\CONTROL.PRD' REUSE
    Completed: ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program F
    Fri Nov 28 05:42:14 2008
    Beginning log switch checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
    Thread 1 advanced to log sequence 30706
      Current log# 2 seq# 30706 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30706 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 05:42:14 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30705
    ARC0: Beginning to archive log 3 thread 1 sequence 30705
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30705.001'
    Fri Nov 28 05:42:27 2008
    ARC1: Evaluating archive   log 3 thread 1 sequence 30705
    ARC1: Unable to archive log 3 thread 1 sequence 30705
          Log actively being archived by another process
    Fri Nov 28 05:42:29 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30705
    Fri Nov 28 06:11:26 2008
    Completed checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
    Fri Nov 28 09:20:59 2008
    Beginning log switch checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
    Thread 1 advanced to log sequence 30707
      Current log# 1 seq# 30707 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
      Current log# 1 seq# 30707 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
    Fri Nov 28 09:20:59 2008
    ARC0: Evaluating archive   log 2 thread 1 sequence 30706
    ARC0: Beginning to archive log 2 thread 1 sequence 30706
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30706.001'
    Fri Nov 28 09:21:18 2008
    ARC0: Completed archiving  log 2 thread 1 sequence 30706
    Fri Nov 28 09:51:06 2008
    Completed checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
    Fri Nov 28 10:44:43 2008
    Beginning log switch checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
    Thread 1 advanced to log sequence 30708
      Current log# 4 seq# 30708 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
      Current log# 4 seq# 30708 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
    Fri Nov 28 10:44:43 2008
    ARC0: Evaluating archive   log 1 thread 1 sequence 30707
    ARC0: Beginning to archive log 1 thread 1 sequence 30707
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30707.001'
    Fri Nov 28 10:45:07 2008
    ARC1: Evaluating archive   log 1 thread 1 sequence 30707
    ARC1: Unable to archive log 1 thread 1 sequence 30707
          Log actively being archived by another process
    Fri Nov 28 10:45:09 2008
    ARC0: Completed archiving  log 1 thread 1 sequence 30707
    Fri Nov 28 11:13:20 2008
    Completed checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
    Fri Nov 28 11:45:09 2008
    Beginning log switch checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
    Thread 1 advanced to log sequence 30709
      Current log# 3 seq# 30709 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
      Current log# 3 seq# 30709 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
    Fri Nov 28 11:45:09 2008
    ARC0: Evaluating archive   log 4 thread 1 sequence 30708
    ARC0: Beginning to archive log 4 thread 1 sequence 30708
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30708.001'
    Fri Nov 28 11:45:13 2008
    ARC1: Evaluating archive   log 4 thread 1 sequence 30708
    ARC1: Unable to archive log 4 thread 1 sequence 30708
          Log actively being archived by another process
    Fri Nov 28 11:45:29 2008
    ARC0: Completed archiving  log 4 thread 1 sequence 30708
    Fri Nov 28 12:15:17 2008
    Completed checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
    Fri Nov 28 12:50:23 2008
    Beginning log switch checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
    Thread 1 advanced to log sequence 30710
      Current log# 2 seq# 30710 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30710 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 12:50:24 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30709
    ARC0: Beginning to archive log 3 thread 1 sequence 30709
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30709.001'
    Fri Nov 28 12:50:40 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30709
    Fri Nov 28 13:20:29 2008
    Completed checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
    Fri Nov 28 14:27:01 2008
    Beginning log switch checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
    Thread 1 advanced to log sequence 30711
      Current log# 1 seq# 30711 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
      Current log# 1 seq# 30711 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
    Fri Nov 28 14:27:01 2008
    ARC0: Evaluating archive   log 2 thread 1 sequence 30710
    ARC0: Beginning to archive log 2 thread 1 sequence 30710
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30710.001'
    Fri Nov 28 14:27:25 2008
    ARC0: Completed archiving  log 2 thread 1 sequence 30710
    Fri Nov 28 14:57:08 2008
    Completed checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
    Fri Nov 28 15:57:26 2008
    Beginning log switch checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
    Thread 1 advanced to log sequence 30712
      Current log# 4 seq# 30712 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
      Current log# 4 seq# 30712 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
    Fri Nov 28 15:57:26 2008
    ARC0: Evaluating archive   log 1 thread 1 sequence 30711
    ARC0: Beginning to archive log 1 thread 1 sequence 30711
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30711.001'
    Fri Nov 28 15:57:40 2008
    ARC1: Evaluating archive   log 1 thread 1 sequence 30711
    ARC1: Unable to archive log 1 thread 1 sequence 30711
          Log actively being archived by another process
    Fri Nov 28 15:57:40 2008
    ARC0: Completed archiving  log 1 thread 1 sequence 30711
    Fri Nov 28 16:27:31 2008
    Completed checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
    Fri Nov 28 17:02:12 2008
    Beginning log switch checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
    Thread 1 advanced to log sequence 30713
      Current log# 3 seq# 30713 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
      Current log# 3 seq# 30713 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
    Fri Nov 28 17:02:12 2008
    ARC0: Evaluating archive   log 4 thread 1 sequence 30712
    ARC0: Beginning to archive log 4 thread 1 sequence 30712
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30712.001'
    Fri Nov 28 17:02:25 2008
    ARC0: Completed archiving  log 4 thread 1 sequence 30712
    Fri Nov 28 17:32:17 2008
    Completed checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
    Fri Nov 28 18:00:19 2008
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
    Fri Nov 28 18:00:19 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
    Fri Nov 28 18:00:20 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
    Fri Nov 28 18:00:21 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
    Fri Nov 28 18:00:23 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
    Fri Nov 28 18:00:24 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
    Fri Nov 28 18:00:26 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
    Fri Nov 28 18:00:27 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
    Fri Nov 28 18:00:28 2008
    /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Fri Nov 28 20:00:17 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
    Fri Nov 28 20:00:18 2008
    /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
    Fri Nov 28 20:00:36 2008
    Beginning log switch checkpoint up to RBA [0x77fa.2.10], SCN: 0x0000.3c0cf8e3
    Thread 1 advanced to log sequence 30714
      Current log# 2 seq# 30714 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
      Current log# 2 seq# 30714 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
    Fri Nov 28 20:00:36 2008
    ARC0: Evaluating archive   log 3 thread 1 sequence 30713
    ARC0: Beginning to archive log 3 thread 1 sequence 30713
    Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30713.001'
    Fri Nov 28 20:00:44 2008
    ARC1: Evaluating archive   log 3 thread 1 sequence 30713
    ARC1: Unable to archive log 3 thread 1 sequence 30713
          Log actively being archived by another process
    Fri Nov 28 20:00:47 2008
    ARC0: Completed archiving  log 3 thread 1 sequence 30713
    Dump file f:\oracle\prd\saptrace\background\alert_prd.log
    Mon Dec 01 09:03:16 2008
    ORACLE V9.2.0.8.0 - Production vsnsta=0
    vsnsql=12 vsnxtr=3
    Windows 2000 Version 5.2 Service Pack 2, CPU type 586
    Mon Dec 01 09:03:16 2008
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    SCN scheme 2
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up ORACLE RDBMS Version: 9.2.0.8.0.
    System parameters with non-default values:
      processes                = 120
      sessions                 = 150
      timed_statistics         = TRUE
      shared_pool_size         = 822083584
      large_pool_size          = 8388608
      java_pool_size           = 8388608
      enqueue_resources        = 8000
      filesystemio_options     = SETALL
      control_files            = C:\oracle\PRD\origlogA\ctrlPRD.ctl, F:\oracle\PRD\saparch\ctrlPRD.ctl, E:\oracle\PRD\sapdata1\ctrlPRD.ctl
      db_block_checksum        = FALSE
      db_block_size            = 8192
      db_cache_size            = 1073741824
      compatible               = 9.2.0
      log_archive_start        = TRUE
      log_archive_dest         = F:\oracle\PRD\saparch
      log_buffer               = 4194304
      log_checkpoint_interval  = 0
      db_files                 = 254
      db_file_multiblock_read_count= 8
      log_checkpoints_to_alert = TRUE
      control_file_record_keep_time= 30
      dml_locks                = 2500
      row_locking              = always
      replication_dependency_tracking= FALSE
      transactions_per_rollback_segment= 20
      transaction_auditing     = FALSE
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      undo_retention           = 43200
      remote_os_authent        = TRUE
      db_domain                = world
      service_names            = PRD.world, PRD.WORLD
      hash_join_enabled        = FALSE
      background_dump_dest     = F:\Oracle\PRD\saptrace\background
      user_dump_dest           = F:\Oracle\PRD\saptrace\usertrace
      max_dump_file_size       = 1240
      core_dump_dest           = F:\Oracle\PRD\saptrace\background
      optimizer_features_enable= 9.2.0
      sort_area_size           = 0
      sort_area_retained_size  = 0
      db_name                  = PRD
      open_cursors             = 800
      optimizer_mode           = CHOOSE
      sortelimination_cost_ratio= 10
      btree_bitmap_plans     = FALSE
      eliminatecommon_subexpr= FALSE
      pushjoin_predicate     = FALSE
      optimizerjoin_sel_sanity_check= TRUE
      parallel_threads_per_cpu = 4
      optimizer_index_cost_adj = 10
      optimizer_index_caching  = 50
      optimpeek_user_binds   = FALSE
      pga_aggregate_target     = 536870912
      optimizeror_expansion  = DEPTH
    PMON started with pid=2, OS id=17336
    DBW0 started with pid=3, OS id=16972
    LGWR started with pid=4, OS id=16900
    CKPT started with pid=5, OS id=17236
    SMON started with pid=6, OS id=17436
    Regards
    Pieter

  • Database objects behind stats on "Concurrent Processing Dashboard" ?

    I want to define a new report in oem grid about concurrent processing on a ebs target. i saw that there is some minimal stats in the 'Concurrent Processing Dashboard'. What are the DB tables used to generate this screen ?
    By the way there is a bug, all number are displayed like : 2.000000 ....
    Thanks

    Hi,
    What if both nodes have all modules active? (host,admin,cm,forms,web) except the database of courseDo you mean to say all application services are up and the database is down? If yes, then there must be something wrong with the dashboard itself (if the database is down, you will not be able to access the application or even access OAM).
    Thanks,
    Hussein

  • Querying the toplink cache under high-load

    We've had some interesting experiences with "querying" the TopLink Cache lately.
    It was recently discovered that our "read a single object" method was incorrectly
    setting query.checkCacheThenDB() for all ReadObjectQueries. This was brought to light
    when we upgraded our production servers from 4 cores to 8. We immediatly started
    experiencing very long response times under load.
    We traced this down to the following stack: (TopLink version 10.1.3.1.0)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
    at java.lang.Object.wait(Object.java:474)
    at oracle.toplink.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:179)
    - locked <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
    at oracle.toplink.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:167)
    at oracle.toplink.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:122)
    at oracle.toplink.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:31)
    at oracle.toplink.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:530)
    at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:412)
    at oracle.toplink.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:223)
    at oracle.toplink.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:504)
    at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:564)
    at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
    at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
    We moved the query back to the default, query.checkByPrimaryKey() and this issue went away.
    The bottleneck seemed to stem from the read lock on the CacheKey from IdenityMapKeyEnumeration
    public Object nextElement() {
    if (this.nextKey == null) {
    throw new NoSuchElementException("IdentityMapKeyEnumeration nextElement");
    // CR#... Must check the read lock to avoid
    // returning half built objects.
    this.nextKey.checkReadLock();
    return this.nextKey;
    We had many threads that were having contention while searching the cache for a particular query.
    From the stack we know that the contention was limited to one class. We've since refactored that code
    not to use a query in that code path.
    Question:
    Armed with this better knowledge of how TopLink queries the cache, we do have a few objects that we
    frequently read by something other than the primary key. A natural key, but not the oid.
    We have some other caching mechanisms in place (JBoss TreeCache) to help eliminate queries to the DB
    for these objects. But the TreeCache also tries to acquire a read lock when accessing the cache.
    Presumably a read lock over the network to the cluster.
    Is there anything that can be done about the read lock on CacheKey when querying the cache in a high load
    situation?

    CheckCacheThenDatabase will check the entire cache for a match using a linear search. This can be efficient if the cache is very large. Typically it is more efficient to access the database if your cache is large and the field you are querying in the database is indexed in the table.
    The cache concurrency was greatly improved in TopLink 11g/EclipseLink, so you may wish to try it out.
    Supporting indexes in the TopLink/EclipseLink cache is something desirable (feel free to log the enhancement request on EclipseLink). You can simulate this to some degree using a named query and a query cache.
    -- James : http://www.eclipselink.org

  • Concurrent processing status down in OAM

    Hi ,
    From OAM->Applications Dashboard ->Overview
    the status for concurrent processing shows down but infact it is up
    I also checked from OAM->Hosts->view configuration
    whereas here the status for concurrent processing is up.
    thanks all in advance..
    applications version: 11.5.10.2

    Hi,
    Please see if these documents help.
    Note: 405375.1 - OAM Reports 'Internal Concurrent Manager Found Node To Be Down.' When Node is Up
    Note: 419823.1 - OAM: Web Components Status: Shows Down Even When The System Works Fine
    Note: 309271.1 - Web Components Status in OAM Shows All Down Even When the System Works Fine
    Note: 434569.1 - OAM application dashbord shows all the web components with status down whereas they are up
    Note: 287415.1 - Host Status In Application Dashboard Overview Tab Shown As Down in OAM
    Regards,
    Hussein

  • MDB topic listener and concurrent processing

    Hello to everybody.
    I ve prototyped simple db publisher ,which publishes changed data from database to JMS topic. Then I have my MDB which listens on JMS topic.
    I setup and implemented all correctly and it worked. There was 1 consumer = 1 mdb instance. Then I tried to do load testing , So I triggered 8000 changes from database. The db adapter did the job and published the 8000 messages on the JMS topic in a while. To mimic processing in MDB I put the sleep(10seconds) in MDB.
    MDB topic processing seems to me as not concurrent one. Why? I observed 10 message decrease with 5-10 seconds.
    What I expected was: concurrent processing of 8000 JMS messages by 1 MDB (so MDB is poolable component isnt it?).
    So for 500 instances of topic MDB and 8000 messages on the JMS topic, I expected at least 8000/500 X 10 = 160 seconds to process message load.
    I am using WLS 816.
    Can You give me some hits.
    Thanks
    Roman

    I did the same test on Weblogic 10 and it worked as expected. MDB durable subscriber consumed 8000 messages (each msg processing had 10 secs sleep inside.) within 2 minutes. So it is ok.
    But why it did not worked on WLS 8.1? Is this a bug ?

  • Sync messages failed due to high load

    Hi!
    On our test system we had a very high load of async messages for 3 - 4 hours. The async messages were processed successfully but all sync messages failed during this time frame! As soon as the high load of async messages stopped, the sync messages started executing successfully again.
    Has anybody also experienced this kind of problem and what can be done to avoid this issue? Thanks for any input and suggestions.
    Regards, Tanja

    hey Tanja,
    i had a similar issue when testing the box with Load runner, that a simple sync scenario fails when the size of payload or attachment is high, the thing that i did is just increased the virtual memory ,
    You can also check the trace files to figure out problem ?
    This might not be a definitive answer , hope this helps,
    Vara
    [PLEASE REWARD FOR POSTING]

  • Performance Bottleneck at HIgh Loads

    Hi,
    We are currently load testing our web application and have been facing an issue that has ben bothering us for sometime now.
    If we load test the application at high load for sustained amounts of time, (around 2 hours), the performance suddenly dips to Zero hits/sec, Nothing is processed , no output generated. It seems as though all the processes some how are waiting for a resource to get free.After sometime the performance goes up again and again there is a race condition because of which the performance goes down again. Here on the dipping becomes highly erratic.
    We are trying to find out if this is a known bug with the oracle jdbc driver. "oracle.jdbc.driver.OracleDriver" .
    Would appreciate if anybody could provide any inputs as to why this is happening and how we could alleviate the situation.
    Thanks in advance.
    Deepanjan

    Hi I vaguely remember something like this.... but I thought it typically occurred during lots of heavy insert/update activity
    which version of the jdbc drivers are you using.... are you using thin/thick
    Have you checked the SQL or the database to determine whether any locks are blocking activity or whether you have large database wait issues....
    Dom

Maybe you are looking for

  • Audit Data in BI Publisher Standalone

    Hi: I'm trying to audit who/what/when BI reports are run and I've got the BI Publisher user from XDO_USER_NAME but I need to find out what report is being run. I don't know of an existing XDO parameter or anything else that has that. So I'm trying to

  • Podcast App in IOS 8.0

    Listening to Podcasts using the Podcast APP continues to be a struggle and now even worse in IOS 8.0 Anyone have any suggestions how to make listening to Podcast as easy as it used to be. Everytime I go to download my daily podcasts it is a gong show

  • Safari white screen

    Just today, out of no where, sarafi will only show me a white page, regardless of what address I type in the address bar.  Brand new Macbook Pro for Christmas.  Worked yesterday, not today.  I googled it and based on the hits that came back I thought

  • Web safe fonts - Baby Theme site 1.0

    Hello I'm using the Baby Theme template from 1.0, does anyone know why my content font is being displayed in IE in the Times New Roman font when I have specifically changed the content font to Copper Black in iWeb. Can I change the buttons (for blog,

  • RAID for speed....for use with home recording

    Hi all, I want to get my G5 dual 2.0 faster for home recording with logic 8 using another hard drive. I was told RAID is the way to go? I am NOT familiar with RAID at all or how to do this but I am very good at picking this stuff up. Unfortunately th