High Load on Concurrent Processing node
Hello :
version :11.5.10.CU2 / Linux -86
Recently I have started to notice that my concurrent processing node is loaded heavily between 3.00 AM to 6.00 AM and the load is normal. I need to find out the root cause of this spike. Is there a log that I can look for in concurrent processing node that will give me a clue.
Regards,
Bala
Look at the longest running requests during that time, then enable sql trace so you can start to tune them, or work with Oracle Support if they are standard programs running with suboptimal execution plans.
SELECT CR.REQUEST_ID "Request", U.USER_NAME "User", CP.USER_CONCURRENT_PROGRAM_NAME "Program",
TO_CHAR(CR.REQUESTED_START_DATE,'DD-MON-YY HH24:MI:SS') "Requested Start",
TO_CHAR(CR.ACTUAL_START_DATE,'DD-MON-YY HH24:MI:SS') "Actual Start",
ROUND((CR.ACTUAL_COMPLETION_DATE-ACTUAL_START_DATE)*24*60*60,0) "Duration (sec)",
ROUND((CR.ACTUAL_COMPLETION_DATE-ACTUAL_START_DATE)*24*60,2) "Duration (min)",
CR.PHASE_CODE "Phase", CR.STATUS_CODE "Status", CR.ARGUMENT_TEXT "Arguments"
FROM FND_CONCURRENT_REQUESTS CR,
FND_USER U,
FND_CONCURRENT_PROGRAMS_TL CP,
FND_RESPONSIBILITY_TL R
WHERE CR.REQUESTED_BY = U.USER_ID
AND CR.PROGRAM_APPLICATION_ID = CP.APPLICATION_ID
AND CR.CONCURRENT_PROGRAM_ID = CP.CONCURRENT_PROGRAM_ID
AND CP.LANGUAGE='US'
AND CR.RESPONSIBILITY_APPLICATION_ID = R.APPLICATION_ID
AND CR.RESPONSIBILITY_ID = R.RESPONSIBILITY_ID AND R.LANGUAGE='US'
AND (TO_CHAR(CR.ACTUAL_START_DATE,'DD-MON-YYYY HH24:MI') BETWEEN '30-JUN-2011 03:00' and '30-JUN-2011 06:00')
ORDER BY ROUND((CR.ACTUAL_COMPLETION_DATE-ACTUAL_START_DATE)*24*60*60,0) DESC
Similar Messages
-
Hi,
I'm using ArchLinux 64bit, with KDE 4.3.4 on a sony vaio, everything work fine, but after some normal work I see that process X takes more then 90% of CPU and never slow down. Even if I closed all my application it stays at 90%. I don't know how to see which application could cause this.
As first impression it seems when I use samba adn move some file from my machine to a windows machine, but today it takes high load cpu without using samba, so I don't know what else do, any good advice?
thanks a lot
cesareHi thank you very much for your reply, I'm sorry to reply just now.
Anyway I have a Sony Vaio VGN.SR21M, using opensource driver xf86-video-radeonhd with hardware acc on. My graphics card is a Radeon3400HD
I used the closed driver till I switch to Xorg 1.7 and so I can't using it anymore so I switched to the free one.
Should I disable desktop effects?
Thanks again
Cesare
Last edited by cesare (2010-01-25 08:48:01) -
Better utilization of cpus and memory - Concurrent processing tier
I have a three node 11.5.10.2 setup:
Node1 - Database only (10.1.0.5)
Node2 - Admin, Forms, Web
Node3 - Concurrent processing
My concurrent processing node has lot of free cpus and memory
Cpus: Average utilization 20%
Memory : Average utilization 15%
The highest spikes I've had are:
Cpus- 24% utilization
Memory - 20% utilization
What I can do on the concurrent processing tier that would make better use of the cpus/memory resources and improve performance?
I have 2 dual core Intel(R) Xeon(R) processors and 16 GB Physical memory on the concurrent processing server.
Thanks & regards,Hi Asif,
You are seeing a manifestation of the old myth from the earlier part of this decade that you need many middle tier servers to effectively interact with users in a busy ERP environment. In reality, the middle tier processes which are normally web, forms, and concurrent managers typically consume only memory, and maybe10-20 percent CPU utilization of a modern quad core Xeon/Opteron Linux server. I have seen several 200 plus user single node 11i installations that handled the load just fine. In my experience, 95-97% of the ERP CPU utilization comes from database sessions, not from middle tier processes. Occasionally, a forms session goes to a runaway, or a ar60run process runs at 100% CPU while the request is running, but if you follow the Oracle recommendations of limiting fnd_concurrent_requests to 50k rows or less, you will not see much CPU utilization from FNDLIBR processes.
Most 32 GB 2 socket quad core servers can support 200 plus forms user connections plus host the concurrent manager as well. I would only add additional web/forms nodes after you start paging on the middle tier box. Having concurrent managers separated on a 10-20 percent utilized server is sort of a waste of a server.
Regards,
-Michael -
One of 4 node RAC always have higher load averages and higher than others
Hello,
We have a 4 node rac, 9208 on linux 4. When viewing top, we noticed the same one node always have a higher load average than the other 3 nodes. Is this normal. Loan balance is working fine but this one node always have higher load average. This is the node where we do the rac installation. Thank you.I do not remember what is the default for clb_goal (client load balancing) for 9i but 10g is LONG.
check it
select clb_goal from dba_services where name = <service name>
you may have to change from LONG to SHORT OR SHORT to LONG depending your connection types.
dbms_service.MODIFY_SERVICE(‘<service>’,clb_goal=> dbms_service.CLB_GOAL_LONG);
Read the following article.
http://www.databasejournal.com/features/oracle/article.php/3659411/Oracle-RAC-Administration---Part-15-Connection-Load-Balancing-and-FAN.htm -
High load and high CPU caused by perl process
Using: Oracle 10.2.0.1.0
OS: Red Hat ES 4
Since this morning I have high load on one of my prod database server, this server has 3 databases of similar size.
High load is caused by perl process, one process has been running for over 75 min’s. process info
14302 ? R 74:50 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.pl
26530 ? R 23:36 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.pl
31152 ? R 7:58 /opt/oracle/product/10.2.0/perl/bin/perl /opt/oracle/product/10.2.0/sysman/admin/scripts/segAdv.plSearching segAdv.pl on google, Metalink and this forum gives me ZERO results.
Any idea why this is running and cuasing this issue now (for last 8 months, I had not simlar perl issue), can I stop it without kill –9, maybe with emctl stop dbconsole.
Thanks in advance for any help.Hi Lee,
You say this is a core router, but as people use this terms to mean different things, are there any directly connected hosts? If there are one thing to check for is hosts sending multicast traffic with TTL=1. If you run the command show ip traffic | in bad hop and look at the rate the "bad hop count" is increasing. If I had a £ for every time I've seen that problem.... I couldn't retire just yet, but at least I could go buy myself a beer or two.
If you're seeing the hop count increasing at a rapid rate there are a couple of options:
Take a read of Troubleshooting high CPU on a 6500 with sup720 starting from the section Tools used to determine the source of the traffic and identify the hosts sending multicast with TTL=1.
If the Catalyst 6500 use PFC 3B add the command mls rate-limit all ttl-failure 100 10 to drop the multicast traffic in hardware.
The first option will take you longer, but at least you identify the cause and can advise the application owner why they should change their applications behaviour.
Regards -
2008 R2/ Win7 Offline Files Sync causing High Load on server
Hi Everyone,
I have recently been investigating extremely high CPU Usage from the System Process on my company's main File Cluster.
We managed to track SRV2.sys Threads causing high CPU load within the system process, but was having issues identifying why this was the case.
As per Microsoft's direction via support call, we have installed the latest SRV2.sys Hotfixes, but this does not appear to have allivated the issue we are experiencing. We have also added more CPU and Memory into both nodes, which has not helped either.
We have since managed to create a system dump and is being sent to MS Support for analysis.
I have noticed the following that appears to happen on our cluster:
Whenever our CAD/Design department run certain functions within their apps running on a windows 7 client (apps include MicroStation, Revit, AutoCAD etc) we see a massive spike and flatline of the system process.
We found several users with Windows 7 clients that have Configured Offline File to sync an entire Network Volume (some volumes are 2TB Plus, so would never fit on a users computer anyway, i was quite shocked when I found this). How we spotted this was through
Resource Monitor, the System Process was trolling through all the folders and files in a given volume (it was "reading every single folder). Now, while this was the system process, we could identify the user by using the Open Files view in Server Manager's
Share and Storage Management tool.
I have done a fair bit of research and found that a lot of CAD/Drawing applications in the market have issues with using SMB2 (srv2.sys). When reviewing the products that we use, I noticed that a lot of these products actually recommended disabling SMB2
and reverting to SMB1 on File Server and/or the clients.
I have gone down the path of disabling SMB2 on all Windows 7 clients that have these CAD Applications installed to assist with lowering the load (our other options are to potentially shift the CAD Volumes off our main file cluster to further isolation these
performance issues we have been experiencing.) We will be testing this again tomorrow to confirm that the issue is resolved when mass amounts of CAD users access data on these volumes via their CAD Application.
We only noticed the issue with Offline Files today with trying to sync an ENTIRE Volume. My questions are:
Should Offline File sync's really cause this much load on a File Server?
Would the the size of the volume the sync is trying to complete create this additional load within the system process?
What is the industry considered "Best Practice" regarding Offline Files setup for Volumes which could have up to 1000+ users attached? (My personal opinion is that Offline Files should only be sync of user "Personal/Home" Folders
as they users themselves have a 1 to 1 relationship with this data.)
Is there an easier way to identify what users have Offline Files enabled and "actually being used" (from my understanding, Sync Center and Offline Files are enabled by default, but you obviously have to add the folders/drives you wish to sync)?
If I disable the ability for Offline Files on the volumes, what will the user experience be when/if they try to sync their offline files config after this has been disabled on the volume?
Hoping for some guidance regarding this setup with Offline Files.
Thanks in Advance!
SimonHi Everyone,
Just thought I would give an update on this.
While we're still deploying http://support.microsoft.com/kb/2733363/en-us to
the remainder of our Windows 7 SP1 fleet, according to some Network Traces and XPerf that were sent to MS Support, it looks as though the issu with the Offline Files is now resolved.
However, we are still having some issues with High CPU with the System Process in particular. Upon review of the traces, they have found a lot of ABE related traffic, leading them to believe that the issue may also be caused by ABE Access on our File Cluster.
We already have the required hotfix for ABE on 2008 R2 installed (please see
http://support.microsoft.com/kb/2732618/en-us), although we have it set to a certain value that MS believe may be too high. Our current value is set to "5", as that is the lowest level of any type of permission is set (i.e. where the lowest level of inheritance
is broken).
I think I will mark this as resolved regarding the Offline Files issue (as it did resolve the problem with the Offline Files)...
Fingers crossed we can identify the rest of the high load within the System Process!!! -
COncurrent process fails with warning for OPP
Concurrent process generates xml output (version 5.6.2) .
When it tries to publish it , the log suggests checking out the OPP log.
The OPP log has a series of java entries for the request, including one of the type
Caused by: oracle.xdo.parser.v2.XPathException: Expected 'QNAME' instead of ''.
header of the OPP log says
Resolution History
18-OCT-06 20:36:05 GMT
Can you easily recover from, bypass or work around the problem? = NO
Does your system or application continue normally after the problem occurs? =
YES
Are the standard features of the system or application still available; is the
loss of service minor? = YES
### Informational Only:Warning Presented to Customer ###
You chose 'Other technical issues with this product' from the Type of Problem
list. This is a generic Service Request form which will require additional time
to be routed appropriately. You may be able to expedite the resolution of your
Service Request by doing the following: 1. Use your browser’s back button to
return to the Create a SR – Brief Description page. 2. Select an option from
the Type of Problem list that more closely describes the problem.
### Requested files:ACT(Note 183274.1),RDA (Note 175853.1). Files to be loaded.
None
### Detailed Problem Statement ###
Concurrent process for XML Publisher fails with warning. Cannot see Diagnostics
--> XML output completely. Only first page visible. Buttons and Menu items
disabled not allowing to view the whole XML file to troubleshoot for
correct/incorrect tags.
Attached are OPP Manager log file, Screen shot of 'Preview' output jjst before
submitting request. Also log file of failed concurrent request which contains
standard error " See OPP Service log file for more".
### Steps to Reproduce Problem ###
Registered a concurrent process with XML template , and running the process
from Receivables Manager responsibility.
Ran concurrent process for 1) Standard RAXINV report , with output method as
'XML'. 2) Customised RAXINV report, with output method as 'XML'
### Instance Name(s) and Type of System(s) Where Error Occurs ###
Lunix, 11.5.10 on 10g
### Recent Changes to this Environment ###
None
### Workarounds Used ###
None
### How is this Issue Impacting Your Business ###
Business is readying for CRP3, so would like to have reports ready by then.
Contact me via : MetaLink
18-OCT-06 20:48:07 GMT
The customer : [email protected] : has uploaded the following file via MetaLink:
C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\XML_Pub1.bmp
18-OCT-06 20:49:14 GMT
The customer : [email protected] : has uploaded the following file via MetaLink:
C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\XMLPublisher_java.t
xt
18-OCT-06 21:17:26 GMT
Hi Bashobi,
Thank you for using MetaLink. We are currently reviewing/researching the situation and will update the SR as soon as we have relevant information
Best Regards,
Bill
Oracle Support Services
STATUS
=======
@WIP -- Work In Progress
18-OCT-06 21:18:04 GMT
Email Update button has been pressed: Sending email to [email protected].
18-OCT-06 21:21:45 GMT
ISSUE CLARIFICATION
====================
On 11.5.10.2 in Production:
Find concurrent process for XML Publisher fails with warning. Cannot see Diagnostics
--> XML output completely. Only first page visible. Buttons and Menu items
disabled not allowing to view the whole XML file to troubleshoot for
correct/incorrect tags.
EXPECTED BEHAVIOR
Expect the processes run successfully
STEPS
The issue can be reproduced at will with the following steps:
1. Registered a concurrent process with XML template .
2. Run the process from Receivables Manager responsibility.
3. Ran concurrent process for
1) Standard RAXINV report , with output method as 'XML'
2) Customised RAXINV report, with output method as 'XML'
BUSINESS IMPACT
The issue has the following business impact:
Business is readying for CRP3, so would like to have reports ready by then.
ACTION PLAN
============
Hi Bashobi,
1. Can you reproduce the issue from sysadmin responsibility with Active users report? if yes
please upload OPP log file after reproduce it and upload the request log file.
Thanks.
18-OCT-06 23:09:02 GMT
New info : [email protected] : Did the same as suggested, Request
completes successfully.
But what is the relation between Active Users report and a report sent via XML
Publisher?
19-OCT-06 15:54:26 GMT
ACTION PLAN
============
Hi Bashobi,
1. When you reproduce the issue from sysadmin responsibility with Active users report change the format out put to XML,
Save your change:
2. Upload OPP manager log file and upload the request log file.
Thanks,
Bill.
19-OCT-06 15:54:33 GMT
Email Update button has been pressed: Sending email to [email protected].
19-OCT-06 16:42:23 GMT
New info : [email protected] : Bill,
No, I could not reproduce the problem with Active Users program.
Attaching OPP manager log in the next update. Bashobi
19-OCT-06 17:05:07 GMT
New info : [email protected] : Bill, Attaching internal log and manager
log
Bashobi
19-OCT-06 17:06:08 GMT
The customer : [email protected] : has uploaded the following file via MetaLink:
C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\OPP_internal_log.tx
t
19-OCT-06 17:07:10 GMT
The customer : [email protected] : has uploaded the following file via MetaLink:
C:\Documents_and_Settings\broy\BareEscentuals\Reports\Errors\OPP_manager_log.txt
19-OCT-06 18:14:32 GMT
UPDATE
=======
Hi Bashobi,
Thank you for providing the requested information.
Best Regards,
Bill
STATUS
=======
@WIP -- Work In Progress
19-OCT-06 18:14:36 GMT
Email Update button has been pressed: Sending email to [email protected].
19-OCT-06 18:59:25 GMT
DATA COLLECTED
===============
SCREEN SHOTS
XML_Pub1.bmp:
Error:
Jave.sql SQL Exception No corresponding LOB data found: select File_data_dbms_lob...........
LOG FILE
1. XMLPublisher_java.txt:
[10/18/06 11:16:57 AM] [5670:RT277479] Completed post-processing actions for request 277479.
[10/18/06 11:29:06 AM] [OPPServiceThread1] Post-processing request 277488.
[10/18/06 11:29:06 AM] [5670:RT277488] Executing post-processing actions for request 2
77488.
[10/18/06 11:29:06 AM] [5670:RT277488] Starting XML Publisher post-processing action.
[10/18/06 11:29:06 AM] [5670:RT277488]
Template code: XXBE_TEST_INV
Template app: AR
Language: en
Territory: US
Output type: null
[10/18/06 11:29:07 AM] [UNEXPECTED] [5670:RT277488] java.lang.reflect.InvocationTarg
etException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10/18/06 11:29:07 AM] [5670:RT277488] Completed post-processing actions for reque
st 277488.
2. OPP_manager_log.txt:
[10/19/06 5:22:36 AM] [main] Starting GSF service with concurrent process id = 5688.
[10/19/06 5:22:36 AM] [main] Initialization Parameters: oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5
[10/19/06 5:22:37 AM] [Thread-12] Service thread starting up.
[10/19/06 5:22:37 AM] [Thread-13] Service thread starting up.
ISSUE VERIFICATION
=================
Issue veryfied by the screen shots: XML_Pub1.bmp which shows:
Jave.sql SQL Exception No corresponding LOB data found: select File_data_dbms_lob...........
ACTION PLAN
============
Hi Bashobi,
1. Since you can not reproduce the issue from sysadmin responsibility can we say the issue specific to Receivables Manager
responsibility<INV xml report>?
2. You said before: "Concurrent process for XML Publisher fails with warning. Cannot see Diagnostics
--> XML output completely. Only first page visible. Buttons and Menu items
disabled not allowing to view the whole XML file to troubleshoot for
correct/incorrect tags.":
Is the above mean :
-The xml publisher report completed but you can not view the whole XML file
-Or the xml publisher report error out with "Jave.sql SQL Exception No corresponding LOB data found: select File_data_dbms_lob.."
Thanks.
19-OCT-06 18:59:32 GMT
Email Update button has been pressed: Sending email to [email protected].
20-OCT-06 21:36:13 GMT
Email Update button has been pressed: Sending email to [email protected].
24-OCT-06 21:50:06 GMT
New info : [email protected] : Hi,
We are in the process of upgrading XML Publisher at the moment. I would like to
wait a day or two to test the report with the upgrade.
Thanks,Bashobi
26-OCT-06 20:15:07 GMT
New info : [email protected] : Hi,
Our DBA has applied the 5.6.2 patch on our dev instance. My concurrent process
runs to a warning.
The warnings in the OPP manager log file say
Template code: BE_RAXINV
Template app: XXBE
Language: en
Territory: US
Output type: PDF
[10/26/06 12:50:13 PM] [UNEXPECTED] [5849:RT281617]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Any idea?Hi All,
I have ran into the same issue.
"The Output Post-processor is running but has not picked up this request.
Any ideas.
Thanks
Ravi -
How to configure number of concurrent processes in XI?
Hi experts,
I have a simple interface:
FILE -> XI -> FILE
The input file is a huge file containing thousands of records.
XI will split the huge input files, so that each record gets written to one file each.
Owing to the huge number of files being generated, I have many messages in status = "To Be Delivered" in the RWB.
When I do a filter for status = "Delivering", I see 4 messages in that status. (Of course the message IDs will differ from time to time, but at any point in time, I can only see 4 messages with the status = "Delivering").
How can I increase the number of concurrent processes that produces the output files?
Please advise.
Thanks.
RonRon,
From this perfromance tuning guide , page 12 ,
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/70ada5ef-0201-0010-1f8b-c935e444b0ad
<i>The Adapter Framework requires a set of fixed application threads within the J2EE Engine for proc-essing different internal messaging queues (AFW queue, BC Adapter, CIDX Adapter, and so on). Ad-ditionally, certain channel configurations also require a fixed set of application threads (for example, RFC sender channel, File adapter channels). This may require a higher number of application threads. Therefore, within the J2EE Engine configuration (configtool or Visual Admin), go to:
Global Configuration -> Server -> Kernel -> ApplicationThreadManager
Set the MaxThreadCount property to 300</i>
Not sure if this is the exact property but this is sure worth giving a shot.
Regards
Bhavesh -
Strange Oracle Behaviour - only month end or high load periods
Hi SAP Gurus
In one of our systems I am expreriencing a strange behaviour from Oracle, in that it seems to be in reconnect, but yet it is not, it resets and carries on but do produce Dump and SM21 log messages, the dump is not really helpfull as it is the same dump but with different program everytime.
I have looked in oracle alert log and found no event tying back to the SM21 log message or dump.
The following is what I found in the developer trace of one of these insident, but there are many, about 125 - 200 for the day during high load:
Fri Nov 28 10:05:55 2008
C *** ERROR => Failed to register application info: orc=3113
[dboci.c 3042]
C *** ERROR => Error 3114 in stmt_fetch()
[dbsloci.c 13001]
C *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
[dbsloci.c 13002]
C sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB
C SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
C NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
C , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
C "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
C GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
C INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
C ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
C TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
C DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
C PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
C "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
C SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
C FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
C sc_p=045B18B4,no=29,idc_p=00000000,con=0,act=1,slen=1291,smax=1536,#vars=4,stmt=34DA78C8,table=CATSDB
C prep=0,lit=0,nsql=0,lobret=0,#exec=1,dbcnt=0,upsh_p=00000000,ocistmth_p=345E20EC
C IN : cols=4,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=17,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=3470A8D8
C lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C OUT: cols=105,rmax=58,xcnt=58,rpc=1,rowi=0,rtot=1,upto=-1,rsize=895,vmax=128,bound=1,iobuf_p=0E0A4EC8,vda_p=34F8F7E8
C lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C SELECT "MANDT" , "COUNTER" , "PERNR" , "WORKDATE" , "SKOSTL" , "LSTAR" , "SEBELN" , "SEBELP" , "SPRZ\
C NR" , "LSTNR" , "RKOSTL" , "RPROJ" , "RAUFNR" , "RNPLNR" , "RAUFPL" , "RAPLZL" , "RKDAUF" , "RKDPOS"\
C , "RKSTR" , "RPRZNR" , "PAOBJNR" , "FUND" , "FUNC_AREA" , "GRANT_NBR" , "S_FUND" , "S_FUNC_AREA" , \
C "S_GRANT_NBR" , "AWART" , "LGART" , "KAPID" , "SPLIT" , "REINR" , "WABLNR" , "VERSL" , "WTART" , "BW\
C GRL" , "WAERS" , "AUFKZ" , "TRFGR" , "TRFST" , "PRAKN" , "PRAKZ" , "OTYPE" , "PLANS" , "KOKRS" , "ME\
C INH" , "TCURR" , "PRICE" , "ARBID" , "WERKS" , "AUTYP" , "HRCOSTASG" , "HRKOSTL" , "HRLSTAR" , "HRFU\
C ND" , "HRFUNC_AREA" , "HRGRANT_NBR" , "BEMOT" , "UNIT" , "STATKEYFIG" , "TASKTYPE" , "TASKLEVEL" , "\
C TASKCOMPONENT" , "BUKRS" , "ERSDA" , "ERSTM" , "ERNAM" , "LAEDA" , "LAETM" , "AENAM" , "APNAM" , "AP\
C DAT" , "WORKITEMID" , "LOGSYS" , "STATUS" , "REFCOUNTER" , "REASON" , "BELNR" , "EXTSYSTEM" , "EXTAP\
C PLICATION" , "EXTDOCUMENTNO" , "TASKCOUNTER" , "CATSHOURS" , "BEGUZ" , "ENDUZ" , "VTKEN" , "ALLDF" ,\
C "OFMNW" , "PEDD" , "AUERU" , "LTXA1" , "LONGTEXT" , "ERUZU" , "CATSAMOUNT" , "CATSQUANTITY" , "ZCAT\
C SPLAN" , "ROUTE" , "KUNNR" , "MVGR1" , "GSBER" , "SNAME" , "MAKTX" , "ZAVGHRS" , "ZQANT" , "ZPERNR" \
C FROM "CATSDB" WHERE "MANDT" = :A0 AND "WORKDATE" = :A1 AND "WERKS" = :A2 AND "STATUS" < :A3;
B ***LOG BYM=> severe DB error 3114 ; work process in reconnect status [dbsh#3 @ 1123] [dbsh 1123 ]
B ***LOG BY4=> sql error 3114 performing FET on table CATSDB [dbtran#10 @ 7292] [dbtran 7292 ]
B ***LOG BY0=> ORA-03114: not connected to ORACLE [dbtran#10 @ 7292] [dbtran 7292 ]
B dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
B RSLT: {dbsl=99, tran=1}
B FHDR: {tab='CATSDB', fcode=225, mode=2, bpb=0, dbcnt=0, crsr=1,
B hold=0, keep=1, xfer=0, pkg=0, upto=0, init:b=0,
B init:p=00000000, init:#=871, wa:p=0XA000DC88, wa:#=871}
B dbtran ERROR LOG (hdl_dbsl_error): DbSl 'FET'
B STMT:
B CRSR: {tab='CATSDB', id=1, hold=0, prop=0x1000, max.in@0=1, fae:blk=1,
B con:id=0, con:vndr=5, val=2,
B key:#=4, xfer=0, xin:#=0, row:#=0, upto=0, wa:p=0XA000DC88}
C *** ERROR => Error 3114 in stmt_fetch()
[dbsloci.c 13001]
C *** ERROR => ORA-3114 occurred when executing SQL statement (parse error offset=0)
[dbsloci.c 13002]
C sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD
C SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
C R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
C sc_p=045B52E4,no=225,idc_p=045C400C,con=0,act=1,slen=184,smax=2048,#vars=3,stmt=35BBB778,table=REPOLOAD
C prep=0,lit=0,nsql=0,lobret=0,#exec=29,dbcnt=0,upsh_p=00000000,ocistmth_p=34B9ADD4
C IN : cols=3,rmax=1,xcnt=0,rpc=0,rowi=0,rtot=0,upto=-1,rsize=45,vmax=32,bound=1,iobuf_p=0E0A4EC8,vda_p=31361B18
C lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C OUT: cols=9,rmax=1,xcnt=1,rpc=1,rowi=0,rtot=1,upto=-1,rsize=56,vmax=32,bound=1,iobuf_p=045DFFA8,vda_p=345C39B8
C lobs=0,lmax=0,lpcnt=0,larr=00000000,lcurr_p=00000000,rret=0
C SELECT "UNAM" , "UDAT" , "UTIME" , "L_DATALG" , "Q_DATALG" , "SDAT" , "STIME" , "MINOR_VERS" , "MAJO\
C R_VERS" FROM "REPOLOAD" WHERE "PROGNAME" = :A0 AND "R3STATE" = :A1 AND "MACH" = :A2 ;
B ***LOG BYM=> severe DB error 3114 ; work process in reconnect status [dbsh#3 @ 1123] [dbsh 1123 ]
B ***LOG BY4=> sql error 3114 performing SEL on table REPOLOAD [dbrepo#6 @ 2657] [dbrepo 2657 ]
B ***LOG BY0=> ORA-03114: not connected to ORACLE [dbrepo#6 @ 2657] [dbrepo 2657 ]
A TH VERBOSE LEVEL FULL
A ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.
A ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.
A ** RABAX: level LEV_RX_VMIT_CLEANUP entered.
A ** RABAX: level LEV_RX_VMIT_CLEANUP completed.
A ** RABAX: level LEV_RX_ROLLBACK entered.
C *** ERROR => OCI-call 'OCITransRollback' failed: rc = 3114
[dboci.c 3127]
B ***LOG BYM=> severe DB error 3114 ; work process in reconnect status [dbsh#3 @ 1123] [dbsh 1123 ]
B Reconnect state is entered by connection:
B 0: name = R/3, con_id = 000000000 state = ACTIVE , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
B hdl_error_on_commit_rollback: DB-ROLLBACK detected RECONNECT state
M ThShortCommit: db unusable
B db_con_reconnect disconnecting connection:
B 0: name = R/3, con_id = 000000000 state = INACTIVE , perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
C Disconnecting from connection 0 ...
C Close user session (con_hdl=0,svchp=045D7FA4,usrhp=345E67A0)
C Detaching from DB Server (con_hdl=0,svchp=045D7FA4,srvhp=045D0174)
C Now I'm disconnected from ORACLE
B db_con_reconnect performing the reconnect for con:
B 0: name = R/3, con_id = 000000000 state = DISCONNECTED, perm = YES, reco = YES, timeout = 000, con_max = 255, con_opt = 255, occ = NO
C got NLS_LANG='AMERICAN_AMERICA.US7ASCII' from environment
C Client NLS settings:
C Logon as OPS$-user to get SAPR3's password
C Connecting as /@PRD on connection 0 (nls_hdl 0) ... (dbsl 640 250407)
C Nls CharacterSet NationalCharSet C EnvHp ErrHp ErrHpBatch
C 0 US7ASCII 1 045CBB38 0E0A4648 0E0A40B0
C Attaching to DB Server PRD (con_hdl=0,svchp=045
SM51 log entries:
09:00:32 MS Q0I Operating system call recv failed (error no. 10054)
09:00:32 DP Q0K Connection to message server (on FCMSPROD) established
09:00:32 DIA 004 040 181JOLENES VA02 BYM SQL error 3114 . Work processes in reconnect status
09:00:32 DP Q1C MsgServer Hardware ID Was Determined
09:00:32 DIA 001 040 181ANNELINEJ VA42 BYY Work process has left reconnect status
09:00:32 DIA 001 040 181ANNELINEJ VA42 AB0 Run-time error "DBIF_REPO_SQL_ERROR" occurred
09:00:32 DIA 014 040 161ADRIV VA02 BYY Work process has left reconnect status
09:00:32 DIA 013 040 140ELECIA BYY Work process has left reconnect status
09:00:32 DIA 003 040 133MIRANDAV KO88 BYY Work process has left reconnect status
09:00:32 DIA 003 040 133MIRANDAV KO88 AB0 Run-time error "DBIF_REPO_SQL_ERROR" occurred
09:00:32 DIA 001 040 181ANNELINEJ VA42 AB1 > Short dump "081128 090032 FCMSPROD 181ANNELINEJ" generated
09:00:32 DIA 003 040 133MIRANDAV KO88 AB1 > Short dump "081128 090032 FCMSPROD 133MIRANDAV " generated
09:00:32 DIA 009 040 120ELIZEP FB01 BYY Work process has left reconnect status
09:00:33 DIA 009 040 120ELIZEP FB01 AB0 Run-time error "DBIF_REPO_SQL_ERROR" occurred
09:00:33 DIA 004 040 181JOLENES VA02 BYY Work process has left reconnect status
09:00:33 DIA 002 040 133KOEKIEJ BYY Work process has left reconnect status
09:00:33 DIA 002 040 133KOEKIEJ AB0 Run-time error "DBIF_REPO_SQL_ERROR" occurred
09:00:33 DIA 004 040 181JOLENES VA02 AB0 Run-time error "DBIF_REPO_SQL_ERROR" occurred
09:00:33 DIA 009 040 120ELIZEP FB01 AB1 > Short dump "081128 090033 FCMSPROD 120ELIZEP " generated
09:00:33 DIA 005 040 181CLIVEB VA02 BYY Work process has left reconnect status
09:00:33 DIA 003 R0Z The update dispatch info was reset
09:00:33 DIA 003 R0Z The update dispatch info was reset
Please can you assist in maybe you have seen this before.
Regards
PieterHi
Yes I have gone through the Oracle Alert log several time, It is huge, but I am pasting an extract for the entire day this was happening into this reply. Oracle version is 9.2.0.8.0 with the only intrim path I can see being installed : 6130293.
Following is the extract from Alert Log for the day this occurred, again I must stress that this happens only on high system load days like month end:
Fri Nov 28 03:43:33 2008
ALTER TABLESPACE "PSAPBTABI" END BACKUP
Fri Nov 28 03:43:33 2008
Completed: ALTER TABLESPACE "PSAPBTABI" END BACKUP
Fri Nov 28 03:43:37 2008
ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPCLUD" BEGIN BACKUP
Fri Nov 28 03:53:38 2008
ALTER TABLESPACE "PSAPCLUD" END BACKUP
Fri Nov 28 03:53:38 2008
Completed: ALTER TABLESPACE "PSAPCLUD" END BACKUP
Fri Nov 28 03:53:42 2008
ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPCLUI" BEGIN BACKUP
Fri Nov 28 03:54:36 2008
ALTER TABLESPACE "PSAPCLUI" END BACKUP
Fri Nov 28 03:54:36 2008
Completed: ALTER TABLESPACE "PSAPCLUI" END BACKUP
Fri Nov 28 03:54:38 2008
ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDDICD" BEGIN BACKUP
Fri Nov 28 03:55:47 2008
ALTER TABLESPACE "PSAPDDICD" END BACKUP
Fri Nov 28 03:55:47 2008
Completed: ALTER TABLESPACE "PSAPDDICD" END BACKUP
Fri Nov 28 03:55:50 2008
ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDDICI" BEGIN BACKUP
Fri Nov 28 03:56:41 2008
ALTER TABLESPACE "PSAPDDICI" END BACKUP
Fri Nov 28 03:56:41 2008
Completed: ALTER TABLESPACE "PSAPDDICI" END BACKUP
Fri Nov 28 03:56:45 2008
ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUD" BEGIN BACKUP
Fri Nov 28 03:56:49 2008
ALTER TABLESPACE "PSAPDOCUD" END BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUD" END BACKUP
Fri Nov 28 03:56:52 2008
ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUI" BEGIN BACKUP
Fri Nov 28 03:56:55 2008
ALTER TABLESPACE "PSAPDOCUI" END BACKUP
Completed: ALTER TABLESPACE "PSAPDOCUI" END BACKUP
Fri Nov 28 03:56:59 2008
ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPEL640D" BEGIN BACKUP
Fri Nov 28 04:03:46 2008
ALTER TABLESPACE "PSAPEL640D" END BACKUP
Fri Nov 28 04:03:46 2008
Completed: ALTER TABLESPACE "PSAPEL640D" END BACKUP
Fri Nov 28 04:03:50 2008
ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPEL640I" BEGIN BACKUP
Fri Nov 28 04:04:37 2008
ALTER TABLESPACE "PSAPEL640I" END BACKUP
Fri Nov 28 04:04:37 2008
Completed: ALTER TABLESPACE "PSAPEL640I" END BACKUP
Fri Nov 28 04:04:40 2008
ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPES640D" BEGIN BACKUP
Fri Nov 28 04:20:17 2008
ALTER TABLESPACE "PSAPES640D" END BACKUP
Fri Nov 28 04:20:17 2008
Completed: ALTER TABLESPACE "PSAPES640D" END BACKUP
Fri Nov 28 04:20:20 2008
ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPES640I" BEGIN BACKUP
Fri Nov 28 04:36:26 2008
ALTER TABLESPACE "PSAPES640I" END BACKUP
Fri Nov 28 04:36:26 2008
Completed: ALTER TABLESPACE "PSAPES640I" END BACKUP
Fri Nov 28 04:36:29 2008
ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPLOADD" BEGIN BACKUP
Fri Nov 28 04:36:40 2008
ALTER TABLESPACE "PSAPLOADD" END BACKUP
Fri Nov 28 04:36:40 2008
Completed: ALTER TABLESPACE "PSAPLOADD" END BACKUP
Fri Nov 28 04:36:43 2008
ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPLOADI" BEGIN BACKUP
Fri Nov 28 04:36:53 2008
ALTER TABLESPACE "PSAPLOADI" END BACKUP
Completed: ALTER TABLESPACE "PSAPLOADI" END BACKUP
Fri Nov 28 04:36:56 2008
ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPOOLD" BEGIN BACKUP
Fri Nov 28 04:41:04 2008
ALTER TABLESPACE "PSAPPOOLD" END BACKUP
Fri Nov 28 04:41:04 2008
Completed: ALTER TABLESPACE "PSAPPOOLD" END BACKUP
Fri Nov 28 04:41:08 2008
ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPOOLI" BEGIN BACKUP
Fri Nov 28 04:45:17 2008
ALTER TABLESPACE "PSAPPOOLI" END BACKUP
Fri Nov 28 04:45:17 2008
Completed: ALTER TABLESPACE "PSAPPOOLI" END BACKUP
Fri Nov 28 04:45:21 2008
ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPROTD" BEGIN BACKUP
Fri Nov 28 04:47:25 2008
ALTER TABLESPACE "PSAPPROTD" END BACKUP
Fri Nov 28 04:47:25 2008
Completed: ALTER TABLESPACE "PSAPPROTD" END BACKUP
Fri Nov 28 04:47:29 2008
ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPPROTI" BEGIN BACKUP
Fri Nov 28 04:48:09 2008
ALTER TABLESPACE "PSAPPROTI" END BACKUP
Fri Nov 28 04:48:09 2008
Completed: ALTER TABLESPACE "PSAPPROTI" END BACKUP
Fri Nov 28 04:48:13 2008
ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSOURCED" BEGIN BACKUP
Fri Nov 28 04:48:46 2008
ALTER TABLESPACE "PSAPSOURCED" END BACKUP
Fri Nov 28 04:48:46 2008
Completed: ALTER TABLESPACE "PSAPSOURCED" END BACKUP
Fri Nov 28 04:48:49 2008
ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSOURCEI" BEGIN BACKUP
Fri Nov 28 04:49:29 2008
ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
Fri Nov 28 04:49:29 2008
Completed: ALTER TABLESPACE "PSAPSOURCEI" END BACKUP
Fri Nov 28 04:49:34 2008
ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSTABD" BEGIN BACKUP
Fri Nov 28 05:05:16 2008
ALTER TABLESPACE "PSAPSTABD" END BACKUP
Fri Nov 28 05:05:16 2008
Completed: ALTER TABLESPACE "PSAPSTABD" END BACKUP
Fri Nov 28 05:05:23 2008
ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPSTABI" BEGIN BACKUP
Fri Nov 28 05:21:15 2008
ALTER TABLESPACE "PSAPSTABI" END BACKUP
Fri Nov 28 05:21:15 2008
Completed: ALTER TABLESPACE "PSAPSTABI" END BACKUP
Fri Nov 28 05:21:19 2008
ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPUSER1D" BEGIN BACKUP
Fri Nov 28 05:21:42 2008
ALTER TABLESPACE "PSAPUSER1D" END BACKUP
Fri Nov 28 05:21:42 2008
Completed: ALTER TABLESPACE "PSAPUSER1D" END BACKUP
Fri Nov 28 05:21:46 2008
ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
Completed: ALTER TABLESPACE "PSAPUSER1I" BEGIN BACKUP
Fri Nov 28 05:23:03 2008
ALTER TABLESPACE "PSAPUSER1I" END BACKUP
Fri Nov 28 05:23:03 2008
Completed: ALTER TABLESPACE "PSAPUSER1I" END BACKUP
Fri Nov 28 05:23:06 2008
ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
Completed: ALTER TABLESPACE "SYSTEM" BEGIN BACKUP
Fri Nov 28 05:24:03 2008
ALTER TABLESPACE "SYSTEM" END BACKUP
Fri Nov 28 05:24:03 2008
Completed: ALTER TABLESPACE "SYSTEM" END BACKUP
Fri Nov 28 05:24:07 2008
ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
Completed: ALTER TABLESPACE "UNDOTBS1" BEGIN BACKUP
Fri Nov 28 05:42:06 2008
ALTER TABLESPACE "UNDOTBS1" END BACKUP
Fri Nov 28 05:42:06 2008
Completed: ALTER TABLESPACE "UNDOTBS1" END BACKUP
Fri Nov 28 05:42:07 2008
ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program Files\CA\BrightStor ARCserve Backup Enterprise Option for SAP R3 for Oracle\CONTROL.PRD' REUSE
Completed: ALTER DATABASE BACKUP CONTROLFILE TO 'C:\Program F
Fri Nov 28 05:42:14 2008
Beginning log switch checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
Thread 1 advanced to log sequence 30706
Current log# 2 seq# 30706 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
Current log# 2 seq# 30706 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
Fri Nov 28 05:42:14 2008
ARC0: Evaluating archive log 3 thread 1 sequence 30705
ARC0: Beginning to archive log 3 thread 1 sequence 30705
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30705.001'
Fri Nov 28 05:42:27 2008
ARC1: Evaluating archive log 3 thread 1 sequence 30705
ARC1: Unable to archive log 3 thread 1 sequence 30705
Log actively being archived by another process
Fri Nov 28 05:42:29 2008
ARC0: Completed archiving log 3 thread 1 sequence 30705
Fri Nov 28 06:11:26 2008
Completed checkpoint up to RBA [0x77f2.2.10], SCN: 0x0000.3c0ae436
Fri Nov 28 09:20:59 2008
Beginning log switch checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
Thread 1 advanced to log sequence 30707
Current log# 1 seq# 30707 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
Current log# 1 seq# 30707 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
Fri Nov 28 09:20:59 2008
ARC0: Evaluating archive log 2 thread 1 sequence 30706
ARC0: Beginning to archive log 2 thread 1 sequence 30706
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30706.001'
Fri Nov 28 09:21:18 2008
ARC0: Completed archiving log 2 thread 1 sequence 30706
Fri Nov 28 09:51:06 2008
Completed checkpoint up to RBA [0x77f3.2.10], SCN: 0x0000.3c0b739c
Fri Nov 28 10:44:43 2008
Beginning log switch checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
Thread 1 advanced to log sequence 30708
Current log# 4 seq# 30708 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
Current log# 4 seq# 30708 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
Fri Nov 28 10:44:43 2008
ARC0: Evaluating archive log 1 thread 1 sequence 30707
ARC0: Beginning to archive log 1 thread 1 sequence 30707
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30707.001'
Fri Nov 28 10:45:07 2008
ARC1: Evaluating archive log 1 thread 1 sequence 30707
ARC1: Unable to archive log 1 thread 1 sequence 30707
Log actively being archived by another process
Fri Nov 28 10:45:09 2008
ARC0: Completed archiving log 1 thread 1 sequence 30707
Fri Nov 28 11:13:20 2008
Completed checkpoint up to RBA [0x77f4.2.10], SCN: 0x0000.3c0bb876
Fri Nov 28 11:45:09 2008
Beginning log switch checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
Thread 1 advanced to log sequence 30709
Current log# 3 seq# 30709 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
Current log# 3 seq# 30709 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
Fri Nov 28 11:45:09 2008
ARC0: Evaluating archive log 4 thread 1 sequence 30708
ARC0: Beginning to archive log 4 thread 1 sequence 30708
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30708.001'
Fri Nov 28 11:45:13 2008
ARC1: Evaluating archive log 4 thread 1 sequence 30708
ARC1: Unable to archive log 4 thread 1 sequence 30708
Log actively being archived by another process
Fri Nov 28 11:45:29 2008
ARC0: Completed archiving log 4 thread 1 sequence 30708
Fri Nov 28 12:15:17 2008
Completed checkpoint up to RBA [0x77f5.2.10], SCN: 0x0000.3c0bfc29
Fri Nov 28 12:50:23 2008
Beginning log switch checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
Thread 1 advanced to log sequence 30710
Current log# 2 seq# 30710 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
Current log# 2 seq# 30710 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
Fri Nov 28 12:50:24 2008
ARC0: Evaluating archive log 3 thread 1 sequence 30709
ARC0: Beginning to archive log 3 thread 1 sequence 30709
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30709.001'
Fri Nov 28 12:50:40 2008
ARC0: Completed archiving log 3 thread 1 sequence 30709
Fri Nov 28 13:20:29 2008
Completed checkpoint up to RBA [0x77f6.2.10], SCN: 0x0000.3c0c3298
Fri Nov 28 14:27:01 2008
Beginning log switch checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
Thread 1 advanced to log sequence 30711
Current log# 1 seq# 30711 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD01_M1.ORA
Current log# 1 seq# 30711 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD01_M2.ORA
Fri Nov 28 14:27:01 2008
ARC0: Evaluating archive log 2 thread 1 sequence 30710
ARC0: Beginning to archive log 2 thread 1 sequence 30710
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30710.001'
Fri Nov 28 14:27:25 2008
ARC0: Completed archiving log 2 thread 1 sequence 30710
Fri Nov 28 14:57:08 2008
Completed checkpoint up to RBA [0x77f7.2.10], SCN: 0x0000.3c0c75c1
Fri Nov 28 15:57:26 2008
Beginning log switch checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
Thread 1 advanced to log sequence 30712
Current log# 4 seq# 30712 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD04_M1.ORA
Current log# 4 seq# 30712 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD04_M2.ORA
Fri Nov 28 15:57:26 2008
ARC0: Evaluating archive log 1 thread 1 sequence 30711
ARC0: Beginning to archive log 1 thread 1 sequence 30711
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30711.001'
Fri Nov 28 15:57:40 2008
ARC1: Evaluating archive log 1 thread 1 sequence 30711
ARC1: Unable to archive log 1 thread 1 sequence 30711
Log actively being archived by another process
Fri Nov 28 15:57:40 2008
ARC0: Completed archiving log 1 thread 1 sequence 30711
Fri Nov 28 16:27:31 2008
Completed checkpoint up to RBA [0x77f8.2.10], SCN: 0x0000.3c0cb34a
Fri Nov 28 17:02:12 2008
Beginning log switch checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
Thread 1 advanced to log sequence 30713
Current log# 3 seq# 30713 mem# 0: C:\ORACLE\PRD\ORIGLOGA\LOGPRD03_M1.ORA
Current log# 3 seq# 30713 mem# 1: F:\ORACLE\PRD\MIRRLOGA\LOGPRD03_M2.ORA
Fri Nov 28 17:02:12 2008
ARC0: Evaluating archive log 4 thread 1 sequence 30712
ARC0: Beginning to archive log 4 thread 1 sequence 30712
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30712.001'
Fri Nov 28 17:02:25 2008
ARC0: Completed archiving log 4 thread 1 sequence 30712
Fri Nov 28 17:32:17 2008
Completed checkpoint up to RBA [0x77f9.2.10], SCN: 0x0000.3c0cc245
Fri Nov 28 18:00:19 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
Fri Nov 28 18:00:19 2008
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
Fri Nov 28 18:00:19 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
Fri Nov 28 18:00:19 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
Fri Nov 28 18:00:20 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
Fri Nov 28 18:00:21 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
Fri Nov 28 18:00:23 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
Fri Nov 28 18:00:23 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
Fri Nov 28 18:00:23 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
Fri Nov 28 18:00:24 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
Fri Nov 28 18:00:26 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
Fri Nov 28 18:00:27 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
Fri Nov 28 18:00:28 2008
/* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Fri Nov 28 20:00:17 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPBTABI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUD" COALES
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPCLUI" COALES
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDDICI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPDOCUI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640D" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPEL640I" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640D" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPES640I" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPLOADI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPOOLI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPPROTI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCED" COA
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSOURCEI" COA
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABD" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPSTABI" COALE
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1D" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "PSAPUSER1I" COAL
Fri Nov 28 20:00:18 2008
/* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Completed: /* BRCONNECT */ ALTER TABLESPACE "SYSTEM" COALESCE
Fri Nov 28 20:00:36 2008
Beginning log switch checkpoint up to RBA [0x77fa.2.10], SCN: 0x0000.3c0cf8e3
Thread 1 advanced to log sequence 30714
Current log# 2 seq# 30714 mem# 0: C:\ORACLE\PRD\ORIGLOGB\LOGPRD02_M1.ORA
Current log# 2 seq# 30714 mem# 1: F:\ORACLE\PRD\MIRRLOGB\LOGPRD02_M2.ORA
Fri Nov 28 20:00:36 2008
ARC0: Evaluating archive log 3 thread 1 sequence 30713
ARC0: Beginning to archive log 3 thread 1 sequence 30713
Creating archive destination LOG_ARCHIVE_DEST_1: 'F:\ORACLE\PRD\SAPARCH\ARC30713.001'
Fri Nov 28 20:00:44 2008
ARC1: Evaluating archive log 3 thread 1 sequence 30713
ARC1: Unable to archive log 3 thread 1 sequence 30713
Log actively being archived by another process
Fri Nov 28 20:00:47 2008
ARC0: Completed archiving log 3 thread 1 sequence 30713
Dump file f:\oracle\prd\saptrace\background\alert_prd.log
Mon Dec 01 09:03:16 2008
ORACLE V9.2.0.8.0 - Production vsnsta=0
vsnsql=12 vsnxtr=3
Windows 2000 Version 5.2 Service Pack 2, CPU type 586
Mon Dec 01 09:03:16 2008
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
processes = 120
sessions = 150
timed_statistics = TRUE
shared_pool_size = 822083584
large_pool_size = 8388608
java_pool_size = 8388608
enqueue_resources = 8000
filesystemio_options = SETALL
control_files = C:\oracle\PRD\origlogA\ctrlPRD.ctl, F:\oracle\PRD\saparch\ctrlPRD.ctl, E:\oracle\PRD\sapdata1\ctrlPRD.ctl
db_block_checksum = FALSE
db_block_size = 8192
db_cache_size = 1073741824
compatible = 9.2.0
log_archive_start = TRUE
log_archive_dest = F:\oracle\PRD\saparch
log_buffer = 4194304
log_checkpoint_interval = 0
db_files = 254
db_file_multiblock_read_count= 8
log_checkpoints_to_alert = TRUE
control_file_record_keep_time= 30
dml_locks = 2500
row_locking = always
replication_dependency_tracking= FALSE
transactions_per_rollback_segment= 20
transaction_auditing = FALSE
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 43200
remote_os_authent = TRUE
db_domain = world
service_names = PRD.world, PRD.WORLD
hash_join_enabled = FALSE
background_dump_dest = F:\Oracle\PRD\saptrace\background
user_dump_dest = F:\Oracle\PRD\saptrace\usertrace
max_dump_file_size = 1240
core_dump_dest = F:\Oracle\PRD\saptrace\background
optimizer_features_enable= 9.2.0
sort_area_size = 0
sort_area_retained_size = 0
db_name = PRD
open_cursors = 800
optimizer_mode = CHOOSE
sortelimination_cost_ratio= 10
btree_bitmap_plans = FALSE
eliminatecommon_subexpr= FALSE
pushjoin_predicate = FALSE
optimizerjoin_sel_sanity_check= TRUE
parallel_threads_per_cpu = 4
optimizer_index_cost_adj = 10
optimizer_index_caching = 50
optimpeek_user_binds = FALSE
pga_aggregate_target = 536870912
optimizeror_expansion = DEPTH
PMON started with pid=2, OS id=17336
DBW0 started with pid=3, OS id=16972
LGWR started with pid=4, OS id=16900
CKPT started with pid=5, OS id=17236
SMON started with pid=6, OS id=17436
Regards
Pieter -
Database objects behind stats on "Concurrent Processing Dashboard" ?
I want to define a new report in oem grid about concurrent processing on a ebs target. i saw that there is some minimal stats in the 'Concurrent Processing Dashboard'. What are the DB tables used to generate this screen ?
By the way there is a bug, all number are displayed like : 2.000000 ....
ThanksHi,
What if both nodes have all modules active? (host,admin,cm,forms,web) except the database of courseDo you mean to say all application services are up and the database is down? If yes, then there must be something wrong with the dashboard itself (if the database is down, you will not be able to access the application or even access OAM).
Thanks,
Hussein -
Querying the toplink cache under high-load
We've had some interesting experiences with "querying" the TopLink Cache lately.
It was recently discovered that our "read a single object" method was incorrectly
setting query.checkCacheThenDB() for all ReadObjectQueries. This was brought to light
when we upgraded our production servers from 4 cores to 8. We immediatly started
experiencing very long response times under load.
We traced this down to the following stack: (TopLink version 10.1.3.1.0)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
at java.lang.Object.wait(Object.java:474)
at oracle.toplink.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:179)
- locked <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
at oracle.toplink.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:167)
at oracle.toplink.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:122)
at oracle.toplink.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:31)
at oracle.toplink.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:530)
at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:412)
at oracle.toplink.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:223)
at oracle.toplink.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:504)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:564)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
We moved the query back to the default, query.checkByPrimaryKey() and this issue went away.
The bottleneck seemed to stem from the read lock on the CacheKey from IdenityMapKeyEnumeration
public Object nextElement() {
if (this.nextKey == null) {
throw new NoSuchElementException("IdentityMapKeyEnumeration nextElement");
// CR#... Must check the read lock to avoid
// returning half built objects.
this.nextKey.checkReadLock();
return this.nextKey;
We had many threads that were having contention while searching the cache for a particular query.
From the stack we know that the contention was limited to one class. We've since refactored that code
not to use a query in that code path.
Question:
Armed with this better knowledge of how TopLink queries the cache, we do have a few objects that we
frequently read by something other than the primary key. A natural key, but not the oid.
We have some other caching mechanisms in place (JBoss TreeCache) to help eliminate queries to the DB
for these objects. But the TreeCache also tries to acquire a read lock when accessing the cache.
Presumably a read lock over the network to the cluster.
Is there anything that can be done about the read lock on CacheKey when querying the cache in a high load
situation?CheckCacheThenDatabase will check the entire cache for a match using a linear search. This can be efficient if the cache is very large. Typically it is more efficient to access the database if your cache is large and the field you are querying in the database is indexed in the table.
The cache concurrency was greatly improved in TopLink 11g/EclipseLink, so you may wish to try it out.
Supporting indexes in the TopLink/EclipseLink cache is something desirable (feel free to log the enhancement request on EclipseLink). You can simulate this to some degree using a named query and a query cache.
-- James : http://www.eclipselink.org -
Concurrent processing status down in OAM
Hi ,
From OAM->Applications Dashboard ->Overview
the status for concurrent processing shows down but infact it is up
I also checked from OAM->Hosts->view configuration
whereas here the status for concurrent processing is up.
thanks all in advance..
applications version: 11.5.10.2Hi,
Please see if these documents help.
Note: 405375.1 - OAM Reports 'Internal Concurrent Manager Found Node To Be Down.' When Node is Up
Note: 419823.1 - OAM: Web Components Status: Shows Down Even When The System Works Fine
Note: 309271.1 - Web Components Status in OAM Shows All Down Even When the System Works Fine
Note: 434569.1 - OAM application dashbord shows all the web components with status down whereas they are up
Note: 287415.1 - Host Status In Application Dashboard Overview Tab Shown As Down in OAM
Regards,
Hussein -
MDB topic listener and concurrent processing
Hello to everybody.
I ve prototyped simple db publisher ,which publishes changed data from database to JMS topic. Then I have my MDB which listens on JMS topic.
I setup and implemented all correctly and it worked. There was 1 consumer = 1 mdb instance. Then I tried to do load testing , So I triggered 8000 changes from database. The db adapter did the job and published the 8000 messages on the JMS topic in a while. To mimic processing in MDB I put the sleep(10seconds) in MDB.
MDB topic processing seems to me as not concurrent one. Why? I observed 10 message decrease with 5-10 seconds.
What I expected was: concurrent processing of 8000 JMS messages by 1 MDB (so MDB is poolable component isnt it?).
So for 500 instances of topic MDB and 8000 messages on the JMS topic, I expected at least 8000/500 X 10 = 160 seconds to process message load.
I am using WLS 816.
Can You give me some hits.
Thanks
RomanI did the same test on Weblogic 10 and it worked as expected. MDB durable subscriber consumed 8000 messages (each msg processing had 10 secs sleep inside.) within 2 minutes. So it is ok.
But why it did not worked on WLS 8.1? Is this a bug ? -
Sync messages failed due to high load
Hi!
On our test system we had a very high load of async messages for 3 - 4 hours. The async messages were processed successfully but all sync messages failed during this time frame! As soon as the high load of async messages stopped, the sync messages started executing successfully again.
Has anybody also experienced this kind of problem and what can be done to avoid this issue? Thanks for any input and suggestions.
Regards, Tanjahey Tanja,
i had a similar issue when testing the box with Load runner, that a simple sync scenario fails when the size of payload or attachment is high, the thing that i did is just increased the virtual memory ,
You can also check the trace files to figure out problem ?
This might not be a definitive answer , hope this helps,
Vara
[PLEASE REWARD FOR POSTING] -
Performance Bottleneck at HIgh Loads
Hi,
We are currently load testing our web application and have been facing an issue that has ben bothering us for sometime now.
If we load test the application at high load for sustained amounts of time, (around 2 hours), the performance suddenly dips to Zero hits/sec, Nothing is processed , no output generated. It seems as though all the processes some how are waiting for a resource to get free.After sometime the performance goes up again and again there is a race condition because of which the performance goes down again. Here on the dipping becomes highly erratic.
We are trying to find out if this is a known bug with the oracle jdbc driver. "oracle.jdbc.driver.OracleDriver" .
Would appreciate if anybody could provide any inputs as to why this is happening and how we could alleviate the situation.
Thanks in advance.
DeepanjanHi I vaguely remember something like this.... but I thought it typically occurred during lots of heavy insert/update activity
which version of the jdbc drivers are you using.... are you using thin/thick
Have you checked the SQL or the database to determine whether any locks are blocking activity or whether you have large database wait issues....
Dom
Maybe you are looking for
-
Audit Data in BI Publisher Standalone
Hi: I'm trying to audit who/what/when BI reports are run and I've got the BI Publisher user from XDO_USER_NAME but I need to find out what report is being run. I don't know of an existing XDO parameter or anything else that has that. So I'm trying to
-
Listening to Podcasts using the Podcast APP continues to be a struggle and now even worse in IOS 8.0 Anyone have any suggestions how to make listening to Podcast as easy as it used to be. Everytime I go to download my daily podcasts it is a gong show
-
Just today, out of no where, sarafi will only show me a white page, regardless of what address I type in the address bar. Brand new Macbook Pro for Christmas. Worked yesterday, not today. I googled it and based on the hits that came back I thought
-
Web safe fonts - Baby Theme site 1.0
Hello I'm using the Baby Theme template from 1.0, does anyone know why my content font is being displayed in IE in the Times New Roman font when I have specifically changed the content font to Copper Black in iWeb. Can I change the buttons (for blog,
-
RAID for speed....for use with home recording
Hi all, I want to get my G5 dual 2.0 faster for home recording with logic 8 using another hard drive. I was told RAID is the way to go? I am NOT familiar with RAID at all or how to do this but I am very good at picking this stuff up. Unfortunately th