All Concurrent managers going down (Except Internal conc manager)

Hi All,
We are on Oracle Apps R12.0.4 and Db 10g
We have done a cloning from RAC ASM 2 node instance to Non-RAC NonASM single node.
The problem is that concurrent managers (Standard manager and other manangers) are going down automatically.
ICm obviously is trying to bring them back .
here is the log file:
Starting INVMGR Concurrent Manager : 29-JUN-2010 07:49:18
Process monitor session ended : 29-JUN-2010 07:49:18
Process monitor session started : 29-JUN-2010 07:51:18
Adding Node:(APP2), to unavailable list
Found dead process: spid=(28003), cpid=(240974), ORA pid=(44), manager=(0/4)
Found dead process: spid=(28002), cpid=(240973), ORA pid=(43), manager=(0/5)
Starting FNDSCH Concurrent Manager : 29-JUN-2010 07:51:18
Starting FNDCRM Concurrent Manager : 29-JUN-2010 07:51:18
Starting OAMCOLMGR Concurrent Manager : 29-JUN-2010 07:51:18
Starting FTE_TXN_MANAGER Concurrent Manager : 29-JUN-2010 07:51:18
Starting IEU_SH_CS Concurrent Manager : 29-JUN-2010 07:51:18
Starting IEU_WL_CS Concurrent Manager : 29-JUN-2010 07:51:18
Starting PASMGR Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting STANDARD Concurrent Manager : 29-JUN-2010 07:51:18
Starting INVMGR Concurrent Manager : 29-JUN-2010 07:51:18
Process monitor session ended : 29-JUN-2010 07:51:18
+++++++++++++++++++++++++++++++++++++++++
What i wanted to know is "Adding Node:(APP2), to unavailable list"
What does this mean and how to troubleshoot it ?
Also, i can see that STATUS column in FND_NODE table is set to "N"
Does it mean something which affects the concurrent managers?
Please help me resolve this.
Thanks,
KK

Hi,
The problem is that concurrent managers (Standard manager and other manangers) are going down automatically.
ICm obviously is trying to bring them back .Please verify the definition of the CMs and make sure none points to the RAC instances.
I would suggest you stop the CM, purge FND_NODES table and run AutoConfig on the database/application tier nodes, run cmclean.sql script, then start the CM and see if you can reproduce the issue.
Thanks,
Hussein

Similar Messages

Concurrent mgr goes down

hi all,
I am facing a new problem at my client side...whenever there are more requests concurrent mgr goes down - no request runs...application goes down...as it has clustering at OS level, application part is transferred to the other node and goes up from there...but why is it happening?? I mean it is normal more users more requests...conc mgr should not be down i think...plz help
Annex

What about the CM log file?--should I post the log file here?? but it is huge in size...plz suggest

Services of all the databases goes down

Hi ,
2 days back, Services of all the databases goes down on one of the PROD server.
When i tried to start the services using SRVCTL command .
I was getting the error below.
oracle@shbaorc07b bin]$ srvctl status database -d EDW
PRKH-1010 : Unable to communicate with CRS services.
[Communications Error(Native: prsr_initCLSS:[3])]
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: EDWX
PRKC-1056 : Failed to get the hostname for node + PRKH-1010 + prsr_initCLSS
after this error, i came to know that there is some problem with the services.
When i try to start the crs services, was not able to start the services
as a root user user,
I run the below command.
crsctl start crs
to my surprise the services of crs was not coming up,
Finally I have to start the server and all the cluster services came up.
I want to find out why this was happen, i need to find the bottle neck so this should not happen again as this is a PROD server and lots of prod database are residing on this server.
Please find crsd.log below.
2010-04-22 15:38:22.361: [ CRSRES][1539545408]0ora.shbaorc07b.LISTENER_UC4XPRD2.lsnr failed on shbaorc07b relocating.
2010-04-22 15:38:22.397: [ CRSRES][1539545408]0Cannot relocate ora.shbaorc07b.LISTENER_UC4XPRD2.lsnrStopping dependents
2010-04-22 15:38:22.407: [ CRSRES][1539545408]0StopResource: setting CLI values
2010-04-22 15:38:27.318: [ CRSAPP][1539545408]0CheckResource error for ora.shbaorc07b.LISTENER_CTLGPRD2.lsnr error code = 1
2010-04-22 15:38:27.321: [ CRSRES][1539545408]0In stateChanged, ora.shbaorc07b.LISTENER_CTLGPRD2.lsnr target is ONLINE
2010-04-22 15:38:27.322: [ CRSRES][1539545408]0ora.shbaorc07b.LISTENER_CTLGPRD2.lsnr on shbaorc07b went OFFLINE unexpectedly
2010-04-22 15:38:27.322: [ CRSRES][1539545408]0StopResource: setting CLI values
2010-04-22 15:38:27.336: [ CRSRES][1539545408]0Attempting to stop `ora.shbaorc07b.LISTENER_CTLGPRD2.lsnr` on member `shbaorc07b`
one more strange thing happen while restarting the 2nd server, the first server also restarted.
Please help me regarding this?
Thanks in advance.

Hi,
Thanks.
please find the OS version
Linux shbaorc07a.corp.shire.com 2.6.18-53.1.14.el5 #1 SMP Tue Feb 19 07:18:46 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
database version:
10.2.0.4.0
Node 2:
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGDPAXPRDALL type 2
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(IGDPAXPRDALL) birth(2/0)
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGGMCXPRDALL type 2
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGEASYPRDALL type 2
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(IGEASYPRDALL) birth(2/0)
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGCTRMPRDALL type 2
[    CSSD]2010-04-22 19:41:19.438 [1273047360] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(IGCTRMPRDALL) birth(2/0)
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGEDWXPRDALL type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGNUSBPRDALL type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGETLXPRDALL type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(IGETLXPRDALL) birth(2/0)
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGCTXXPRDALL type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(IGCTXXPRDALL) birth(2/0)
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGVVC30PRDALL type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock IGVUM30PRDALL type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupGrocks: cleaning up grock #CSS_CLSSOMON type 2
[    CSSD]2010-04-22 19:41:19.439 [1273047360] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(#CSS_CLSSOMON) birth(2/0)
[    CSSD]2010-04-22 19:41:19.445 [1273047360] >TRACE: clssgmEstablishConnections: 3 nodes in cluster incarn 5
[    CSSD]2010-04-22 19:41:19.449 [1231087936] >TRACE: clssgmPeerListener: connects done (3/3)
[    CSSD]2010-04-22 19:41:19.449 [1273047360] >TRACE: clssgmEstablishMasterNode: MASTER for 5 is node(3) birth(1)
[    CSSD]2010-04-22 19:41:19.458 [1231087936] >TRACE: clssgmHandleDBDone(): src/dest (3/65535) size(72) incarn 5
[    CSSD]CLSS-3000: reconfiguration successful, incarnation 5 with 3 nodes
[    CSSD]CLSS-3001: local node number 2, master node number 3
[    CSSD]2010-04-22 19:41:19.458 [1273047360] >TRACE: clssgmReconfigThread: completed for reconfig(5), with status(1)
[    CSSD]2010-04-22 19:49:20.816 [1189128512] >TRACE: clssnmConnComplete: probe from node 1, your version: 10.2.1.2
, support PENDINA: 1
[    CSSD]2010-04-22 19:49:20.816 [1189128512] >TRACE: clssnmConnComplete: MSGSRC 1, type 5, node 1, flags 0x0001, con 0x2aaab01abb00, probe (nil)
[    CSSD]2010-04-22 19:49:20.816 [1189128512] >TRACE: clssnmConnComplete: node 1, shbaorc07a, con(0x2aaab01abb00), probcon((nil)), ninfcon((nil)), node unique 1271962158, prev unique 0, msg unique 1271962158 node state 0
[    CSSD]2010-04-22 19:49:20.816 [1189128512] >TRACE: clssnmsendConnAck: node 1, node state 0
[    CSSD]2010-04-22 19:49:20.816 [1189128512] >TRACE: clssnmConnComplete: connecting to node 1 (con 0x2aaab01abb00), ninfcon (0x2aaab01abb00), state (0)
[    CSSD]2010-04-22 19:49:20.816 [1189128512] >TRACE: clssnmConnComplete: connected to node 1 (con 0x2aaab01abb00), ninfcon (0x2aaab01abb00), state (0), flag (1037)
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmDoSyncUpdate: Initiating sync 6
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmDoSyncUpdate: diskTimeout set to (57000)ms
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmSetupAckWait: Ack message type (11)
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmSetupAckWait: node(2) is ALIVE
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmSetupAckWait: node(3) is ALIVE
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmSetupAckWait: node(4) is ALIVE
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmSendSync: syncSeqNo(6)
[    CSSD]2010-04-22 19:49:20.927 [1189128512] >TRACE: clssnmHandleSync: diskTimeout set to (57000)ms
[    CSSD]2010-04-22 19:49:20.927 [1262557504] >TRACE: clssnmWaitForAcks: Ack message type(11), ackCount(4)
[    CSSD]2010-04-22 19:49:20.927 [1189128512] >TRACE: clssnmHandleSync: Acknowledging sync: src[2] srcName[shbaorc07b] seq[1] sync[6]
6911,1 99%
node 1:
[    CSSD]2010-04-22 18:56:00.996 [1189128512] >USER: clssnmHandleUpdate: SYNC(4) from node(4) completed
[    CSSD]2010-04-22 18:56:00.996 [1189128512] >USER: clssnmHandleUpdate: NODE 1 (shbaorc07a) IS ACTIVE MEMBER OF CLUSTER
[    CSSD]2010-04-22 18:56:00.996 [1189128512] >USER: clssnmHandleUpdate: NODE 2 (shbaorc07b) IS ACTIVE MEMBER OF CLUSTER
[    CSSD]2010-04-22 18:56:00.996 [1189128512] >USER: clssnmHandleUpdate: NODE 3 (shbaorc07c) IS ACTIVE MEMBER OF CLUSTER
[    CSSD]2010-04-22 18:56:00.996 [1189128512] >USER: clssnmHandleUpdate: NODE 4 (shbaorc07d) IS ACTIVE MEMBER OF CLUSTER
[    CSSD]2010-04-22 18:56:00.996 [1189128512] >TRACE: clssnmHandleUpdate: diskTimeout set to (200000)ms
[    CSSD]2010-04-22 18:56:01.008 [1273047360] >TRACE: clssgmReconfigThread: started for reconfig (4)
[    CSSD]2010-04-22 18:56:01.008 [1273047360] >USER: NMEVENT_RECONFIG [00][00][00][1e]
[    CSSD]2010-04-22 18:56:01.020 [1273047360] >TRACE: clssgmEstablishConnections: 4 nodes in cluster incarn 4
[    CSSD]2010-04-22 18:56:01.096 [1231087936] >TRACE: clssgmInitialRecv: (0x2aaab4001310) accepted a new connection from node 2 born at 4 active (4, 4), vers (10,3,1,2)
[    CSSD]2010-04-22 18:56:01.096 [1231087936] >TRACE: clssgmInitialRecv: conns done (4/4)
[    CSSD]2010-04-22 18:56:01.096 [1273047360] >TRACE: clssgmEstablishMasterNode: MASTER for 4 is node(3) birth(1)
[    CSSD]2010-04-22 18:56:01.138 [1231087936] >TRACE: clssgmHandleDBDone(): src/dest (3/65535) size(72) incarn 4
[    CSSD]CLSS-3000: reconfiguration successful, incarnation 4 with 4 nodes
[    CSSD]CLSS-3001: local node number 1, master node number 3
[    CSSD]2010-04-22 18:56:01.138 [1273047360] >TRACE: clssgmReconfigThread: completed for reconfig(4), with status(1)
[    CSSD]2010-04-22 19:40:00.531 [1231087936] >TRACE: clscsendx: (0x2aaab00fa3b0) Connection not active
[    CSSD]2010-04-22 19:40:00.531 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab00fa3b0), client (0x2aaab00fa6b0), proc ((nil))
[    CSSD]2010-04-22 19:40:19.410 [1231087936] >TRACE: clscsendx: (0x2aaab02acd00) Connection not active
[    CSSD]2010-04-22 19:40:19.410 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab02acd00), client (0x2aaab02ad7c0), proc ((nil))
[    CSSD]2010-04-22 19:40:19.410 [1231087936] >TRACE: clscsendx: (0x2aaab02ad570) Connection not active
[    CSSD]2010-04-22 19:40:19.410 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab02ad570), client (0x2aaab02af660), proc ((nil))
[    CSSD]2010-04-22 19:40:19.438 [1231087936] >TRACE: clscsendx: (0x2aaab0295780) Connection not active
[    CSSD]2010-04-22 19:40:19.438 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab0295780), client (0x2aaab02859f0), proc ((nil))
[    CSSD]2010-04-22 19:40:19.476 [1231087936] >TRACE: clscsendx: (0x2aaab029d150) Connection not active
[    CSSD]2010-04-22 19:40:19.476 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab029d150), client (0x2aaab029dbb0), proc ((nil))
[    CSSD]2010-04-22 19:40:19.476 [1231087936] >TRACE: clscsendx: (0x2aaab029d890) Connection not active
[    CSSD]2010-04-22 19:40:19.476 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab029d890), client (0x2aaab029e1b0), proc ((nil))
[    CSSD]2010-04-22 19:40:19.707 [1231087936] >TRACE: clscsendx: (0x2aaab02cdb30) Connection not active
[    CSSD]2010-04-22 19:40:19.707 [1231087936] >TRACE: clssgmSendClient: Send failed rc 6, con (0x2aaab02cdb30), client (0x2aaab02cdf60), proc ((nil))
[    CSSD]2010-04-22 19:40:19.710 [1231087936] >TRACE: clscsendx: (0x2aaab02ca9e0) Connection not active
please help me on this.
Thanks
Edited by: user646034 on Apr 26, 2010 1:13 AM

All concurrent managers will not start

We installed the minimum baseline patches for 11.5.10.2 along with the July CPU patches. After doing that, the standard manager (fndlibr) starts but all of the specialized managers do not (invlibr). We have relinked everything, run cmclean.sql, shortened the system path variable, etc. without success. This is on windows. I am looking for suggestions on what else to try.

Here's the script. Already tried setting the environment variable and all that does is echo the setting of the environment commands. Notice the script also says can just start the service.
@echo OFF
@if not "%ORAWIN_DBG%"=="" echo %ORAWIN_DBG%
@REM $Header: adcmctl_nt.cmd 115.19 2009/03/20 09:07:36 jmajumde ship $
@REM *========================================================================+
@REM | Copyright (c) 2000 Oracle Corporation, Redwood Shores, California, USA |
@REM | All rights reserved |
@REM | Applications Division |
@REM +========================================================================+
@REM |
@REM | FILENAME
@REM | adcmactl.cmd
@REM |
@REM | DESCRIPTION
@REM | Start / Stop Concurrent Manager for orat_rta08
@REM |
@REM | USAGE
@REM | adcmactl.cmd start <APPS username/APPS password>
@REM |
@REM | PLATFORM
@REM | NT Generic
@REM |
@REM | NOTES
@REM | On NT you can start and stop the concurrent manager process using the
@REM | services utility in the control panel.
@REM |
@REM | HISTORY
@REM |
@REM +========================================================================+
@REM
REM ###############################################################
REM
REM This file is automatically generated by AutoConfig. It will be read and
REM overwritten. If you were instructed to edit this file, or if you are not
REM able to use the settings created by AutoConfig, refer to Metalink document
REM 165195.1 for assistance.
REM
REM ###############################################################
@REM
echo
if "%ORAWIN_DBG%"=="ON" echo You are running adcmctl.cmd
echo
@REM Get Date and Time
FOR /F "DELIMS=" %%a IN ('DATE /T') DO (
SET dateline=%%a
FOR /F "DELIMS=" %%a IN ('TIME /T') DO (
SET timeline=%%a
echo %dateline%%timeline%
set env_file="f:\oracle\orapappl\APPSorat_rta08.cmd"
if exist %env_file% goto env_ok
echo Cannot find environment file %env_file%
VERIFY OTHER 2> NUL
goto end
:env_ok
call %env_file%
IF "%1" == "" GOTO usage
set control_code=%1
setlocal enableextensions
if "%2" == "" (
set /P unpw="Enter the APPS Username/Password: "
) else (
if "%2" == "-nopromptmsg" (
set /P unpw=""
) else (
set unpw=%2
for /F "usebackq tokens=1,2 delims=/" %%i in ('%unpw%') do set APPS_UNAME=%%i && set APPS_PWD=%%j
IF NOT EXIST f:\oracle\orapcomn\admin\log\orat_rta08 mkdir f:\oracle\orapcomn\admin\log\orat_rta08
REM
REM Check Reviver pid file
REM
set CMPIDFILE="f:\oracle\orapappl\fnd\11.5.0\log\reviver_orat_rta08.pid"
if exist %CMPIDFILE% goto :get_pidfile
REM As we are providing the functionality to check the status via bug 3835198.
REM The following line may be conflicting .So commenting it.
REM echo Concurrent managers (no pid file) not running
set RUNNING=0
goto :reviver_cont
:get_pidfile
FOR /F %%a IN ('type %CMPIDFILE%') DO (
set reviver_pid=%%a
IF "x%reviver_pid%" == "x" (
echo Concurrent managers ( %reviver_pid% ) not running
set RUNNING=0
) ELSE (
echo Concurrent managers ( %reviver_pid% ) is running
set RUNNING=1
:reviver_cont
REM
REM check to see the status and set to 1 if started
REM
SET status=0
FOR /F "delims=" %%a IN ('NET START ^| findstr /C:"OracleConcMgrorat_rta08"') DO (
SET status=1
IF /i "%control_code%" == "status" GOTO control_ok
IF /i "%control_code%" == "start" GOTO control_ok
IF /i "%control_code%" == "stop" GOTO control_ok
echo
echo $program: You must either specify 'start' or 'stop' or 'status'
goto usage
:control_ok
IF /i "%control_code%" == "status" GOTO statuscm
IF /i "%control_code%" == "start" GOTO startcm
IF /i "%control_code%" == "stop" GOTO stopcm
:statuscm
IF %status% == 0 (
ECHO The service "OracleConcMgrorat_rta08" hasn't been started
GOTO :end
IF %status% == 1 (
ECHO The service "OracleConcMgrorat_rta08" is already started
GOTO :end
:startcm
IF %status% == 1 (
ECHO The service "OracleConcMgrorat_rta08" is already started
GOTO :end
echo Starting concurrent manager for %DB_NAME% ...
net start OracleConcMgrorat_rta08
goto end
:stopcm
IF %status% == 0 (
ECHO The service "OracleConcMgrorat_rta08" hasn't been started
GOTO :end
IF %RUNNING% == 0 goto :disable_rev
IF %RUNNING% == 1 (
f:\oracle\orapappl\fnd\11.5.0\bin\reviver.exe -k %reviver_pid%
DEL /Q %CMPIDFILE%
:disable_rev
echo Shutting down concurrent managers for %DB_NAME% ...
( echo %APPS_PWD% ) | f:\oracle\orapappl\fnd\11.5.0\bin\concsub.exe %APPS_UNAME% SYSADMIN "System Administrator" SYSADMIN WAIT=Y CONCURRENT FND DEACTIVATE > f:\oracle\orapcomn\admin\log\orat_rta08\adcmctl.txt 2>&1
goto end
:usage
echo Usage:
echo adcmctl.cmd {start^|stop^|status} ^<APPS username/APPS password^>
echo
VERIFY OTHER 2> NUL
:end
echo adcmctl.cmd exiting with status %ERRORLEVEL%

ORA-00600: internal error code - Concurrent managers are down...!!

Dear All,
I am getting the following error while opening the concurrent manager Administer Window:
and none of the managers are up and running..!!!
The Error is:
ORA-00600: internal error code, arguments: [12700], [41258], [96660311], [4], [105091700], [25], [], []
FRM-40735: POST-QUERY trigger raised unhandled exception ORA-00600.
This is appearing while the records are being loaded in the window...
Please tell me how can I rectify this dangerous situation...
the More worry is that this is in production...
Please update....!!
Many thanks in advance.....

Hi,
Please mention the application release, database version, and the OS.
What changes have been done recently?
Can you find any details about the error in the Database/CM log files?
Any invalid objects in the database?
Did you try to regenerate the form via adadmin and see if this helps?
Does AutoConfig complete successfully?
Please see if (Note: 153788.1 - Troubleshoot an ORA-600 or ORA-7445 Error Using the Error Lookup Tool) helps. You may also run cmclean.sql script (Note: 134007.1).
Regards,
Hussein

Concurrent managers down

Dear All,
Today morning concurrent managers were down on production. when i checked the concurrent logs observed the following error.
Shutting down Concurrent Manager : 27-FEB-2012 04:21:24
List of errors encountered:
_ 1 _
Routine AFPSMG encountered an ORACLE error. ORA-00604: error occurred
at recursive SQL level 2
ORA-04031: unable t.
Review your error messages for the cause of the error. (=<POINTER>)
_ 2 _
Routine AFPGSR encountered an ORACLE error. ORA-04031: unable to
allocate 16 bytes of shared memory ("shared pool".
Review your error messages for the cause of the error. (=<POINTER>)
then we restarted the cocurrent managers and they are working fine.
I just want to know the cause of this error.
Thanks.

Hi
The error :ORA-04031: unable to allocate 16 bytes of shared memory.
indicates that there is not enough sufficient memory in the shared pool for the processes
You need to check the alert log file of the database to see if there are similar error messages and then do Flush shared pool. This should take care of the issue if it has come once.
ALTER SYSTEM FLUSH SHARED_POOL;
If this does not work:
Increase the shared memory or a simple DB bounce could resolve the issue.
Regards
Shalini

Workflow mailer goes down frequently

Dear All
workflow mailer goes down twice a day. we are using E- Buisness veriosn 11.5.10.2. we faced the similr problem some time back but it was resolved unknowingly.
Regards
Arifuddin

Please check the Workflow log file for details about the error.
You may also see the docs referenced in the old threads which discuss the same topic -- http://forums.oracle.com/forums/search.jspa?threadID=&q=Workflow+AND+Down&objID=c3&dateRange=all&userID=&numResults=15&rankBy=10001
Thanks,
Hussein

SAN goes down (Solved/Recovered Database 100%)

database 10g release 2
linux version 4.6
dear alls,
my SAN goes down its raid level 5 ,two hard disk has gone down ,i performed daily database and archivelog backup on tape,i keep last two days backup and controlfile autobackup with enabled option at my tape as well at FRA,FRA is also at SAN and no longer be available,
anyway my vendor is about to reach here to make the SAN up.Vendor claim that and i also know that with raid level 5 if 2 hard disk goes down then the SAN cant up i.e mount will no longer be restored, for the sake of safety can anybody correct me is it like that or we have some alternate to up the SAN??
i know how to recover in this scenario ,can anybody keep in touch with me this thread ,being a human being i may make some blunder or may skip some steps for recovery database from tape.Please correct me
CASE 1
if SAN cant up the orahome mount point then what i will do to
-install the oracle
-re create the lost instance "PROD"
-put the init file within respected path
-create the flash recovery area with same path as it was before lost
-connect to RMAN
-mount db
-restore the autopbackup controlfile
-restore database
-recover database
CASE 2
if SAN can up the orahome mount point then i think there would not be any issue if there is please let me know.
Please share me yours experience.
Thanks to all
Khurram

kora i am seeing yours post now thanks for corporates,dont mind wana one corrections in yours steps.
Set dbid=(the id that you find on the controlfile backup)
restore spfile from autobackup
startup mount
restore controlfile from autobackup
restore database
recover database
alter database open resetlogs startup mount will always occur after restoring controfile not after spfile/pfile.
Khurram

Sharepoint portal goes down Whenever a specific user logs into the portal

hi All,
Sharepoint portal goes down Whenever a specific user logs into the portal
When a specific user logs into the portal , the iis goes down and no one able to access the portal.
once we reset iis then portal works fine .Please help us to identify the root cause for this issue.

What is the error message it throws for everyone after that user user logged into the site. Did you encountered the same behavior for the other web application in the same farm as well?
It would be worth to check health of that particular application pool and the account used for it during the time you face the issue.
Veeramani.S

EBS 11i - Concurrent Manager goes down due to network failure.

Hi,
We have a Single Node Oracle EBS 11i (11.5.10) [upgraded from 11.0.3] Production Instance on a Windows 2003 (32-Bit) server.
We have UPS support for the Server but the Netwok Switch is not on UPS. Due to this whenever there is a power trip the network goes down and due to this the Concurrent Manager (only)goes down. All other Apps Tier and DB Tier services are up inlcluding the 806 Listener (APPS_SID).
Is this how it is designed to work? Is there anyway to ensure that the Conc. Manager does not go down due to network failure.
Rgds,
Thiru

Here it is
Process monitor session ended : 02-OCT-2009 07:30:04
The Internal Concurrent Manager has encountered an error.
Review concurrent manager log file for more detailed information. : 02-OCT-2009 07:43:16 -
Shutting down Internal Concurrent Manager : 02-OCT-2009 07:45:56
Reviver is not enabled, not spawning a reviver process.
List of errors encountered:
_ 1 _
Routine AFPCMT encountered an ORACLE error. ORA-01041: internal error.
hostdef extension doesn't exist
Review your error messages for the cause of the error. (=<POINTER>)
_ 2 _
Routine AFPSMG encountered an ORACLE error. ORA-03114: not connected
to ORACLE
Review your error messages for the cause of the error. (=<POINTER>)
_ 3 _
Routine FDPCRQ encountered an ORACLE error. ORA-03113: end-of-file on
communication channel
Review your error messages for the cause of the error. (=<POINTER>)
APP-FND-01564: ORACLE error 1041 in fdudat
Cause: fdudat failed due to ORA-01041: internal error. hostdef extension doesn't exist.
The SQL statement being executed at the time of the error was: &SQLSTMT and was executed from the file &ERRFILE.
List of errors encountered:
_ 1 _
Routine AFPCAL received failure code while parsing or running your
concurrent program CPMGR
Review your concurrent request log file for more detailed information.
Make sure you are passing arguments in the correct format.
The PROPTRN_1002@PROPTRN internal concurrent manager has terminated with status 1 - giving up.

JMS Exception when server goes down (Urgent) ?

Hi All,
I have three topics in the JMS Server which is on the top of Web logic server 8.1.
On the client side I have Swing thick cleint.For each topic I have a listener.
When the web logic server goes down ( I shut down manually ) Only one listener is recieving the Exception and the Swing client reacts accordingly.
After this ,
Client continousoly checks whether the server is up or not. Once the server is up again the Exception is thrown that server is down ( I guess now the second listener is recieving the message).
After some time again the third listener recives the message.....
and the client reacts on that exception 3 times.
How can I get rid of this ?
thanks in advance.
san

Hi,
this is probably a very good question to ask support. 'Logical' deployment sequence
was added to the WebLogic only recently (6.1sp?) ('logical' means that first datasources
are deployed, then EJB's and then web-apps, deployment-order parameter was added etc).
So, for applications where shutdown sequence is important there must be a document which
describes how server shutdown works.
On the other hand, this may not be as important as deployment sequence, because running
instance can crash at any time and it should not affect anything.
Jen <[email protected]> wrote:
Repost
Hi,
Most of my ejbs use connection from the connection pool. At the weblogic server shut
down time (heavy load), I can see the jsps call the ejb, sometime jsp can't find
the bean because it's removed from jndi. sometime jsp found the bean but when the
bean try to talk to database, ejb can't use the connection because the datasource
is removed from jndi. Also my shutdown class waits for the threads finish up, and
the threads calls the ejb and do the db operation, and the conn is cut when the db
operation is half way. So, the order of events when server shut down is important
to me. I am wondering how other people deal with it. Thanks --
Dimitri

Shut down the concurrent managers to terminate the process ORA-01000

Dear All,
Error As Follows
Shut down the concurrent managers to terminate the process or use port-specific operating system commands to terminate the process.
Job number is: 638
Job number is: 639
01-FEB-2013 10:46:36 - ORACLE error 1000 in get_status
The SQL statement being executed at the time of the error was: &SQLSTMT and wJob number is: 642
The call to function PingProcess in cartridge AQCART has failed.
Routine FNDSM cannot terminate the concurrent process with concurrent process ID &CPID and
operating system process ID 7537140
The SQL statement being executed at the time of the error was: &SQLSTMT and was executed from the file &ERRFILE.
01-FEB-2013 10:44:34 - ORACLE error 1000 in get_status
Cause: get_status failed due to ORA-01000: maximum open cursors exceeded.
Again after some time i used to get this error
Cause: get_status failed due to ORA-01000: maximum open cursors exceeded.
The SQL statement being executed at the time of the error was: &SQLSTMT and was executed from the file &ERRFILE.
01-FEB-2013 11:12:15 - ORACLE error 1000 in get_status
Cause: get_status failed due to ORA-01000: maximum open cursors exceeded.
The SQL statement being executed at the time of the error was: &SQLSTMT and was executed from the file &ERRFILE.
The call to function PingProcess in cartridge AQCART has failed.
Job number is: 651
Suggestions are welcome
Regards
Surinder Singh
Edited by: Surinder Singh on Feb 1, 2013 11:16 AM

Surinder Singh wrote:
Database Version - Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
Application Version - 11.5.10.2
Current Open cursors value
SUM(A.VALUE) NAME
28396 opened cursors current
Regards
Surinder SinghThis value is high.
Please run AutoConfig and make sure it completes successfully.
Can you find any errors in the database log file?
Thanks,
Hussein

MS SQL server is going down frequently(10 days of gap) with exception message: "A severe error occurred on the current command. The results, if any, should be discarded".

MS SQL server is going down frequently(10 days of gap) with exception message: "A severe error occurred
on the current command. The results, if any, should be discarded". We are facing this issue for past two month.But funny thing is server will be up automatically with out any service restart manually. Also if we try to restart the SQL
server service manually then SQL server will be in a dead lock situation and it will not come up even if we wait for long time. Then we should do a windows server machine restart to
make the SQL sever up. As a suggestion from Microsoft to fix this kind of similar issue,
we have installed service pack 3 for SQL Server. But even after we are facing same issue.
Server Details:
Server OS: Windows Server 2008 R2
Two type of database servers are installed on server:
1. MS SQL Server 2008 R2
2. My SQL
Also Reporting server is configured for the purpose of generating SSRS report from a dot net website.
NOTE:Immediately after the data retrieval/save, we are closing the connection explicitly by the
application.
we have checked the windows event log and below are the details:
Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Application information:
    Application domain: /LM/W3SVC/5/ROOT-1-130718142067856406
    Trust level: Full
    Application Virtual Path: /
    Application Path: E:\WebSpaceFolder\ACSQuiK\Production\
    Machine name: DBSERVER
Process information:
    Process ID: 148
    Process name: w3wp.exe
    Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
    Exception type: SqlException
    Exception message: A severe error occurred on the current command. The results, if any, should be discarded.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
Log Name:      Application
Source:        Report Server Windows Service (MSSQLSERVER)
Description:
Report Server Windows Service (MSSQLSERVER) cannot connect to the report server database.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Report Server Windows Service (MSSQLSERVER)" />
    <EventID Qualifiers="0">107</EventID>
Could anybody can suggest any kind of fix for this issue? Thanks in advance.

Hi YesYemPee,
I have tried but still not clear about your issue, I would like you provide more details information about your issue based on below points to better analysis about the issue:
What action did you do and caused the error "A severe error occurred on the current command. The results, if any, should be discarded", did you run report on the web application or something else then the error happen?
If you rendering the report and got the error, please try to provide us more error information in the log files which path like:
C:\Program Files\Microsoft SQL Server\MSRS10_50.MSSQLSERVER\Reporting Services\LogFiles
If it is not the case in step1, Please try to provide SQL Server Error log(SQL Server Logs) and more details information.
If you still have any problem, please feel free to ask.
Regards,
Vicky Liu
Vicky Liu
TechNet Community Support

MDM 7.1 SP05 server is going down all of a sudden : Any Inputs?

Hi All,
I am working on SAP MDM 7.1 SP05 and my repository is going down all of a sudden and when I get in to Log I see the following message :
The specified value for the Server Listening Port (0) is outside the range of available listening ports for system use. Please use only values between 49152 and 65535.
I went through the SAP Note and we upgraded all the components as per the guidlines... Does any one been through this issue, please share how to get rid of it.
Thanks
Rajeev.

HI,
we have MDM server port, MDS Port, MDSS Port, MDIS port..
all these ports are differnet... as we are integrated with other applicaitons.. other applicaitons are accessing MDM using MDM server port....
Apart from this wondering need to dig down .... we are not sure where the issue is... why MDM server getting restarted...
Your inputs are appreciated.
Regards,
Rajeev.

My screen has greens and black lines going down the middle an I can't access anything. What could be wrong? It happened all the sudden.

My screen has greens and black lines going down the middle an I can't access anything. What could be wrong? It happened all the sudden.

You're welcome! I feel your pain - both "closest" Apple Stores to me are almost 2 hours away. When I called for another problem under warranty the AppleCare guy checked my zip code and found an AASP (Authorized Apple Service Provider) 20 minutes from me I had no idea existed. They diagnosed the problem (bad bluetooth keyboard) and ordered a new one for me. The AppleCare person will probably use this http://www.apple.com/buy/locator/service/ so you can check it now. And AppleCare could well diagnose your problem over the phone anyway. And I feel certain it wasn't something you did inadvertently.

All Concurrent managers going down (Except Internal conc manager)

Similar Messages

Maybe you are looking for