ASM not starting up
ASM is not starting up on couple of nodes with error
Maximum Tranmission Unit (mtu) of the ether adapter is different
on the node running instance 3, and this node.
Ether adapters connecting the cluster nodes must be configured
with identical mtu on all the nodes, for Oracle
I have checked the mtu and is set to 1500 on all the nodes using ip link list. What else to be checked?
Thanks.
what is ORA error when u start the ASM.
Please run
export ORACLE_SID=+ASM(put ASM instance name)
sqlplus "/as sysdba"
startup
Please send the output
Similar Messages
-
Cssd does not start in non-RAC environment Thus we can not bring up ASM
Non-RAC environment
ASM version = 11.1.0.7
HP-UX Itanium 11.23
After power outage, CSSD does not start on non-RAC environment
Running as root "/sbin/init.d/init.cssd start" does not start cssd
Oracle support tried "$ASM_HOME/bin/localconfig delete" and "$ASM_HOME/bin/localconfig add"
but it did not start CSS
Oracle support tried "$ASM_HOME/bin/localconfig reset $ASM_HOME"
It started the CSSD and the "crsctl check css" came back with CSS is healthy
But around 1 minute later it rebooted the server and when it came up again CSS does not start.
They checked /etc/inittab and it looked fine.
Before the reboot we saw this message in the /var/adm/syslog/OLDsyslog.log:
Cluster Ready Services completed waiting on dependencies
Again it is a NON-RAC environment. We only need CSSD for ASM. We do not have CRS installed on this server.
Our test system has been down for a week and we did not get the resolution from Oracle support yet !
Any pointers are greately appriciated.
Thanks,
DzungHere is the message in $ASM_HOME/log/<hostname>/alert<hostname>.log :
2010-07-16 09:42:02.956
[client(11930)]CRS-1006:The OCR location /db/app/oracle/product/11.1/cdata/localhost/local.ocr is inacce
ssible. Details in /db/app/oracle/product/11.1/log/rmodbd01/client/clscfg10.log.
2010-07-16 09:42:02.971
[client(11930)]CRS-1006:The OCR location /db/app/oracle/product/11.1/cdata/localhost/local.ocr is inacce
ssible. Details in /db/app/oracle/product/11.1/log/rmodbd01/client/clscfg10.log.
2010-07-16 09:42:03.054
[client(11930)]CRS-1013:The OCR at /db/app/oracle/product/11.1/cdata/localhost/local.ocr was successfull
y formatted using version 2. Ignore earlier CRS-1006 messages if any.
2010-07-16 09:42:46.379
[cssd(12297)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rmodbd01 .
Here is the message in $ASM_HOME/log/<hostname>/cssd/cssdOUT.log:
setsid: failed with -1/1
s0clssscGetEnvOracleUser: calling getpwnam_r for user oracle
s0clssscGetEnvOracleUser: info for user oracle complete
07/16/10 09:42:36: CSSD starting
Here is the message in $ASM_HOME/log/<hostname>/cssd/ocssd.log:
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 7, from 6, changes 6
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmSetVersions: properties common to all peers: 1,2,3,4,5,6,7
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmEstablishMasterNode: MASTER for 174732166 is node(0) birth(174732166)
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 8, from 7, changes 7
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 9, from 8, changes 8
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 10, from 9, changes 9
[ CSSD]CLSS-3000: reconfiguration successful, incarnation 174732166 with 1 nodes
[ CSSD]CLSS-3001: local node number 0, master node number 0
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssscSAGEInitFenceCompl: Completing kgzf fence initialization
[ CSSD]2010-07-16 09:42:46.394 [12] >TRACE: clssgmUpdateEventValue: Client listener incarn val 174732166, changes 1
[ CSSD]2010-07-16 09:42:46.395 [12] >TRACE: clssgmAllocProc: (60000000003c7120) allocated
[ CSSD]2010-07-16 09:42:46.395 [12] >TRACE: clssgmAllocProc: (60000000003c73a0) allocated
[ CSSD]2010-07-16 09:42:46.396 [14] >TRACE: Connect request from user oracle
[ CSSD]2010-07-16 09:42:46.396 [14] >TRACE: Connect request from user root
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c7120 - 1,2,3
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2810) proc(60000000003c7120) pid(12350) version 11:1:1:4
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c73a0 - 1,2,3
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2990) proc(60000000003c73a0) pid(12131) version 11:1:1:4
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(1/600000000096e7c0)
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 1 (600000000096e7c0)
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmJoinGrock: local grock CSS_INTERNAL_NODE_GROUP new client 600000000096e7c0 with con 60000000003b2b10, requested num 0
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmAddNodeGrpMember: member (60000000009e0030) added
[ CSSD]2010-07-16 09:42:46.403 [12] >TRACE: clssgmGroupState: requested group state of group localhost_NG, member count 0
[ CSSD]2010-07-16 09:42:46.403 [12] >TRACE: clssgmGroupState: requested group state of group localhost_NG, member count 0
[ CSSD]2010-07-16 09:42:46.404 [12] >TRACE: clssgmDeadProc: proc 60000000003c73a0
[ CSSD]2010-07-16 09:42:46.404 [12] >TRACE: clssgmDestroyProc: cleaning up proc(60000000003c73a0) con(60000000003b2990) skgpid ospid 12131 with 0 clients, refcount 0
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmGroupState: requested group state of unknown group MASTER#DISKMON#GROUP
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmGroupState: requested group state of group MASTER#DISKMON#GROUP, member count 0
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: KGZF: context successfully initialized, API version 1.4, using pipe default
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssscSAGEInitFenceCompl: kgzf fence initialization successfully completed
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmReconfigThread: CSS/GM open for global group registrations
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmReconfigThread: completed for reconfig(174732166), with status(1)
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmUpdateEventValue: Reconfig Event val 2, changes 2
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmWaitOnEventValue: after Reconfig Event val 2, eval 2 waited 47
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(2/600000000096e870)
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmUpdateEventValue: Reconfig Event val 0, changes 3
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmStartNMMon: previous reconfig complete, incarnation(174732166)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmJoinGrock: global grock MASTER#DISKMON#GROUP#MX new client 600000000096e870 with con 60000000003b2990, requested num -1
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddGrockMember: adding member to grock MASTER#DISKMON#GROUP#MX
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddMember: granted member(0) flags(0x2) node(0) grock (6000000000989e50/MASTER#DISKMON#GROUP#MX)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmQueueGrockEvent: lockName(MASTER#DISKMON#GROUP#MX) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmCommonAddMember: global lock grock MASTER#DISKMON#GROUP#MX member(0/Local) node(0) flags 0x2 0x2
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(3/600000000096e920)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 3 (600000000096e920)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmJoinGrock: global grock MASTER#DISKMON#GROUP new client 600000000096e920 with con 60000000003b2bd0, requested num 0
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddGrockMember: adding member to grock MASTER#DISKMON#GROUP
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddMember: new master 0 for group(MASTER#DISKMON#GROUP)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmAddMember: Adding fencing for member 0, group MASTER#DISKMON#GROUP, death 1, SAGE 0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmAddMember: member (0/60000000009e0230) added. pbsz(72) prsz(0) flags 0x0 to grock (600000000098a170/MASTER#DISKMON#GROUP)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmQueueGrockEvent: groupName(MASTER#DISKMON#GROUP) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x78, state 0x0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmCommonAddMember: global group grock MASTER#DISKMON#GROUP member(0/Local) node(0) flags 0x0 0x1e00
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmExitGrock: client 2 (600000000096e870), grock MASTER#DISKMON#GROUP#MX, member 0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock MASTER#DISKMON#GROUP#MX
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmRemoveMember: grock MASTER#DISKMON#GROUP#MX, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 3
[ CSSD]2010-07-16 09:42:48.405 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:48.405 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:42:52.444 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:52.444 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:42:56.484 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:56.484 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:00.516 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:00.516 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:04.563 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:04.563 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:08.603 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:08.603 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:12.643 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:12.643 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:16.676 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:16.676 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:20.723 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:20.723 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:24.762 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:24.762 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:28.802 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:28.802 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:32.842 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:32.842 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:36.882 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:36.882 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:40.922 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:40.922 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:44.964 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:44.964 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:49.002 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:49.002 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:53.043 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:53.043 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:57.085 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:57.085 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:58.354 [12] >TRACE: clssgmAllocProc: (60000000003c7ee0) allocated
[ CSSD]2010-07-16 09:43:58.355 [14] >TRACE: Connect request from user oracle
[ CSSD]2010-07-16 09:43:58.356 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c7ee0 - 1,2,3
[ CSSD]2010-07-16 09:43:58.356 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2990) proc(60000000003c7ee0) pid(13157) version 11:1:1:4
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmRegisterClient: proc(2/60000000003c7ee0), client(1/600000000096e870)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 1 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmJoinGrock: global grock CLSSSCHECK_GROUP new client 600000000096e870 with con 60000000003b2c90, requested num -1
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddGrockMember: adding member to grock CLSSSCHECK_GROUP
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: new master 0 for group(CLSSSCHECK_GROUP)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: Adding fencing for member 0, group CLSSSCHECK_GROUP, death 1, SAGE 0
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: member (0/60000000009e0130) added. pbsz(8) prsz(8) flags 0x0 to grock (6000000000989e50/CLSSSCHECK_GROUP)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmQueueGrockEvent: groupName(CLSSSCHECK_GROUP) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x0, state 0x0
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmCommonAddMember: global group grock CLSSSCHECK_GROUP member(0/Local) node(0) flags 0x0 0x0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 1 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExitGrock: client 1 (600000000096e870), grock CLSSSCHECK_GROUP, member 0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock CLSSSCHECK_GROUP
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRemoveMember: grock CLSSSCHECK_GROUP, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 2
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRegisterClient: proc(2/60000000003c7ee0), client(2/600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmJoinGrock: global grock CLSSSCHECK_LOCK new client 600000000096e870 with con 60000000003b2c90, requested num -1
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmAddGrockMember: adding member to grock CLSSSCHECK_LOCK
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmAddMember: granted member(0) flags(0x2) node(0) grock (6000000000989e50/CLSSSCHECK_LOCK)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmQueueGrockEvent: lockName(CLSSSCHECK_LOCK) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmCommonAddMember: global lock grock CLSSSCHECK_LOCK member(0/Local) node(0) flags 0x2 0x2
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExitGrock: client 2 (600000000096e870), grock CLSSSCHECK_LOCK, member 0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock CLSSSCHECK_LOCK
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRemoveMember: grock CLSSSCHECK_LOCK, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 3
[ CSSD]2010-07-16 09:43:58.362 [12] >TRACE: clssgmDeadProc: proc 60000000003c7ee0
[ CSSD]2010-07-16 09:43:58.362 [12] >TRACE: clssgmDestroyProc: cleaning up proc(60000000003c7ee0) con(60000000003b2990) skgpid ospid 13157 with 0 clients, refcount 0
[ CSSD]2010-07-16 09:44:01.125 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:01.125 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:05.164 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:05.164 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:09.196 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:09.196 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:13.236 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:13.236 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:17.284 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:17.284 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:21.322 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:21.322 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:25.316 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:25.316 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:30.333 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:30.333 [8] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[ CSSD]2010-07-16 09:44:34.324 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:34.324 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
Here is the message in $ASM_HOME/log/<hostname>/diskmon/client.log:
[ DISKMON] 07/16/2010 18:33:32.050 dskm_send_command: process 23246 sending command 8 to master diskmon listening on default pipe
[ DISKMON] 07/16/2010 18:33:32.080 dskm_send_command3: skgznp_connect failed with error 56815
[ DISKMON] 07/16/2010 18:33:32.080 dskm_send_command3: error 56815 at location skgznpcon6 - connect() - Connection refused
Here is the message in $ASM_HOME/log/<hostname>/diskmon/diskmonOUT.log:
setsid: failed with -1/1
dskm_getenv_oracle_user: calling getpwnam_r for user oracle
dskm_getenv_oracle_user: info for user oracle complete
07/16/10 18:33:31: Master Diskmon starting
Here is the message in $ASM_HOME/log/<hostname>/diskmon/diskmon.log:
[ DISKMON] 07/16/2010 09:42:35.573 dskm main: starting up
[ DISKMON] 07/16/2010 09:42:35.588 [12350:3] dskm_rac_thrd_main: running
[ DISKMON] 07/16/2010 09:42:35.588 [12350:1] dskm_rac_thrd_creat2: got the post from the css event handling thread
[ DISKMON] 07/16/2010 09:42:35.589 [12350:1] dskm main: startup complete
[ DISKMON] 07/16/2010 09:42:35.589 [12350:1] listening on -> default pipe
[ DISKMON] 07/16/2010 09:42:35.792 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:36.385 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:36.906 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:37.426 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:37.945 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:38.465 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:46.376 [12350:1] dskm_slave_thrd_creat: thread created
[ DISKMON] 07/16/2010 09:42:46.376 [12350:11] dskm_slave_thrd_main1: slave 0 running
[ DISKMON] 07/16/2010 09:42:46.376 [12350:11] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_proc_identify8: client kgzf/12297, version 0x01020000, slave 0, reid cid=3e0391f05e06cfafbf7419d7cf085a44,icin=174732166,nmn=0,lnid=174732166,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_send_version1:
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_send_version4: done
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_process_msg5: received msg type KGZM_KGZF_HANDSHAKE (0x0010)
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_proc_kgzf_handshake3: client kgzf/12297, kgzf version 0x00010004, slave 0
[ DISKMON] 07/16/2010 09:42:46.401 [12350:3] dskm_clss_ini2: successful clsssinit(), clssvers 2.1
[ DISKMON] 07/16/2010 09:42:46.402 [12350:3] dskm_clss_ini12: node rmodbd01 (0) registered in cluster
[ DISKMON] 07/16/2010 09:42:46.403 [12350:3] dskm_reid_ini12: diskmon reid cid=3e0391f05e06cfafbf7419d7cf085a44,icin=174732166,nmn=0,lnid=174732166,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=12350,opsn=1279291355,lvl=process
[ DISKMON] 07/16/2010 09:42:46.424 [12350:3] dskm_sage_config: CELL storage configuration file cellinit.ora not found
[ DISKMON] 07/16/2010 09:42:46.425 [12350:3] dskm_nfy_kgzf1: notified thread kgzf enabled
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_proc_kgzf_handshake5: got the post from the hb thread
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_proc_kgzf_handshake9: done, kgzf enabled
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
[ DISKMON] 07/16/2010 09:42:46.426 [12350:3] dskm_rac_ini22: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 0 (GSDGRPSZ 512) -
CSS not starting while installing ASM on Standalone Server
Hi All,
While installing ASM on AIX server, CSS is not coming up. This server is not on cluster. I have given below the output of after executing the root.sh script.
bash-3.00# /orahome/asm/root.sh
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started
Can anyone help me? It's bit urgent!!!!
Thank you.
Regards,
Mithra.Hi All,
Thanks for all the help.
I have found the solution.
I had to delete all .oracle folders and then run the root.sh
Thanks once again.
Regards,
Mithra. -
Database is not starting after RAC installation
Hi All,
I have installed 10g R2 2-node RAC on RHEL4 without any problem. But after the installation the database is not starting and i'm not able to connect. It is showing the below error..
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory
Can anybody help what might be the problem?
Thanks,
PraveenHi,
I'm giving the contents of alter log file from the first node...
Mon Feb 19 16:45:59 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
sga_target = 1073741824
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database_instances= 1
db_create_file_dest = +ORCL_DATA1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 16:46:00 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=31282
DIAG started with pid=3, OS id=31284
PSP0 started with pid=4, OS id=31286
LMON started with pid=5, OS id=31300
LMD0 started with pid=6, OS id=31315
MMAN started with pid=7, OS id=31317
DBW0 started with pid=8, OS id=31319
LGWR started with pid=9, OS id=31321
CKPT started with pid=10, OS id=31323
SMON started with pid=11, OS id=31325
RECO started with pid=12, OS id=31327
CJQ0 started with pid=13, OS id=31329
MMON started with pid=14, OS id=31331
Mon Feb 19 16:46:00 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=31333
Mon Feb 19 16:46:00 2007
starting up 1 shared server(s) ...
Mon Feb 19 16:46:00 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 16:46:01 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Reconfiguration complete
Mon Feb 19 16:46:01 2007
CREATE DATABASE "bsaprod"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET WE8ISO8859P1
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
Mon Feb 19 16:46:01 2007
Starting background process ASMB
ASMB started with pid=19, OS id=31365
Starting background process RBAL
RBAL started with pid=20, OS id=31369
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 16:46:05 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 16:46:06 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 16:46:06 2007
Database mounted in Exclusive Mode
Mon Feb 19 16:46:09 2007
Successful mount of redo thread 1, with mount id 3309011657
Assigning activation ID 3309011657 (0xc53b82c9)
Thread 1 opened at log sequence 1
Current log# 1 seq# 1 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 16:46:09 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 16:46:09 2007
SMON: enabling cache recovery
Mon Feb 19 16:46:09 2007
create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL online
Mon Feb 19 16:46:20 2007
Completed: create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL online
Mon Feb 19 16:46:20 2007
create rollback segment SYSTEM tablespace SYSTEM
storage (initial 50K next 50K)
Completed: create rollback segment SYSTEM tablespace SYSTEM
storage (initial 50K next 50K)
Mon Feb 19 16:46:24 2007
CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:31 2007
Successfully onlined Undo Tablespace 1.
Completed: CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:31 2007
create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
Completed: create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
Mon Feb 19 16:46:36 2007
CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Completed: CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Mon Feb 19 16:46:36 2007
ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
Completed: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
Mon Feb 19 16:46:36 2007
ALTER DATABASE DEFAULT TABLESPACE SYSTEM
Completed: ALTER DATABASE DEFAULT TABLESPACE SYSTEM
Mon Feb 19 16:46:38 2007
SMON: enabling tx recovery
Mon Feb 19 16:46:39 2007
Threshold validation cannot be done before catproc is loaded.
Threshold validation cannot be done before catproc is loaded.
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=22, OS id=32303
Mon Feb 19 16:46:39 2007
Completed: CREATE DATABASE "bsaprod"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET WE8ISO8859P1
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
Mon Feb 19 16:46:39 2007
CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:46 2007
CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Completed: CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Mon Feb 19 16:46:46 2007
ALTER DATABASE DEFAULT TABLESPACE "USERS"
Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"
Mon Feb 19 16:47:05 2007
Thread 1 advanced to log sequence 2
Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:47:34 2007
Thread 1 cannot allocate new log, sequence 3
Checkpoint not complete
Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 3
Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:48:10 2007
Thread 1 cannot allocate new log, sequence 4
Checkpoint not complete
Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 4
Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:49:05 2007
Thread 1 cannot allocate new log, sequence 5
Checkpoint not complete
Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 5
Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:50:19 2007
Thread 1 cannot allocate new log, sequence 6
Checkpoint not complete
Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 6
Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:51:00 2007
Thread 1 cannot allocate new log, sequence 7
Checkpoint not complete
Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 7
Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:51:54 2007
Thread 1 cannot allocate new log, sequence 8
Checkpoint not complete
Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 8
Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:55:32 2007
Thread 1 cannot allocate new log, sequence 9
Checkpoint not complete
Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 9
Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:56:35 2007
Thread 1 cannot allocate new log, sequence 10
Checkpoint not complete
Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 10
Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:56:52 2007
Thread 1 cannot allocate new log, sequence 11
Checkpoint not complete
Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 11
Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:57:04 2007
Thread 1 cannot allocate new log, sequence 12
Checkpoint not complete
Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 12
Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:57:14 2007
Thread 1 cannot allocate new log, sequence 13
Checkpoint not complete
Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 13
Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:57:31 2007
Thread 1 cannot allocate new log, sequence 14
Checkpoint not complete
Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 14
Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:58:13 2007
Thread 1 cannot allocate new log, sequence 15
Checkpoint not complete
Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 15
Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:59:42 2007
Thread 1 cannot allocate new log, sequence 16
Checkpoint not complete
Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 16
Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:00:33 2007
Thread 1 cannot allocate new log, sequence 17
Checkpoint not complete
Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 17
Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:01:49 2007
Thread 1 cannot allocate new log, sequence 18
Checkpoint not complete
Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 18
Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:02:23 2007
Thread 1 cannot allocate new log, sequence 19
Checkpoint not complete
Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 19
Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:04:06 2007
Thread 1 cannot allocate new log, sequence 20
Checkpoint not complete
Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 20
Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:05:22 2007
Thread 1 cannot allocate new log, sequence 21
Checkpoint not complete
Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 21
Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:06:22 2007
Thread 1 cannot allocate new log, sequence 22
Checkpoint not complete
Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 22
Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:08:42 2007
Thread 1 cannot allocate new log, sequence 23
Checkpoint not complete
Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 23
Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:09:43 2007
Thread 1 cannot allocate new log, sequence 24
Checkpoint not complete
Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 24
Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:10:42 2007
Thread 1 cannot allocate new log, sequence 25
Checkpoint not complete
Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 25
Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:11:31 2007
Thread 1 cannot allocate new log, sequence 26
Checkpoint not complete
Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 26
Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:12:01 2007
Thread 1 cannot allocate new log, sequence 27
Checkpoint not complete
Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 27
Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:13:12 2007
Thread 1 cannot allocate new log, sequence 28
Checkpoint not complete
Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 28
Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:14:05 2007
Thread 1 cannot allocate new log, sequence 29
Checkpoint not complete
Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 29
Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:14:57 2007
Thread 1 cannot allocate new log, sequence 30
Checkpoint not complete
Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 30
Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:16:29 2007
Thread 1 cannot allocate new log, sequence 31
Checkpoint not complete
Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:17:54 2007
Starting background process EMN0
EMN0 started with pid=25, OS id=15651
Mon Feb 19 17:17:54 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:17:55 2007
Stopping background process QMNC
Mon Feb 19 17:17:55 2007
Stopping background process CJQ0
Mon Feb 19 17:17:57 2007
Stopping background process MMNL
Mon Feb 19 17:17:58 2007
Stopping background process MMON
Mon Feb 19 17:17:59 2007
Shutting down instance (immediate)
License high water mark = 1
Mon Feb 19 17:17:59 2007
Stopping Job queue slave processes
Mon Feb 19 17:17:59 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:18:01 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:01 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:18:03 2007
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:18:04 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:04 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:18:04 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:18:04 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:18:09 2007
freeing rdom 0
Mon Feb 19 17:18:12 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database_instances= 1
db_create_file_dest = +ORCL_DATA1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:18:13 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=16124
DIAG started with pid=3, OS id=16126
PSP0 started with pid=4, OS id=16128
LMON started with pid=5, OS id=16130
LMD0 started with pid=6, OS id=16132
MMAN started with pid=7, OS id=16134
DBW0 started with pid=8, OS id=16136
LGWR started with pid=9, OS id=16138
CKPT started with pid=10, OS id=16140
SMON started with pid=11, OS id=16142
RECO started with pid=12, OS id=16144
CJQ0 started with pid=13, OS id=16146
MMON started with pid=14, OS id=16148
Mon Feb 19 17:18:13 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=16150
Mon Feb 19 17:18:13 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:18:13 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:18:14 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Reconfiguration complete
Mon Feb 19 17:18:14 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:18:14 2007
Starting background process ASMB
ASMB started with pid=19, OS id=16182
Starting background process RBAL
RBAL started with pid=20, OS id=16186
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:18:18 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:18:23 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:18:23 2007
Successful mount of redo thread 1, with mount id 3309012054
Mon Feb 19 17:18:23 2007
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:18:23 2007
alter database archivelog
Completed: alter database archivelog
Mon Feb 19 17:18:23 2007
alter database open
Mon Feb 19 17:18:23 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=22, OS id=16393
Mon Feb 19 17:18:23 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=23, OS id=16395
Mon Feb 19 17:18:23 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:18:23 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:18:23 2007
SMON: enabling cache recovery
Mon Feb 19 17:18:23 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:18:23 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:18:23 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=24, OS id=16422
Mon Feb 19 17:18:23 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:18:23 2007
SMON: enabling tx recovery
Mon Feb 19 17:18:23 2007
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=25, OS id=16449
Mon Feb 19 17:18:25 2007
Completed: alter database open
Mon Feb 19 17:18:25 2007
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
GROUP 4 SIZE 51200K
Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
GROUP 4 SIZE 51200K
Mon Feb 19 17:18:28 2007
ALTER DATABASE ENABLE PUBLIC THREAD 2
Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:18:29 2007
Starting background process EMN0
EMN0 started with pid=26, OS id=16564
Mon Feb 19 17:18:29 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:18:29 2007
Stopping background process QMNC
Mon Feb 19 17:18:29 2007
Stopping background process CJQ0
Mon Feb 19 17:18:30 2007
Stopping background process MMNL
Mon Feb 19 17:18:31 2007
Stopping background process MMON
Mon Feb 19 17:18:32 2007
Shutting down instance (immediate)
License high water mark = 3
Mon Feb 19 17:18:32 2007
Stopping Job queue slave processes
Mon Feb 19 17:18:32 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:18:34 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:34 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:18:34 2007
Shutting down archive processes
Archiving is disabled
Mon Feb 19 17:18:39 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:18:44 2007
ARCH shutting down
ARC1: Archival stopped
Mon Feb 19 17:18:49 2007
ARCH shutting down
ARC0: Archival stopped
Mon Feb 19 17:18:50 2007
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:18:51 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:51 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:18:51 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:18:51 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:18:56 2007
freeing rdom 0
Mon Feb 19 17:18:59 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +ORCL_DATA1
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:19:00 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=17373
DIAG started with pid=3, OS id=17375
PSP0 started with pid=4, OS id=17377
LMON started with pid=5, OS id=17379
LMD0 started with pid=6, OS id=17381
LMS0 started with pid=7, OS id=17383
LMS1 started with pid=8, OS id=17387
MMAN started with pid=9, OS id=17391
DBW0 started with pid=10, OS id=17393
LGWR started with pid=11, OS id=17395
CKPT started with pid=12, OS id=17397
SMON started with pid=13, OS id=17399
RECO started with pid=14, OS id=17412
CJQ0 started with pid=15, OS id=17428
MMON started with pid=16, OS id=17430
Mon Feb 19 17:19:00 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=17432
Mon Feb 19 17:19:00 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:19:01 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:19:01 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Feb 19 17:19:01 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Feb 19 17:19:01 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Mon Feb 19 17:19:01 2007
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:19:01 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:19:01 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=17460
Mon Feb 19 17:19:02 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:19:02 2007
This instance was first to mount
Mon Feb 19 17:19:02 2007
Starting background process ASMB
ASMB started with pid=22, OS id=17466
Starting background process RBAL
RBAL started with pid=23, OS id=17470
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:19:06 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:19:09 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:19:10 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:19:10 2007
Successful mount of redo thread 1, with mount id 3308997510
Mon Feb 19 17:19:10 2007
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:19:11 2007
ALTER DATABASE OPEN
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Mon Feb 19 17:19:11 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=26, OS id=17679
Mon Feb 19 17:19:11 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=27, OS id=17706
Mon Feb 19 17:19:11 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:19:11 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:19:11 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:19:11 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:19:11 2007
SMON: enabling cache recovery
Mon Feb 19 17:19:11 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=28, OS id=17708
Mon Feb 19 17:19:11 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:19:11 2007
SMON: enabling tx recovery
Mon Feb 19 17:19:11 2007
Database Characterset is WE8ISO8859P1
Mon Feb 19 17:19:11 2007
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Mon Feb 19 17:19:12 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:19:12 2007
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=17735
Mon Feb 19 17:19:13 2007
Completed: ALTER DATABASE OPEN
Mon Feb 19 17:20:11 2007
Shutting down archive processes
Mon Feb 19 17:20:16 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:23:35 2007
Starting background process EMN0
EMN0 started with pid=74, OS id=28350
Mon Feb 19 17:23:35 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:23:36 2007
Stopping background process QMNC
Mon Feb 19 17:23:36 2007
Stopping background process CJQ0
Mon Feb 19 17:23:38 2007
Stopping background process MMNL
Mon Feb 19 17:23:39 2007
Stopping background process MMON
Mon Feb 19 17:23:40 2007
Shutting down instance (immediate)
License high water mark = 34
Mon Feb 19 17:23:40 2007
Stopping Job queue slave processes
Mon Feb 19 17:23:40 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:24:07 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:24:07 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:24:08 2007
Shutting down archive processes
Archiving is disabled
Mon Feb 19 17:24:18 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC1: Archiving disabled
ARCH shutting down
Mon Feb 19 17:24:18 2007
ARC1: Archival stopped
Mon Feb 19 17:24:18 2007
ARC0: Becoming the heartbeat ARCH
Mon Feb 19 17:24:18 2007
ARC0: Archiving disabled
ARCH shutting down
ARC0: Archival stopped
Mon Feb 19 17:24:19 2007
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:24:19 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:24:19 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:24:19 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:24:19 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:24:25 2007
freeing rdom 0
Mon Feb 19 17:24:35 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
__shared_pool_size = 255852544
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 0
spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
__db_cache_size = 801112064
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +ORCL_DATA1
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:24:35 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=29950
DIAG started with pid=3, OS id=29952
PSP0 started with pid=4, OS id=29954
LMON started with pid=5, OS id=29956
LMD0 started with pid=6, OS id=29958
LMS0 started with pid=7, OS id=29960
LMS1 started with pid=8, OS id=29964
MMAN started with pid=9, OS id=29968
DBW0 started with pid=10, OS id=29970
LGWR started with pid=11, OS id=29972
CKPT started with pid=12, OS id=29974
SMON started with pid=13, OS id=29976
RECO started with pid=14, OS id=29978
CJQ0 started with pid=15, OS id=29980
MMON started with pid=16, OS id=29982
Mon Feb 19 17:24:36 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=29984
Mon Feb 19 17:24:36 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:24:36 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:24:39 2007
Reconfiguration started (old inc 0, new inc 4)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 1
* domain 0 valid = 0 according to instance 1
Mon Feb 19 17:24:39 2007
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Feb 19 17:24:39 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Feb 19 17:24:39 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Mon Feb 19 17:24:39 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:24:39 2007
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:24:39 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=30069
Mon Feb 19 17:24:40 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:24:41 2007
Starting background process ASMB
ASMB started with pid=22, OS id=30121
Starting background process RBAL
RBAL started with pid=23, OS id=30125
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:24:45 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:24:49 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:24:49 2007
Successful mount of redo thread 1, with mount id 3309017062
Mon Feb 19 17:24:49 2007
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Mon Feb 19 17:24:51 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:24:52 2007
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:24:52 2007
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Mon Feb 19 17:24:52 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=25, OS id=30406
Mon Feb 19 17:24:53 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=26, OS id=30422
Mon Feb 19 17:24:53 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:24:53 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:24:53 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:24:53 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:24:53 2007
SMON: enabling cache recovery
Mon Feb 19 17:24:53 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=27, OS id=30436
Mon Feb 19 17:25:07 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:25:07 2007
SMON: enabling tx recovery
Mon Feb 19 17:25:07 2007
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=30882
Mon Feb 19 17:26:31 2007
Completed: ALTER DATABASE OPEN
Mon Feb 19 17:26:38 2007
ALTER SYSTEM SET service_names='bsaprod','bsatest' SCOPE=MEMORY SID='bsaprod1';
Mon Feb 19 17:26:53 2007
Shutting down archive processes
Mon Feb 19 17:26:58 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:32:16 2007
Error: KGXGN aborts the instance (6)
Mon Feb 19 17:32:16 2007
Errors in file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_lmon_29956.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Mon Feb 19 17:32:17 2007
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_diag_29952.trc
Mon Feb 19 17:32:18 2007
Trace dumping is performing id=[cdmp_20070219173217]
Mon Feb 19 17:32:21 2007
Instance terminated by LMON, pid = 29956
and also the output of crs_stat -t in first node is....
Name Type Target State Host
ora....d1.inst application ONLINE OFFLINE
ora....d2.inst application ONLINE OFFLINE
ora....od1.srv application ONLINE OFFLINE
ora....od2.srv application ONLINE OFFLINE
ora....test.cs application ONLINE OFFLINE
ora.bsaprod.db application ONLINE OFFLINE
ora....SM1.asm application ONLINE UNKNOWN dsrvbd003
ora....ER.lsnr application ONLINE UNKNOWN dsrvbd003
ora....03.lsnr application ONLINE UNKNOWN dsrvbd003
ora....003.gsd application ONLINE UNKNOWN dsrvbd003
ora....003.ons application ONLINE UNKNOWN dsrvbd003
ora....003.vip application ONLINE ONLINE dsrvbd003
ora....SM2.asm application ONLINE OFFLINE
ora....04.lsnr application ONLINE OFFLINE
ora....004.gsd application ONLINE OFFLINE
ora....004.ons application ONLINE OFFLINE
ora....004.vip application ONLINE OFFLINE -
The Weblogic Server not start.
Weblogic 10.3.6.0 For SOA 11.1.1.6
When I start the weblogic server it does not start, the message that the server is running is not displayed.
Can anyone help me?
$ pwd
/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/bin
$ nohup ./startWebLogic.sh > /app/oracle/logs/weblogic.log 2>&1 &
]$ cat /app/oracle/logs/weblogic.log
** Setting up SOA specific environment...
EXTRA_JAVA_PROPERTIES= -da:org.apache.xmlbeans...
LD_LIBRARY_PATH=/app/oracle/product/fwm11g/patch_wls1036/profiles/default/native:/app/oracle/product/fwm11g/wlserver_10.3/server/native/linux/x86_64:/app/oracle/product/fwm11g/wlserver_10.3/server/native/linux/x86_64/oci920_8:/app/oracle/product/fwm11g/wlserver_10.3/sip/server/native/linux/x86_64:/app/oracle/product/fwm11g/wlserver_10.3/sip/server/native/linux/x86_64/oci920_8:/app/oracle/product/fwm11g/Oracle_SOA1/soa/thirdparty/edifecs/XEngine/bin
USER_MEM_ARGS=-Xms768m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m
** End SOA specific environment setup
** SOA specific environment is already set, skipping...
JAVA Memory arguments: -Xms768m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m
WLS Start Mode=Development
CLASSPATH=/app/oracle/product/fwm11g/oracle_common/modules/oracle.jdbc_11.1.1/ojdbc6dms.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/user-patch.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/soa-startup.jar::/app/oracle/product/fwm11g/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java/jdk1.6.0_25/lib/tools.jar:/app/oracle/product/fwm11g/wlserver_10.3/server/lib/weblogic_sp.jar:/app/oracle/product/fwm11g/wlserver_10.3/server/lib/weblogic.jar:/app/oracle/product/fwm11g/modules/features/weblogic.server.modules_10.3.6.0.jar:/app/oracle/product/fwm11g/wlserver_10.3/server/lib/webservices.jar:/app/oracle/product/fwm11g/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/app/oracle/product/fwm11g/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/app/oracle/product/fwm11g/wlserver_10.3/sip/server/lib/weblogic_sip.jar:/app/oracle/product/fwm11g/oracle_common/soa/modules/commons-cli-1.1.jar:/app/oracle/product/fwm11g/oracle_common/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/oracle.soa.common.adapters_11.1.1/oracle.soa.common.adapters.jar:/app/oracle/product/fwm11g/oracle_common/webcenter/modules/oracle.portlet.server_11.1.1/oracle-portlet-api.jar:/app/oracle/product/fwm11g/Oracle_SOA1/communications/modules/usermessaging-config_11.1.1.jar:/app/oracle/product/fwm11g/oracle_common/modules/oracle.xdk_11.1.0/xsu12.jar:/app/oracle/product/fwm11g/modules/features/weblogic.server.modules.xquery_10.3.1.0.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/db2jcc4.jar:/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/config/soa-infra:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/fabric-url-handler_11.1.1.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/quartz-all-1.6.5.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/oracle.soa.fabric_11.1.1/oracle.soa.fabric.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/oracle.soa.fabric_11.1.1/fabric-runtime-ext-wls.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/oracle.soa.adapter_11.1.1/oracle.soa.adapter.jar:/app/oracle/product/fwm11g/Oracle_SOA1/soa/modules/oracle.soa.b2b_11.1.1/oracle.soa.b2b.jar:/app/oracle/product/fwm11g/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/app/oracle/product/fwm11g/wlserver_10.3/common/derby/lib/derbyclient.jar:/app/oracle/product/fwm11g/wlserver_10.3/server/lib/xqrl.jar
PATH=/app/oracle/product/fwm11g/wlserver_10.3/server/bin:/app/oracle/product/fwm11g/modules/org.apache.ant_1.7.1/bin:/usr/java/jdk1.6.0_25/jre/bin:/usr/java/jdk1.6.0_25/bin:/usr/kerberos/bin:/usr/java/default/bin:/usr/local/bin:/bin:/usr/bin:/home/appsoa/bin
* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http://hostname:port/console *
starting weblogic with Java version:
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)
Starting WLS with line:
/usr/java/jdk1.6.0_25/bin/java -client -Xms768m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/app/oracle/product/fwm11g/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -ea:com.bea.wcp... -da -Dplatform.home=/app/oracle/product/fwm11g/wlserver_10.3 -Dwls.home=/app/oracle/product/fwm11g/wlserver_10.3/server -Dweblogic.home=/app/oracle/product/fwm11g/wlserver_10.3/server -Dcommon.components.home=/app/oracle/product/fwm11g/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain -Djrockit.optfile=/app/oracle/product/fwm11g/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/config/fmwconfig -Digf.arisidbeans.carmlloc=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/config/fmwconfig/carml -Digf.arisidstack.home=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/app/oracle/product/fwm11g/oracle_common/modules/oracle.ossoiap_11.1.1,/app/oracle/product/fwm11g/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.jdbc.remoteEnabled=false -Doracle.security.jps.policy.migration.validate.principal=false -Dportlet.oracle.home=/app/oracle/product/fwm11g/oracle_common -da:org.apache.xmlbeans... -Dbpm.enabled=true -Dsoa.archives.dir=/app/oracle/product/fwm11g/Oracle_SOA1/soa -Dsoa.oracle.home=/app/oracle/product/fwm11g/Oracle_SOA1 -Dsoa.instance.home=/app/oracle/product/fwm11g/user_projects/domains/fwm_domain -Dtangosol.coherence.clusteraddress=227.7.7.9 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/app/oracle/product/fwm11g/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/app/oracle/product/fwm11g/oracle_common -Djava.awt.headless=true -Dbam.oracle.home=/app/oracle/product/fwm11g/Oracle_SOA1 -Dums.oracle.home=/app/oracle/product/fwm11g/Oracle_SOA1 -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/app/oracle/product/fwm11g/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
<Jul 26, 2012 3:39:30 AM BRT> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jul 26, 2012 3:39:30 AM BRT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Jul 26, 2012 3:39:30 AM BRT> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/lib/sipactivator.jar>
<Jul 26, 2012 3:39:31 AM BRT> <Info> <Server> <BEA-002647> <The service plugin, com.oracle.core.sip.activator, was added from /app/oracle/product/fwm11g/user_projects/domains/fwm_domain/lib/sipactivator.jar.>
<Jul 26, 2012 3:39:31 AM BRT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 20.0-b11 from Sun Microsystems Inc.>
<Jul 26, 2012 3:39:31 AM BRT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050 >
<Jul 26, 2012 3:39:34 AM BRT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Jul 26, 2012 3:39:34 AM BRT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Jul 26, 2012 3:39:34 AM BRT> <Notice> <LoggingService> <BEA-320400> <The log file /app/oracle/product/fwm11g/user_projects/domains/fwm_domain/servers/AdminServer/logs/AdminServer.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
<Jul 26, 2012 3:39:34 AM BRT> <Notice> <LoggingService> <BEA-320401> <The log file has been rotated to /app/oracle/product/fwm11g/user_projects/domains/fwm_domain/servers/AdminServer/logs/AdminServer.log00006. Log messages will continue to be logged in /app/oracle/product/fwm11g/user_projects/domains/fwm_domain/servers/AdminServer/logs/AdminServer.log.>
<Jul 26, 2012 3:39:34 AM BRT> <Notice> <Log Management> <BEA-170019> <The server log file /app/oracle/product/fwm11g/user_projects/domains/fwm_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
<Jul 26, 2012 3:39:42 AM BRT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Jul 26, 2012 3:39:48 AM BRT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Jul 26, 2012 3:39:48 AM BRT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Jul 26, 2012 3:39:53 AM BRT> <Warning> <J2EE> <BEA-160195> <The application version lifecycle event listener oracle.security.jps.wls.listeners.JpsAppVersionLifecycleListener is ignored because the application em is not versioned.>
<Jul 26, 2012 3:39:57 AM BRT> <Notice> <SipServer.Resource> <BEA-332401> <Initializing SipServer Resource with configuration com.bea.wcp.sip.management.descriptor.beans.SipServerBeanImpl>
<Jul 26, 2012 3:39:57 AM BRT> <Notice> <SipServer.Resource> <BEA-332404> <Engine AdminServer is in NON-REPLICATED mode>
<Jul 26, 2012 3:39:58 AM BRT> <Notice> <WLSS.Engine> <BEA-330071> <WebLogic Sip Server "AdminServer" patch version: WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050
Javax Server Pages Client Capable 1.3 Tue Jun 21 09:59:39 EDT 2011
Expression Language 2.1 for JSP 1.1 Tue Jun 21 10:21:26 EDT 2011
Javax Enterprise Servlets Client Capable 1.0 Thu Aug 2 12:41:25 EDT 2007
Eclipse Java Development Tools 3.5.3 Fri May 20 04:56:54 EDT 2011
WebLogic java compiler utils package Client Capable 1.2 Thu Feb 11 03:38:50 EST 2010
WebLogic WebApp Container Public API Client Capable 1.4 Fri Oct 1 20:01:15 PDT 2010
Oracle WebLogic Server Module Dependencies 10.3 Thu Sep 29 17:47:37 EDT 2011
Oracle WebLogic Server on JRockit Virtual Edition Module Dependencies 10.3 Wed Jun 15 17:54:24 EDT 2011
ANTLR Java based compiler generator Client 2.7 Mon Jun 11 12:19:48 EDT 2007
WebLogic Descriptors for J2EE 1.6 Wed Dec 1 17:14:50 EST 2010
WebLogic Descriptors for J2EE 1.6 Binding Bundle
WebLogic Specific Descriptors 1.4 Mon Aug 8 09:26:15 MDT 2011
WebLogic Specific Descriptors 1.4 Binding Bundle
WebLogic Datasource 1.10 Sat Nov 12 08:11:09 PST 2011
WebLogic Datasource 1.10 Binding Bundle
WebLogic Beangen Client Capable 1.7 Wed Feb 24 16:02:48 PST 2010
WebLogic Beangen 1.7 Binding Bundle
WLDF Accessor Client Capable 1.5 Fri Sep 3 17:10:52 EDT 2010
WLDF Accessor 1.5 Binding Bundle
WebLogic Management Core Interfaces Client Capable 2.9 Thu Aug 11 17:17:14 PDT 2011
WebLogic Management Core Interfaces 2.9 Binding Bundle
WebLogic EJBGen Client Capable 1.1 Tue Nov 2 03:30:53 PDT 2010
Apache Byte Code Engineering Library (BCEL) extracted from 5.2.zip from http://jakarta.apache.org/site/downloads/downloads_bcel.cgi with packages renamed from org.apache.bcel to com.bea.core.repackaged.apache.bcel Client 5.2 Fri May 20 04:58:21 EDT 2011
Apache commons collections package 3.2 Tue Mar 20 15:48:25 MDT 2007
Apache commons lang package 2.1 Tue Mar 20 15:48:30 MDT 2007
Apache commons pool package 1.3 Tue Mar 20 15:48:36 MDT 2007
Apache commons io 1.4 package 1.0 Wed Jun 2 17:36:36 EDT 2010
Apache commons fileupload 1.2.1 package 1.0 Wed Jun 2 17:36:36 EDT 2010
Apache DOM implementation 1.0 Tue Mar 20 15:36:46 MDT 2007
Apache Logging Support 1.0 Tue Mar 20 15:36:50 MDT 2007
Apache OpenJPA classes 1.3 Thu Sep 22 01:28:26 PDT 2011
XMLBeans - Apache SVN rev 1102771 2.2 Thu Sep 1 08:05:25 PDT 2011
BEA Logging Runtime Support Client Capable 1.9 Wed Jul 27 14:54:39 EDT 2011
BEA Common Security Open SAML 1.0 Sat Jul 16 04:41:59 MDT 2011
BEA OpenSAML 2.0 1.0 Fri Aug 19 08:44:53 MDT 2011
bea-harvester-api2.0 Client Capable 2.3 Mon Feb 15 14:41:06 EST 2010
bea-harvester-jmx2.0 Client Capable 2.4 Wed Aug 10 20:11:27 PDT 2011
bea-harvester-utils Client Capable 1.4 Mon Feb 15 14:41:06 EST 2010
bea-mbean-typing-util 1.4 Wed Feb 24 19:15:33 EST 2010
Javolution 3.7.19 3.7 Tue Aug 28 17:32:21 PDT 2007
Joda-time 1.2.1 1.2 Tue Aug 28 17:32:27 PDT 2007
BEA STAX Build Time Support 1.6 Tue Apr 19 13:32:59 EDT 2011
BEA STAX Runtime Time Support Client Capable 1.8 Thu Sep 1 08:05:25 PDT 2011
BEA Generic Annotations Client Capable 1.4 Thu Jun 2 23:58:32 EDT 2011
BEA Kodo 1.5 Wed Oct 12 03:50:17 EDT 2011
BEA Kodo Integration Client Capable 1.7 Tue Jul 19 08:57:22 MDT 2011
BEA Kodo Integration 1.7 Binding Bundle
BEA Kodo Integration Tools 1.3 Sat Feb 13 09:30:33 PST 2010
XML Beans Marshalling (package renamed com.bea) SVN 1102771 2.5 Thu Sep 1 08:05:25 PDT 2011
WebLogic Utils Client Capable 1.10 Sat Oct 29 15:34:23 MDT 2011
Aspect 5.3 Fri May 20 05:19:03 EDT 2011
BEA Apache Commons Logging Repackaged 1.2 Fri Sep 9 15:46:56 EDT 2011
Spring Framework 1.2 Fri May 20 05:19:03 EDT 2011
Pitchfork 1.4 Tue Sep 6 13:38:59 EDT 2011
${description} 1.3 Wed Jun 29 20:42:12 EDT 2011
CSS i18n 1.0 Fri Aug 19 08:44:53 MDT 2011
CSS xacml 1.0 Fri Aug 19 08:44:53 MDT 2011
SAML2 Utils 1.0 Fri Aug 19 08:44:53 MDT 2011
BEA Common Security Engine Implementation 1.0 Fri Aug 19 08:44:53 MDT 2011
BEA Common Security Engine Interfaces 1.0 Fri Aug 19 08:44:53 MDT 2011
BEA Common Security API 1.0 Fri Aug 19 08:44:53 MDT 2011
BEA Common Security Implementation 1.0 Fri Aug 19 08:44:53 MDT 2011
BEA Common Security JDK Utilities 1.0 Fri Aug 19 08:44:53 MDT 2011
Security Utilities 1.0 Fri Aug 19 08:44:53 MDT 2011
Common Security SAML 2.0 1.0 Fri Aug 19 08:44:53 MDT 2011
Common Security SAML 2.0 Management JavaBeans 1.0 Fri Aug 19 08:44:53 MDT 2011
Security Provider Utilities 1.0 Fri Aug 19 08:44:53 MDT 2011
SAML Utils 1.0 Fri Aug 19 08:44:53 MDT 2011
XACML Utils 1.0 Fri Aug 19 08:44:53 MDT 2011
Security Provider Environment 1.0 Fri Aug 19 08:44:53 MDT 2011
RSA certj 5.2 Wed Aug 31 17:49:27 PDT 2011
Netscape LDAP JDK 1.3 Mon Apr 11 20:11:21 PDT 2011
Commons Networking Utilty classes 1.0 Wed Feb 6 15:01:03 PST 2008
WebLogic SAAJ 1.8 Mon Oct 17 02:49:29 PDT 2011
WebLogic STAX Client Capable 1.10 Wed Jun 8 09:12:28 EDT 2011
jaxb-impl.jar taken from Glassfish JAXB 2.1.14 1.1 Wed Oct 12 19:46:41 PDT 2011
jaxb-impl.jar taken from Glassfish JAXB 2.1.14 1.2 Wed Oct 12 19:46:43 PDT 2011
resolver.jar taken from Glassfish JAXWS 2.1.5 1.0 Thu Dec 3 11:46:24 EST 2009
Fastinfoset.jar taken from Glassfish JAXWS 2.1.5 1.0 Thu Dec 3 11:46:27 EST 2009
jaxws-rt.jar taken from Glassfish JAXWS 2.1.5 1.3 Fri Oct 14 19:19:11 EDT 2011
Java.net implementation of MimePull.jar taken from Glassfish JAXWS 2.1.5 1.1 Wed Jul 27 16:50:24 MDT 2011
Codehaus STaX Interfaces 3.0.1 1.0 Mon Mar 8 20:49:50 PST 2010
Woodstox STaX Parser 4.0.5 1.0 Thu Dec 3 11:35:43 EST 2009
jaxws-tools.jar taken from Glassfish JAXWS 2.1.5 1.2 Sat Apr 2 00:42:47 PDT 2011
Java.net Stax Extensions 1.0 Tue Jun 3 07:12:06 PDT 2008
Java.net xml stream buffer 1.0 Fri Oct 22 10:08:31 PDT 2010
Jakarta ORO 1.0 Wed Feb 6 15:01:03 PST 2008
Javax Enterprise Activation 1.1 Tue Apr 8 09:31:17 PDT 2008
Javax Annotation 1.0 Fri Dec 25 09:02:47 PST 2009
Javax Interceptor 1.0 Tue Mar 20 15:37:16 MDT 2007
Javax Enterprise Beans 3.0 Mon Jun 11 12:21:01 EDT 2007
Java Data Objects 2.0 Mon Jun 11 12:20:56 EDT 2007
Java Enterprise Deployment APIs 1.2 Tue Mar 20 15:37:28 MDT 2007
Java Enterprise Messaging 1.1 Mon Jun 11 12:21:11 EDT 2007
Java Web Services 2.0 Tue Mar 20 15:37:37 MDT 2007
Javax Enterprise Mail 1.1 Mon Jul 6 10:41:09 MDT 2009
Java Enterprise Management APIs 1.0 Tue Mar 20 15:37:49 MDT 2007
Java Persistence Client Capable 1.0 Tue Oct 7 12:18:34 PDT 2008
Java Connector 1.5 Mon Jun 11 12:22:07 EDT 2007
Java Authorization Contract for Containers 1.0 Wed Feb 6 15:01:03 PST 2008
Javax Transaction APIs Client Capable 1.0 Thu Aug 2 12:42:14 EDT 2007
JAXB 2.1 Mon Jun 11 12:22:53 EDT 2007
Java XML Registry 1.0 Wed Feb 6 15:01:03 PST 2008
Java XML Soap Extensions 1.3 Mon Jun 11 12:22:59 EDT 2007
Java Stream XML Extensions 1.1 Mon Jun 11 12:23:05 EDT 2007
JAX-WS APIs 2.1 Mon Jun 11 12:23:16 EDT 2007
Java API for XML-based RPC 1.2 Mon Jun 11 12:23:10 EDT 2007
Monfox Dynamic SNMP Agent 1.1 Fri Mar 19 05:46:27 MDT 2010
Serp bytecode manipulation framework 1.14.4 Thu Oct 6 01:32:56 EDT 2011
WebLogic Apache Classes Client Capable 1.3 Mon Sep 19 23:58:26 EDT 2011
WebLogic BeanInfo Caching and Discovery Client Capable 2.4 Sat Oct 25 20:46:29 PDT 2008
WebLogic Descriptor Client Capable 1.10 Wed Aug 10 12:59:06 PDT 2011
Repackaged ASM-3.2 1.1 Fri May 20 04:56:54 EDT 2011
Repackaged asm-commons-3.2 1.1 Fri May 20 04:56:54 EDT 2011
Repackaged asm-tree-3.2 1.1 Fri May 20 04:56:54 EDT 2011
Repackaged asm-util-3.2 1.1 Fri May 20 04:56:54 EDT 2011
Oracle JFR 1.0 Thu Feb 18 19:06:33 PST 2010
WebLogic Diagnostics Core Interfaces Client Capable 2.6 Thu Oct 6 01:11:08 EDT 2011
WebLogic Diagnostics Logging Client Capable 1.2 Fri Dec 12 11:37:59 MST 2008
WebLogic Diagnostics Query Module Client Capable 1.3 Fri Jul 1 07:32:00 PDT 2011
WebLogic Diagnostics Instrumentor Tool 1.8 Thu Oct 6 01:11:08 EDT 2011
WebLogic Diagnostics Instrumentor Config Tool 1.8 Thu Oct 6 01:11:08 EDT 2011
WebLogic Diagnostics JRockit Flight Recorder Interfaces Client Capable 1.2 Wed Dec 1 17:41:28 EST 2010
Diagnostics Notifications Module Client Capable 1.4 Sun Nov 22 16:03:32 PST 2009
BEA Logging Runtime Support Client Capable 1.5 Thu Apr 29 20:43:42 EDT 2010
WebLogic i18n Runtime Support Client Capable 1.9 Thu Sep 1 07:41:47 PDT 2011
WebLogic i18n Build Support Client Capable 1.5 Fri Feb 19 15:03:15 EST 2010
WebLogic I18N tools Client Capable 1.4 Thu Sep 1 07:41:47 PDT 2011
WebLogic Management JMX Interfaces 1.4 Fri Sep 16 16:19:28 EDT 2011
WebLogic Security Provider Generation Tool 1.5 Wed Oct 14 16:39:28 MDT 2009
WebLogic Security Provider Generation Tool Client Capable 1.5 Wed Oct 14 16:39:28 MDT 2009
WebLogic Messaging Kernel Client Capable 1.8 Mon Aug 23 21:42:11 EDT 2010
WebLogic Resource Pool Client Capable 1.8 Thu Oct 6 16:06:35 PDT 2011
WebLogic Socket Muxer API Client Capable 1.3 Thu Aug 18 16:24:35 EDT 2011
WebLogic RMI Client Capable 1.11 Tue Sep 20 15:07:37 EDT 2011
Common Security WebLogic Server Integration Support 1.0 Fri Aug 19 08:44:53 MDT 2011
Server Lifecycle Interfaces Client Capable 1.5 Fri Dec 10 00:36:46 EST 2010
WebLogic Store Client Capable 1.8 Mon Oct 3 09:57:28 PDT 2011
WebLogic STORE GXA Client Capable 1.7 Fri Apr 1 14:30:50 PDT 2011
WebLogic Store Admin Tool Client Capable 1.3 Thu Apr 28 09:32:45 PDT 2011
WebLogic JDBC Store Client Capable 1.3 Fri Sep 16 08:41:14 MDT 2011
WebLogic JTA implementation Client Capable 2.7 Sat Oct 15 07:12:58 PDT 2011
WebLogic Utils 1.10 Sat Oct 29 15:34:23 MDT 2011
Agent Utililities 1.1 Tue Feb 16 00:16:03 EST 2010
WebLogic Utility Classloader implementations Client Capable 2.0 Wed May 18 10:00:41 PDT 2011
WebLogic Utils for working with Expressions Client Capable 1.4 Tue Sep 29 14:45:53 EDT 2009
WebLogic Utils for Dynamically Generated Class Wrappers Client Capable 1.4 Fri Feb 13 14:44:23 MST 2009
WebLogic Timers Client Capable 1.7 Fri Feb 4 14:23:26 MST 2011
WebLogic Work Manager Client Capable 1.11 Thu Oct 6 11:12:55 PDT 2011
WebLogic Workarea Client Capable 1.8 Tue Jun 28 04:08:48 EDT 2011
WebLogic XML XPath Implementation Client Capable 1.5 Thu Sep 1 22:11:12 EDT 2011
Weblogic Tuxedo Connector Core Client Capable 1.6 Sat Jul 16 15:02:56 MDT 2011
WebLogic Security 1.0 Fri Aug 19 08:44:53 MDT 2011
Weblogic Server Java Authentication Helper Classes Client Capable 1.1 Mon Jul 5 20:42:35 EDT 2010
Weblogic Server Message Digest Utilities Client Capable 1.0 Thu Aug 2 12:51:30 EDT 2007
Weblogic Server Authenticated Subject Client Capable 1.2 Wed Dec 1 17:41:28 EST 2010
Weblogic Server Authenticated Subject Client Capable 1.6 Wed Oct 19 14:48:23 PDT 2011
PrintingSecurityManager - PSM 1.1 Tue Feb 16 05:30:08 PST 2010
WebLogic security ssl classes 1.0 Tue Jun 15 17:39:53 EDT 2010
WebLogic Nodemanager Plugin Client Capable 1.3 Tue Nov 18 18:23:10 EST 2008
nodemanager module for managed processes 1.1 Thu Sep 29 17:28:28 EDT 2011
WebLogic JMS Pool Client Capable 1.9 Wed Apr 13 13:03:26 EDT 2011
Contains compiled schema type from WLS 9.0 for WLP compatibility 1.4 Thu Sep 1 08:29:31 PDT 2011
WebLogic Http Pub/Sub Module Client Capable 1.7 Fri Jul 8 13:06:46 EDT 2011
Class Redefinition Project 1.6 Tue Jul 19 08:57:22 MDT 2011
Class Redefinition Project Client Capable 1.6 Tue Jul 19 08:57:22 MDT 2011
Class Redefinition Project 1.6 Binding Bundle
Commonj SDO 1.0 Wed Sep 24 19:11:23 PDT 2008
WebLogic Coherence Descriptor 1.2 Thu Sep 1 08:29:31 PDT 2011
WebLogic Coherence Descriptor 1.2 Binding Bundle
This module contains all message catalogs 1.2 Wed Aug 24 03:32:14 EDT 2011
WebLogic WebService Public API's 1.1 Tue Sep 21 22:15:05 EDT 2010
WebLogic EclipseLink Integration 1.0 Thu Feb 25 14:56:43 PST 2010
WebLogic SCA Client 1.0 Thu Feb 25 00:27:10 EST 2010
WebLogic RAC Module UCP Client Capable 1.1 Thu Oct 6 16:06:35 PDT 2011
BEA Patches of apache ant Client Capable 1.2 Wed Jan 13 08:48:17 PST 2010
Oracle WebLogic Server 10.3.6.0 at 728648 built on: 2011/09/13
Oracle WebLogic Server 10.3.6.0 at 728648 built on: 2011/09/13
Oracle WebLogic Server Datatier 10.3.6.0 at 728648 built on: 2011/09/13
Oracle WebLogic Server 10.3.6.0 at 728648 built on: 2011/09/13
WebLogic SIPServer Extension API 10.3.6.0 at 727725 built on 2011/08/25
WebLogic SIPServer Extension API 10.3.6 at 727725 built on 2011/08/25
WebLogic SIPServer CallState 10.3.6 at 727725 built on 2011/08/25
Oracle WebLogic Communications SCTP 10.3.1 at 688160 built on 2009/03/11
ALSB Client 1.3 Mon Apr 12 19:21:38 PDT 2010
WebLogic SCA Engine 1.3 Thu Jul 28 14:17:02 MDT 2011
WebLogic WebService Databinding Plugins 1.3 Wed Dec 1 17:41:28 EST 2010
WebLogic WebService Databinding 1.3 Tue Oct 18 16:38:56 EDT 2011
WebLogic SIP Activator 10.3.6 at 729166 built on 2011/09/22
>
<Jul 26, 2012 3:39:58 AM BRT> <Notice> <WLSS.Engine> <BEA-330065> <SIP server replication is DISABLED>
<Jul 26, 2012 3:39:58 AM BRT> <Notice> <WLSS.Engine> <BEA-330000> <WebLogic SIP Server "AdminServer" has started.>
<Jul 26, 2012 3:39:58 AM BRT> <Notice> <WLSS.Transport> <BEA-330687> <Thread "SIP Message processor (Transport TCP)" is listening on port 5060>
<Jul 26, 2012 3:39:58 AM BRT> <Notice> <WLSS.Transport> <BEA-330687> <Thread "SIP Message processor (Transport TCP)" is listening on port 5061>
<Jul 26, 2012 3:39:59 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_soainfra:total_faults" has no key column. It will not be collected.>
<Jul 26, 2012 3:39:59 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "discussion:Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:39:59 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_oim:overall" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "portal:portal_page_engine_response_codes_delta" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "portal:portal_page_engine_response_codes" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "portal:portal_page_engine" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "reports:Reports_Server_Information" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "reports:Reports_Server_Performance" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "reports:Reports_Server_Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "reports:Reports_Servlet_Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "reports:Remote_Bridge_Elements" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_federation:protocol_profiles_rollup" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_federation:protocol_profiles_events_rollup" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_federation:protocol_profiles_protocol_phase_events_rollup" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "domain_oracle_oaam:OAAM_Datasource" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "activitygraph:Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "domain_oracle_oim:jdbc" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "webcache:request_filter_denied_stats" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_forms:aggr_FormsResourceUsage" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_forms:NumberOfSessions" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_forms:JVMResourceUsage" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:00 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "oracle_forms:servlet_rollup_agg" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "analytics:Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "personalization:Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "portaltools:Response" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "weblogic_j2eeserver:jvm_runtime" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "weblogic_j2eeserver:jvm_compilation_time" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "weblogic_j2eeserver:jvm_memory_usage" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "weblogic_j2eeserver:jvm_class_loading" has no key column. It will not be collected.>
<Jul 26, 2012 3:40:01 AM BRT> <Warning> <oracle.dms.collector> <BEA-000000> <Metric table "weblogic_j2eeserver:jvm_threads" has no key column. It will not be collected.>check the server log;
/app/oracle/product/fwm11g/user_projects/domains/fwm_domain/servers/AdminServer/logs/AdminServer.log
you can launch the console and see if it is running; http://<server>:<port>/console -
CSS does not start using 11g RH4 SCSI drives
Hi All...
Followed all necessary procedures for setting up oracle. The problem is when I try and get ASM working, I get to the part that says CSS is not running and then do a $ORACLE_HOME/bin/localconfig add as root with all environmental variables set up properly. Furthermore, oracleasm was previously installed, properly configured because I can run the scan, query and list and it comes back correctly.
[root@db raw]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db raw]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
At the end of this process CSS fails to start. There is nothing placed into the /var/tmp/.oracle directory. I then ran the $ORACLE_HOME/bin/localconfig reset $ORACLE_HOME and get the same result.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started
While it is running I see:
oracle:/u01/app/oracle> ps -ef | grep crs
root 18399 18142 22 19:31 pts/2 00:00:12 /u01/app/oracle/product/11.1.0/db_1/bin/crsctl.bin check install -wait 600
oracle 18521 18432 0 19:32 pts/4 00:00:00 grep crs
ps -ef | grep css
oracle 18653 18432 0 19:33 pts/4 00:00:00 grep css
[root@db oracle]# tail ocr.loc
ocrconfig_loc=/u01/app/oracle/product/11.1.0/db_1/cdata/localhost/local.ocr
local_only=TRUE
Contents of css log file in $ORACLE_HOME/log/db/client
2009-02-05 15:30:26.331: [ CSSCLNT][2556107008]clsssInitNative: failed to connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db_)), rc 9
Contents of cslcfg log is as follows:
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:3: Problem reading buffer 53b000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:3: Problem reading buffer 53e000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.403: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:3: Problem reading buffer 53e000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]ibctx:1:ERROR: INVALID FORMAT
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]proprinit:problem reading the bootblock or superbloc 22
2009-02-05 15:20:14.404: [ default][2557164416]a_init:7!: Backend init unsuccessful : [22]
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:3: Problem reading buffer 531000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:3: Problem reading buffer 531000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:3: Problem reading buffer 531000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]ibctx:1:ERROR: INVALID FORMAT
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]proprinit:problem reading the bootblock or superbloc 22
2009-02-05 15:20:14.404: [ default][2557164416]a_init:7!: Backend init unsuccessful : [22]
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 534000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 533000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 533000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]ibctx:1:ERROR: INVALID FORMAT
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]proprinit:problem reading the bootblock or superbloc 22
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 534000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 533000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.485: [ OCRRAW][2557164416]propriowv: Vote information on disk 0 [u01/app/oracle/product/11.1.0/db_1/cdata/localhost/local.ocr] is adjusted from [0/0] to [2/2]
2009-02-05 15:20:14.815: [ OCRRAW][2557164416]iniconfig:No 92 configuration
2009-02-05 15:20:14.815: [ OCRAPI][2557164416]a_init:6a: Backend init successful
There are no logs in the $ORACLE_HOME/bin/log/db/cssd directory.
Here is our disk information:
brw-rw---- 1 root disk 8, 0 Feb 5 09:56 sda
brw-rw---- 1 root disk 8, 1 Feb 5 09:56 sda1
brw-rw---- 1 root disk 8, 2 Feb 5 09:56 sda2
brw-rw---- 1 root disk 8, 16 Feb 5 09:56 sdb
brw-rw---- 1 root disk 8, 17 Feb 5 14:57 sdb1
brw-rw---- 1 root disk 8, 18 Feb 5 14:57 sdb2
brw-rw---- 1 root disk 8, 32 Feb 5 09:56 sdc
brw-r----- 1 oracle oinstall 8, 33 Feb 5 14:57 sdc1
brw-rw---- 1 root disk 8, 48 Feb 5 09:56 sdd
brw-r----- 1 oracle oinstall 8, 49 Feb 5 14:57 sdd1
brw-rw---- 1 root disk 8, 64 Feb 5 09:56 sde
brw-r----- 1 oracle oinstall 8, 65 Feb 5 14:57 sde1
brw-rw---- 1 root disk 8, 80 Feb 5 09:56 sdf
brw-r----- 1 oracle oinstall 8, 81 Feb 5 14:57 sdf1
Raw devices:
crw------- 1 oracle oinstall 162, 1 Feb 5 14:57 raw1
crw------- 1 oracle oinstall 162, 2 Feb 5 14:57 raw2
crw------- 1 oracle oinstall 162, 3 Feb 5 14:57 raw3
crw------- 1 oracle oinstall 162, 4 Feb 5 14:57 raw4
Tried
[root@db init.d]# ./init.cssd start
Startup will be queued to init within 30 seconds.
[root@db init.d]# ps -ef | grep css
root 19962 17418 0 19:50 pts/2 00:00:00 grep css
Okay, so what is wrong here? Why won;t CSS start? Checked every piece of literature and search engine results and tried almost everything...
So, dear forum people... Help.
Regards...I got same issue with Oracle 10.2.3 on Solaris. This is a single node setting.
Tried the localconfig reset with root, same issue.
Need help.
cssd/cssdOUT.log
setpriority: unable to escalate to priority -20 (13)
client/css.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2009-03-22 11:44:46.916: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
alertos631std.log
2009-03-22 11:44:31.921
[client(10951)]CRS-1006:The OCR location /opt/oracle/current/cdata/localhost/local.ocr is inaccessible. Details in /opt/oracle/current/log/os631std/client/clscfg.log.
2009-03-22 11:44:31.924
[client(10951)]CRS-1006:The OCR location /opt/oracle/current/cdata/localhost/local.ocr is inaccessible. Details in /opt/oracle/current/log/os631std/client/clscfg.log.
2009-03-22 11:44:32.486
[client(10951)]CRS-1001:The OCR was formatted using version 2.
clscfg.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2009-03-22 11:44:31.919: [ OCROSD][1]utread:3: problem reading buffer 2140000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.919: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.919: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.919: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.919: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2009-03-22 11:44:31.920: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.920: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.921: [ OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2009-03-22 11:44:31.921: [ OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
2009-03-22 11:44:31.921: [ default][1]a_init:7!: Backend init unsuccessful : [22]
2009-03-22 11:44:31.923: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.923: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.923: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.923: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2009-03-22 11:44:31.924: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.924: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2009-03-22 11:44:31.924: [ OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
2009-03-22 11:44:31.924: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.924: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.924: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2009-03-22 11:44:32.018: [ OCRRAW][1]propriowv: Vote information on disk 0 [opt/oracle/current/cdata/localhost/local.ocr] is a
djusted from [0/0] to [2/2]
2009-03-22 11:44:32.486: [ OCRRAW][1]propriniconfig:No 92 configuration
2009-03-22 11:44:32.486: [ OCRAPI][1]a_init:6a: Backend init successful
Edited by: user462878 on Mar 22, 2009 9:03 AM
Edited by: user462878 on Mar 22, 2009 9:03 AM -
Hi guys,
After installing Oracle 10g on Solaris 9, I could not get the database to start up on the web browser. I was wondering is the problem is due to the error below. If yes, what steps should I take to remedy it?
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'other'..
Operation successful.
Oracle Cluster Registry for cluster has been initialized
Usage: /etc/init.d/init.cssd start
Usage: /etc/init.d/init.cssd start
cat: cannot open /css/admin/inittab_local
Adding to inittab
/export/home/app/oracle/product/10.1.0/db_1/bin/localconfig:
/usr/bin/cp: not found
Sep 6 16:50:06 unknown root: [ID 702911 user.alert] (Oracle CSSD will
be run out of init)
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS
service is started
Thanx!Thanks for replying. My database does not work with ASM.
I've reinstalled Oracle10g on Solaris as I believed that I made some errors for the previous versions.
Now, for this current Oracle, I can access the database using SQL and dbconsole service to start the EM. However I can't access the EM's database through the web browser after the first reboot.
i am able to use the EM using the path
http://unknown:5500/em
Note: My hostname is unknown as i had used DHCP during installation.
But once after reboot, the following error msgs appear at the startup and i could not access the EM:
Oracle CSSD being stopped
Shutting down CRS daemon
Shutting down EVM daemon
Failure 3 at CSS context initialization
Oracle CSSD will be run out of init
My qn is if it is my unknown hostname that cause the problem? Or is there any other soln pertaining to the error msg. -
RAC does not start after trying to move storage
Hello,
We have a two nodes RAC Oracle 10g and ASM groups defined on one external storage.
RAC does not start after we reformated the storage to RAID 10.
What we did:
1) deleted the database
2) reformated the external storage to RAID 10 and defined needed volumes for OCR, VotingDisk and ASM
3) on both nodes
- recreated VotingDisk
node1:root$ /u01/oracle/product/10gr2/crs/bin/crsctl add css votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3 -force
Now formatting voting disk: /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3
successful addition of votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3.
node2:root$ /u01/oracle/product/10gr2/crs/bin/crsctl add css votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3 -force
Now formatting voting disk: /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3
successful addition of votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3.
- tried to move OCR but was able only on one node:
/u01/oracle/product/10gr2/crs/bin/ocrconfig -replace ocrmirror /dev/rdsk/c7t30d0s3
/u01/oracle/product/10gr2/crs/bin/ocrconfig -replace ocr /dev/rdsk/c7t29d0s3
/u01/oracle/product/10gr2/crs/bin/ocrconfig -replace ocr
These return ok for both node1 and node2:
node1:oracle$ cluvfy stage -post hwos -n node1
node1:oracle$ cluvfy stage -post crsinst -n node1,node2 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "node1"
Destination Node Reachable?
node1 yes
node2 yes
Result: Node reachability check passed from node "node1".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
node2 passed
node1 passed
Result: User equivalence check passed for user "oracle".
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
node2 not running
node1 not running
Result: Daemon status check failed for "CSS daemon".
Cluster manager integrity check failed.
Checking cluster integrity...
Cluster is divided into 2 partitions.
Partition 1 consists of the following members:
Node Name
node1
Partition 2 consists of the following members:
Node Name
node1
node2
Cluster integrity check failed. Cluster is divided into 2 partition(s).
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
WARNING:
CSS is probably working with a non-clustered, local-only configuration on nodes:
node1
Verification will proceed with nodes:
node2
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
Data integrity check for OCR passed.
OCR integrity check failed.
Checking CRS integrity...
Checking daemon liveness...
Check: Liveness for "CRS daemon"
Node Name Running
node2 no
node1 yes
Result: Liveness check failed for "CRS daemon".
Checking daemon liveness...
Check: Liveness for "CSS daemon"
Node Name Running
node2 no
node1 no
Result: Liveness check failed for "CSS daemon".
Checking daemon liveness...
Check: Liveness for "EVM daemon"
Node Name Running
node2 no
node1 yes
Result: Liveness check failed for "EVM daemon".
Liveness of all the daemons
Node Name CRS daemon CSS daemon EVM daemon
node2 no no no
node1 yes no yes
Checking CRS health...
Check: Health of CRS
Node Name CRS OK?
node1 unknown
Result: CRS health check failed.
CRS integrity check failed.
Post-check for cluster services setup was unsuccessful on all the nodes.
Please help.Thanks for the answer.
Problem is that the RAC is not healty, I know how to start it but now it doesn't start and I cannot find helpful logs.
You can see that cluster is not healty from cluvfy output.
When I tried to start crs whith crsctl start crs but:
- on node2 none of the demons starts (CSS, CRS or EVM)
- on node1 CSS and EVM demons start but CRS is not starting and I cannot find helpful logs
Problem 2: Cannot change OCR location
node2:root$ cat /var/opt/oracle/ocr.loc
+#Device/file /dev/rdsk/c7t30d0s3 getting replaced by device /dev/rdsk/c7t30d0s3+
ocrconfig_loc=/dev/rdsk/c7t30d0s3
local_only=falsenode2:root$ ocrconfig -replace ocr /dev/rdsk/c0t600A0B8000347EAD00000F314A5AF5B8d0s3
PROT-1: Failed to initialize ocrconfig
node1:root$ cat /var/opt/oracle/ocr.loc
ocrconfig_loc=/u01/oracle/product/10gr2/db/cdata/localhost/local.ocr
local_only=TRUE
node1:root$ ocrconfig -replace ocr /dev/rdsk/c0t600A0B8000347EAD00000F314A5AF5B8d0s3
PROT-1: Failed to initialize ocrconfig
Problem 3: On node 1 I wrongly issued a command
node1:root$ /u01/oracle/product/10gr2/db/bin/localconfig reset
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started -
CRS could not start after reboot
I installed oracle 10g R2 RAC on RHEL4 U4 with 2 nodes. It runs fine on 2 nodes (without reboot after installation). But when I rebooted one node, I found that the CRS daemon could not start automatically. The boot menu show "Starting CRS" [OK]. The "init.crs start" command won't init the CRS daemon.
$ ./crs_stat
CRS-0184: Cannot communicate with the CRS daemon.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA1/RAC/spfileRAC.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA1/RAC/spfileRAC.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Manager
$ /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
$ cat /etc/inittab
id:5:initdefault:
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
$ ls /etc/rc5.d
K01tog-pegasus K35smb K85mdmpd S05openibd S18rpcidmapd S55sshd S96readahead
K02NetworkManager K35vncserver K87auditd S06cpuspeed S19rpcgssd S56rawdevices S97messagebus
K05saslauthd K35winbind K87ipmi S08arptables_jf S19vmware-tools S56xinetd S97rhnsd
K10dc_server K40smartd K89netplugd S08iptables S25netfs S58ntpd S98cups-config-daemon
K10psacct K50ibmasm K90bluetooth S09isdn S26apmd S80sendmail S98haldaemon
K12dc_client K50netdump K94diskdump S10network S26lm_sensors S85gpm S99local
K15httpd K50snmpd K96pcmcia S12syslog S27vasd S90crond
K20nfs K50snmptrapd S00microcode_ctl S13irqbalance S28autofs S90xfs
K24irda K50tux S01sysstat S13portmap S29oracleasm S95anacron
K25squid K73ypbind S04readahead_early S14nfslock S44acpid S95atd
K30spamassassin K74nscd S05kudzu S15mdmonitor S55cups S96init.crs
Do I miss any configuration to initialize the CRSD at boot?
thanks, ezleedid you run cluster verification utility for node connectivity?
http://docs.oracle.com/cd/E11882_01/rac.112/e16794/cvu.htm#autoId19
also check the logs ad $GRID_HOME/log/<hostname>/...
starting with alert.log, then the ones its referencing -
CSSD does not start (Cluster Manager)
Hi, All,
I had installed Oracle 10.2.0 and I'm gone use ASM. But when I'm starting de CSSD nothing happens. No process are submited.
The description :
# ./localconfig reset $ORACLE_HOME
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Adding to inittab
Startup will be queued to init within 90 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started
Can someone help me?
**The host name are not in capital letterHI thanks for the reply,
Following is the log generated:
/u01/app/oracle/product/10.2.0/db_1/log/ldi/cssd>more ocssd.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
[ CSSD]2007-06-18 15:09:10.371 >USER: Oracle Database 10g CSS Release 10.2.0.1.0 Production Copyright 1996, 2004 Oracle.
All rights reserved.
[ clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ldiDBG_CSSD))
[ CSSD]2007-06-18 15:09:10.371 >USER: CSS daemon log for node ldi, number 0, in cluster localhost
[ CSSD]2007-06-18 15:09:10.418 [1] >TRACE: clssscmain: local-only set to true
[ CSSD]2007-06-18 15:09:10.438 [1] >TRACE: clssnmReadNodeInfo: added node 0 (ldi) to cluster
[ CSSD]2007-06-18 15:09:10.500 [5] >TRACE: clssnm_skgxnmon: skgxn init failed, rc 1
[ CSSD]2007-06-18 15:09:10.501 [1] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
[ CSSD]2007-06-18 15:09:10.502 [1] >TRACE: clssnmInitNMInfo: misscount set to 30
[ CSSD]2007-06-18 15:09:10.508 [1] >TRACE: clssnmFatalInit: recognized localonly mode
[ CSSD]2007-06-18 15:09:10.523 [6] >TRACE: clssnmconnect: connecting to node 0, flags 0x0001, connector 1
[ CSSD]2007-06-18 15:09:10.557 [7] >TRACE: clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLs
tnr_localhost_0))
[ CSSD]2007-06-18 15:09:10.558 [7] >ERROR: clssgmclientlsnr: listening failed for (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_
LclLstnr_localhost_0)) (3)
[ CSSD]2007-06-18 15:09:10.558 [7] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr
localhost0))
[ CSSD]2007-06-18 15:09:10.558 [7] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ldi_localh
ost))
[ CSSD]2007-06-18 17:03:00.680 >USER: Oracle Database 10g CSS Release 10.2.0.1.0 Production Copyright 1996, 2004 Oracle.
All rights reserved.
[ CSSD]2007-06-18 17:03:00.680 >USER: CSS daemon log for node ldi, number 0, in cluster localhost
[ CSSD]2007-06-18 17:03:00.691 [1] >TRACE: clssscmain: local-only set to true
[ clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ldiDBG_CSSD))
[ CSSD]2007-06-18 17:03:00.701 [1] >TRACE: clssnmReadNodeInfo: added node 0 (ldi) to cluster
[ CSSD]2007-06-18 17:03:00.728 [5] >TRACE: clssnm_skgxnmon: skgxn init failed, rc 1
[ CSSD]2007-06-18 17:03:00.728 [1] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
[ CSSD]2007-06-18 17:03:00.729 [1] >TRACE: clssnmInitNMInfo: misscount set to 30
[ CSSD]2007-06-18 17:03:00.729 [1] >TRACE: clssnmFatalInit: recognized localonly mode
[ CSSD]2007-06-18 17:03:00.731 [6] >TRACE: clssnmconnect: connecting to node 0, flags 0x0001, connector 1
[ CSSD]2007-06-18 17:03:00.774 [7] >TRACE: clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLs
tnr_localhost_0))
[ CSSD]2007-06-18 17:03:00.774 [7] >ERROR: clssgmclientlsnr: listening failed for (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_
LclLstnr_localhost_0)) (3)
[ CSSD]2007-06-18 17:03:00.775 [7] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr
localhost0))
[ CSSD]2007-06-18 17:03:00.775 [7] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ldi_localh
ost))
Do we need to configure listener or any thing else also??? -
Database not starting,REDO log file error?
My DB was working fine uptill last evening, when i restarted it for some DB task it showed me an error ..
ORA-00333: redo log read error block 57346 count 8192
I studied the alert log file & related trace file .... but could not conclude what to do ?? Can any one suggest me what could be the solution...
ORACLE 10g R2
Windows 2003 Server
Database in NOARCHIVELOG mode.
I don't have the recent backup.The one i have is 20 days old.
Alert Log file contents : -
Wed Jun 02 20:22:19 2010
Aborting crash recovery due to error 333
Wed Jun 02 20:22:19 2010
Errors in file e:\oracle\product\10.2.0\admin\ora10g\udump\ora10g_ora_140.trc:
ORA-00333: redo log read error block 57346 count 8192
ORA-333 signalled during: ALTER DATABASE OPEN...
Wed Jun 02 20:37:15 2010
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Wed Jun 02 20:43:53 2010
Shutting down instance: further logons disabled
Wed Jun 02 20:43:53 2010
Stopping background process CJQ0
Wed Jun 02 20:43:54 2010
Stopping background process MMNL
Wed Jun 02 20:43:55 2010
Stopping background process MMON
Wed Jun 02 20:43:56 2010
Shutting down instance (immediate)
License high water mark = 3
Wed Jun 02 20:43:56 2010
Stopping Job queue slave processes
Wed Jun 02 20:43:56 2010
Job queue slave processes stopped
Waiting for dispatcher 'D000' to shutdown
All dispatchers and shared servers shutdown
Wed Jun 02 20:43:58 2010
ALTER DATABASE CLOSE NORMAL
ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
Wed Jun 02 20:43:58 2010
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Trace file to read ora10g_ora_140.trc:-
Dump file e:\oracle\product\10.2.0\admin\ora10g\udump\ora10g_ora_140.trc
Wed Jun 02 20:22:18 2010
ORACLE V10.2.0.1.0 - Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
Windows Server 2003 Version V5.2 Service Pack 1
CPU : 2 - type 586, 1 Physical Cores
Process Affinity : 0x00000000
Memory (Avail/Total): Ph:3527M/4094M, Ph+PgF:4386M/5973M, VA:800M/2047M
Instance name: ora10g
Redo thread mounted by this instance: 1
Oracle process number: 15
Windows thread id: 140, image: ORACLE.EXE (SHAD)
*** SERVICE NAME:() 2010-06-02 20:22:18.406
*** SESSION ID:(159.3) 2010-06-02 20:22:18.406
Successfully allocated 2 recovery slaves
Using 545 overflow buffers per recovery slave
Thread 1 checkpoint: logseq 14261, block 2, scn 86974094
cache-low rba: logseq 14260, block 4023
on-disk rba: logseq 14262, block 3758, scn 86978396
start recovery at logseq 14261, block 2, scn 86974094
ORA-00333: redo log read error block 57346 count 8192
ORA-00312: online log 2 thread 1: 'E:\ORACLE\PRODUCT\10.2.0\ORADATA\ORA10G\REDO02.LOG'
ORA-27070: async read/write failed
OSD-04016: Error queuing an asynchronous I/O request.
O/S-Error: (OS 2) The system cannot find the file specified.
ORA-00333: redo log read error block 57346 count 8192
ORA-00312: online log 2 thread 1: 'E:\ORACLE\PRODUCT\10.2.0\ORADATA\ORA10G\REDO02.LOG'
ORA-27091: unable to queue I/O
ORA-27070: async read/write failed
OSD-04006: ReadFile() failure, unable to read from file
O/S-Error: (OS 2) The system cannot find the file specified.
ORA-00333: redo log read error block 57346 count 8192
*** 2010-06-02 20:43:58.812
*** 2010-06-02 20:43:58.812 60680 kcrr.c
ARCH: Archival disabled due to shutdown: 1089
*** 2010-06-02 20:43:59.812 60680 kcrr.c
ARCH: Archival disabled due to shutdown: 1089Hi,Aman this(following) may is not good way for opening database,but OP can try and open own database:
@ DBA4 you will see from following example how you can open your database.In your online log files was corrupted,therefore you have to do following step by step then you database will open.
SQL> alter database clear unarchived logfile group 2;
alter database clear unarchived logfile group 2
ERROR at line 1:
ORA-01624: log 2 needed for crash recovery of instance p (thread 1)
ORA-00312: online log 2 thread 1:
'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\REDO02.LOG'
SQL> alter database backup controlfile to trace as 'D:\oracle\product\10.2.0\ora
data\P\c.txt';
Database altered.
SQL> shutdown abort;
ORACLE instance shut down.
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 125829120 bytes
Fixed Size 1288100 bytes
Variable Size 75499612 bytes
Database Buffers 41943040 bytes
Redo Buffers 7098368 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "P" RESETLOGS ARCHIVELOG
2 MAXLOGFILES 16
3 MAXLOGMEMBERS 3
4 MAXDATAFILES 100
5 MAXINSTANCES 8
6 MAXLOGHISTORY 292
7 LOGFILE
8 GROUP 1 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\REDO01.LOG' SIZE 50M,
9 GROUP 2 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\REDO02.LOG' SIZE 50M,
10 GROUP 3 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\REDO03.LOG' SIZE 50M
11 DATAFILE
12 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\SYSTEM01.DBF',
13 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\UNDOTBS01.DBF',
14 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\SYSAUX01.DBF',
15 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\USERS01.DBF',
16 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\TS_TEST.ORA',
17 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\TEST.ORA',
18 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\TEST1.ORA'
19 CHARACTER SET WE8MSWIN1252;
Control file created.
SQL> alter database open resetlogs;
alter database open resetlogs
ERROR at line 1:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\SYSTEM01.DBF'
SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2032832 generated at 06/03/2010 12:42:29 needed for thread 1
ORA-00289: suggestion :
D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\P\ARCHIVELOG\2010_06_03\O1_MF_1_87_
%U_.ARC
ORA-00280: change 2032832 for thread 1 is in sequence #87
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\SYSTEM01.DBF'
ORA-01112: media recovery not started
then add *_allow_resetlogs_corruption=true* in pfile then startup database with pfile as
SQL> startup mount pfile=D:\oracle\product\10.2.0\admin\P\pfile\init.ora.pr
ORACLE instance started.
Total System Global Area 125829120 bytes
Fixed Size 1288100 bytes
Variable Size 62916700 bytes
Database Buffers 54525952 bytes
Redo Buffers 7098368 bytes
Database mounted.
SQL> recover;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done
SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2032832 generated at 06/03/2010 12:42:29 needed for thread 1
ORA-00289: suggestion :
D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\P\ARCHIVELOG\2010_06_03\O1_MF_1_87_
%U_.ARC
ORA-00280: change 2032832 for thread 1 is in sequence #87
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\P\SYSTEM01.DBF'
ORA-01112: media recovery not started
SQL> alter database open;
alter database open
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
SQL> alter database open resetlogs;
Database altered.
SQL>finally remove allowresetlogs_corruption=true parameter from pfile. -
I have 2 node RAC instance in 2 virtual machines. My node 2 ASM instance is not starting after the reboot but node 1 is working fine.
I got below output from Node 2.
srvctl status asm
PRCR-1070 : Failed to check if resource ora.asm is registered
Cannot communicate with crsd
alertlog
[/u01/app/11.2.0/grid/bin/cssdagent(3782)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/11.2.0/grid/log/ol5-112-rac2/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2015-03-13 16:38:23.671
[ohasd(3315)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/11.2.0/grid/log/ol5-112-rac2/ohasd/ohasd.log.
2015-03-13 16:38:24.264
[ohasd(3315)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:35.915
[cssd(4326)]CRS-1713:CSSD daemon is started in clustered mode
2015-03-13 16:38:41.195
[cssd(4326)]CRS-1707:Lease acquisition for node ol5-112-rac2 number 2 completed
2015-03-13 16:38:41.259
[cssd(4326)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DISK1; details in /u01/app/11.2.0/grid/log/ol5-112-rac2/cssd/ocssd.log.
2015-03-13 16:38:41.562
[crsd(4285)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/ol5-112-rac2/crsd/crsd.log.
2015-03-13 16:38:42.217
[ohasd(3315)]CRS-2765:Resource 'ora.crsd' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:43.313
[crsd(4357)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/ol5-112-rac2/crsd/crsd.log.
2015-03-13 16:38:44.262
[ohasd(3315)]CRS-2765:Resource 'ora.crsd' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:44.716
[ohasd(3315)]CRS-2765:Resource 'ora.diskmon' has failed on server 'ol5-112-rac2'.
crsd log
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmo: kgfoCheckMount returned [7]
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmo: The ASM instance is down
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprioo: No OCR/OLR devices are usable
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmcl: asmhandle is NULL
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprinit: Could not open raw device
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmcl: asmhandle is NULL
2015-03-13 16:39:02.140: [ OCRAPI][1266976496]a_init:16!: Backend init unsuccessful : [26]
2015-03-13 16:39:02.140: [ CRSOCR][1266976496] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2015-03-13 16:39:02.140: [ CRSD][1266976496][PANIC] CRSD exiting: Could not init OCR, code: 26
2015-03-13 16:39:02.140: [ CRSD][1266976496] Done
Pls helpHi Levi,
I got below output from cssd.log
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos op : sgipcnTcpConnect
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos dep : No route to host (113)
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos loc : connect
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos info: addr '192.168.1.101:42729'
2015-03-15 10:38:48.248: [ CSSD][1214789952]clssscSelect: conn complete ctx 0x24d7aa0 endp 0x10d4
2015-03-15 10:38:48.248: [ CSSD][1214789952]clssnmeventhndlr: node(1), endp(0x10d4) failed, probe((nil)) ninf->endp (0x1000010d4) CONNCOMPLETE
2015-03-15 10:38:48.248: [ CSSD][1214789952]clssnmDiscHelper: ol5-112-rac1, node(1) connection failed, endp (0x10d4), probe(0x100000000), ninf->endp 0x10d4
2015-03-15 10:38:48.248: [ CSSD][1214789952]clssnmDiscHelper: node 1 clean up, endp (0x10d4), init state 0, cur state 0
2015-03-15 10:38:48.248: [GIPCXCPT][1214789952]gipcInternalDissociate: obj 0x27c8050 [00000000000010d4] { gipcEndpoint : localAddr 'gipc://ol5-112-rac2:de93-de83-5e0c-a373#192.168.1.102#50439', remoteAddr 'gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x8061a, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2015-03-15 10:38:48.248: [GIPCXCPT][1214789952]gipcDissociateF [clssnmDiscHelper : clssnm.c : 3215]: EXCEPTION[ ret gipcretFail (1) ] failed to dissociate obj 0x27c8050 [00000000000010d4] { gipcEndpoint : localAddr 'gipc://ol5-112-rac2:de93-de83-5e0c-a373#192.168.1.102#50439', remoteAddr 'gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x8061a, usrFlags 0x0 }, flags 0x0
2015-03-15 10:38:48.248: [ CSSD][1214789952]clssnmDiscEndp: gipcDestroy 0x10d4
2015-03-15 10:38:48.414: [ CSSD][1147648320]clssnmvDHBValidateNCopy: node 1, ol5-112-rac1, has a disk HB, but no network HB, DHB has rcfg 320087493, wrtcnt, 207342, LATS 355744, lastSeqNo 207342, uniqueness 1426244111, timestamp 1426396128/31164744
2015-03-15 10:38:48.414: [ CSSD][1214789952]clssnmconnect: connecting to addr gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729
2015-03-15 10:38:48.414: [ CSSD][1214789952]clssscConnect: endp 0x10e0 - cookie 0x24d7aa0 - addr gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729
2015-03-15 10:38:48.414: [ CSSD][1214789952]clssnmconnect: connecting to node(1), endp(0x10e0), flags 0x10002
2015-03-15 10:38:48.710: [ CSSD][1181219136]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2015-03-15 10:38:49.209: [ CSSD][1206397248]clssnmRcfgMgrThread: Local Join
2015-03-15 10:38:49.209: [ CSSD][1206397248]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2015-03-15 10:38:49.209: [ CSSD][1206397248]clssnmLocalJoinEvent: set curtime (356544) for my node
2015-03-15 10:38:49.209: [ CSSD][1206397248]clssnmLocalJoinEvent: scanning 32 nodes
2015-03-15 10:38:49.209: [ CSSD][1206397248]clssnmLocalJoinEvent: Node ol5-112-rac1, number 1, is in an existing cluster with disk state 3
2015-03-15 10:38:49.210: [ CSSD][1206397248]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
Thanks,
Suranga -
11gR2 Cluster node does not start, start of ora.gipcd fails
Hi
I have a clusert node which just does not start, and i do know understand what the problem is.
Does anyone have an idea what the problem cound be?
I am using Oracle 11.2.0.1.0 on OEL.
I appreciate any help, thank you.
Regards Bernhard
NAME TARGET STATE SERVER STATE_DETAILS
Cluster Resources
ora.asm
1 ONLINE OFFLINE
ora.crsd
1 ONLINE INTERMEDIATE node001
ora.cssd
1 ONLINE OFFLINE
ora.cssdmonitor
1 ONLINE OFFLINE
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 ONLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE node001
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE OFFLINE
ora.gpnpd
1 ONLINE OFFLINE
ora.mdnsd
1 ONLINE ONLINE node001 alertnode001.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:43:52.511
[/opt/grid/product/11gR2/grid/bin/cssdmonitor(5144)]CRS-5822:Agent '/opt/grid/product/11gR2/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-06-29 08:48:49.713
[ohasd(4346)]CRS-2112:The OLR service started on node node001.
2012-06-29 08:48:50.010
[ohasd(4346)]CRS-8017:location: /etc/oracle/lastgasp has 50 reboot advisory log files, 0 were announced and 0 errors occurred
2012-06-29 08:48:55.134
[ohasd(4346)]CRS-2772:Server 'node001' has been assigned to pool 'Free'.
[client(4634)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4652)]CRS-10001:ACFS-9322: done.
2012-06-29 08:50:59.362
[ohasd(4346)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.oracssdmonitor_root.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:48:51.003: [ AGFW][1810312288] Starting the agent: /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/
2012-06-29 08:48:51.003: [ AGENT][1810312288] Agent framework initialized, Process Id = 4414
2012-06-29 08:48:51.003: [ USRTHRD][1810312288] to enter agent main
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: setting priority to 4
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] *** Agent Framework Started ***
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_vmondisv: Compatible vendor clusterware not in use
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_thrdspawn: spawn OMON succ
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: spawn omon succ
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: default values for omon, initrate 1000, pollrate 500.
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: starting omon
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:55.041: [ AGFW][1810312288] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Inited lsf context 0x1ae1cdd0
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Initing CLS Framework messaging
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] New Framework state: 2
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] M2M is starting...
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] m_pClscCtx=0x1ae21d70m_pUgblm=0x1ae754b0
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] Starting send thread
2012-06-29 08:48:55.052: [ CRSCOMM][1119893824] clsIpc: sendWork thread started.
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] IPC Client thread started listening
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] init data sent from server
2012-06-29 08:48:55.057: [CLSFRAME][1810312288] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:node001
2012-06-29 08:48:55.058: [CLSFRAME][1810312288] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:node001
2012-06-29 08:48:55.066: [CLSFRAME][1810312288] Starting thread model named: MultiThread
2012-06-29 08:48:55.073: [CLSFRAME][1810312288] Starting thread model named: SingleThread
2012-06-29 08:48:55.076: [CLSFRAME][1810312288] Starting thread model named: SingleThreadT
2012-06-29 08:48:55.080: [CLSFRAME][1810312288] New Framework state: 3
2012-06-29 08:48:55.080: [ AGFW][1810312288] Agent Framework started successfully
2012-06-29 08:48:55.080: [ AGFW][1182832960] Agfw engine module has enabled...
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] Module Enabling is complete
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] New Framework state: 6
2012-06-29 08:48:55.080: [ AGFW][1182832960] Agent is started with userid: root , expected user: root
2012-06-29 08:48:55.081: [ AGFW][1182832960] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-06-29 08:48:55.088: [ AGFW][1182832960] Agent received the message: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.089: [ AGFW][1182832960] Added new restype: ora.cssdmonitor.type
2012-06-29 08:48:55.089: [ AGFW][1182832960] Agent sending last reply for: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.090: [ AGFW][1182832960] Agent received the message: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.090: [ AGFW][1182832960] Added new resource: ora.cssdmonitor 1 1 to the agfw
2012-06-29 08:48:55.091: [ AGFW][1182832960] Agent sending last reply for: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.091: [ AGFW][1182832960] Agent received the message: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.091: [ AGFW][1182832960] Preparing CHECK command for: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [ AGFW][1172343104] Executing command: check for resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_INITRATE, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_POLLRATE, value 500
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr ORA_OPROCD_MODE, value
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr PROCD_TIMEOUT, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr LOGGING_LEVEL, value 1
2012-06-29 08:48:55.096: [ AGFW][1172343104] check for resource: ora.cssdmonitor 1 1 completed with status: OFFLINE
2012-06-29 08:48:55.096: [CRSTIMER][1203812672] Timer Thread Starting.
2012-06-29 08:48:55.096: [ AGFW][1182832960] ora.cssdmonitor 1 1 state changed from: UNKNOWN to: OFFLINE
2012-06-29 08:48:55.096: [ AGFW][1182832960] Agent sending last reply for: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.096: [ AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.096: [ AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:28
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent received the message: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent sending last reply for: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:34
2012-06-29 08:48:55.101: [ AGFW][1182832960] ora.cssdmonitor 1 1 marked as deleted.
2012-06-29 08:48:55.101: [ AGFW][1182832960] Deleting the resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.101: [ AGFW][1182832960] Agent is commiting suicide.
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: CSSD Agent was asked to exit with exit code 1
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_becomeactive: Signaling active state
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_term: still threads up (1)
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: Agent is done.
2012-06-29 08:48:55.101: [ AGFW][1182832960] Agent is exiting with exit code: 1
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29Edited by: Bernhard W on 03-Jul-2012 02:14I am really getting desperate about this.
The ASM persmissions are on both nodes the same, so this should not be the problem and i can not find any problem on the network.
The log files of the healthy node do not say anything about the node which does not startup now, the last log entry was from 6 months ago, when i shut that node down. There seems to be no communication between the two nodes now.
I tried to manually start the ora.cssd but somehow ora.gipcd just does not start and i can not find the problem.
[oracle@dmrs11srvcla001 ~]$ crsctl stat res -t -init
NAME TARGET STATE SERVER STATE_DETAILS
Cluster Resources
ora.asm
1 ONLINE OFFLINE
ora.crsd
1 ONLINE INTERMEDIATE dmrs11srvcla001
ora.cssd
1 ONLINE OFFLINE
ora.cssdmonitor
1 ONLINE ONLINE dmrs11srvcla001
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE dmrs11srvcla001
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE OFFLINE
ora.gpnpd
1 ONLINE OFFLINE
ora.mdnsd
1 ONLINE ONLINE dmrs11srvcla001
[oracle@dmrs11srvcla001 ~]$ crsctl start res ora.cssd -init
CRS-2672: Attempting to start 'ora.gipcd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gipcd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gipcd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gipcd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gpnpd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gpnpd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dmrs11srvcla001'
CRS-2672: Attempting to start 'ora.diskmon' on 'dmrs11srvcla001'
CRS-2676: Start of 'ora.diskmon' on 'dmrs11srvcla001' succeededalertnode001.log
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:40.789
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:41.654
[ohasd(4351)]CRS-2765:Resource 'ora.mdnsd' has failed on server 'node001'.
2012-07-02 17:11:43.404
[cssd(4997)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:11:46.634
[cssd(4997)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:11:46.745
[ohasd(4351)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'node001'.
2012-07-02 17:12:42.384
[/opt/grid/product/11gR2/grid/bin/orarootagent.bin(4488)]CRS-5818:Aborted command 'start for resource: ora.diskmon 1 1' for resource 'ora.diskmon'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-07-02 17:12:46.390
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.diskmon'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:14:50.768
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:55.819
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:58.354
[cssd(5137)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:16:58.425
[cssd(5137)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:25:23.194
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-02 17:25:23.200
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-02 17:26:57.384
[/opt/grid/product/11gR2/grid/bin/cssdagent(5116)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-02 17:27:02.514
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:37:11.971
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.evmd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 02:13:57.821
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:14:37.915
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:44:44.630
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:14:51.303
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:44:57.981
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:15:04.655
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:42:45.015
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:50.068
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:52.369
[cssd(10209)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-03 07:44:52.434
[cssd(10209)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-03 07:45:11.331
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-03 07:54:51.401
[/opt/grid/product/11gR2/grid/bin/cssdagent(10188)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-03 07:54:56.524
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.ohasd.log
2012-07-03 07:45:11.330: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [ AGFW][1347250496] Received state change for ora.crsd 1 1 [old state = PARTIAL, new state = OFFLINE]
2012-07-03 07:45:11.330: [ AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23258]
2012-07-03 07:45:11.330: [ AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [ CRSPE][1357756736] State change received from node001 for ora.crsd 1 1
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Processing PE command id=139. Description: [Resource State Change (ora.crsd 1 1) : 0x1c284d40]
2012-07-03 07:45:11.331: [ CRSPE][1357756736] RI [ora.crsd 1 1] new external state [OFFLINE] old value: [INTERMEDIATE] on node001 label = []
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Resource Resource Instance ID[ora.crsd 1 1]. Values:
STATE=OFFLINE
TARGET=ONLINE
LAST_SERVER=node001
CURRENT_RCOUNT=1
LAST_RESTART=1341292505
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=29
STATE_CHANGE_VERS=29
LAST_FAULT=1341292504
DEGREE_ID=1
ID=ora.crsd 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:none
ReadLocks:|STATE INITED||INITIAL CHECK DONE| has failed!
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Processing unplanned state change for [ora.crsd 1 1]
2012-07-03 07:45:11.331: [ CRSPE][1357756736] ora.crsd 1 1: uptime exceeds uptime threshold , resetting restart count
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Local Recovery picked for [ora.crsd 1 1]
2012-07-03 07:45:11.333: [ CRSPE][1357756736] Sending message to agfw: id = 23260
2012-07-03 07:45:11.333: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:11.333: [ CRSPE][1357756736] CRS-2672: Attempting to start 'ora.crsd' on 'node001'
2012-07-03 07:45:11.333: [ AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260 to the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.349: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [ AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.350: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [ AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.351: [ CRSPE][1357756736] RI [ora.crsd 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on node001 label = []
2012-07-03 07:45:12.351: [ CRSPE][1357756736] CRS-2676: Start of 'ora.crsd' on 'node001' succeeded
2012-07-03 07:45:12.351: [ CRSPE][1357756736] PE Command [ Resource State Change (ora.crsd 1 1) : 0x1c284d40 ] has completed
2012-07-03 07:45:12.351: [ AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [ AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [ AGFW][1347250496] Agfw received reply from PE for resource state change for ora.crsd 1 1
2012-07-03 07:53:17.220: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [ AGFW][1347250496] Received state change for ora.diskmon 1 1 [old state = ONLINE, new state = PLANNED_OFFLINE]
2012-07-03 07:53:17.221: [ AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23497]
2012-07-03 07:53:17.221: [ AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [ CRSPE][1357756736] State change received from node001 for ora.diskmon 1 1
2012-07-03 07:53:17.221: [ CRSPE][1357756736] Processing PE command id=140. Description: [Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0]
2012-07-03 07:53:17.221: [ CRSPE][1357756736] RI [ora.diskmon 1 1] new external state [OFFLINE] old value: [ONLINE] on node001 label = []
2012-07-03 07:53:17.221: [ CRSPE][1357756736] RI [ora.diskmon 1 1] new target state: [OFFLINE] old value: [ONLINE]
2012-07-03 07:53:17.222: [ CRSOCR][1351452992] Multi Write Batch processing...
2012-07-03 07:53:17.222: [ CRSPE][1357756736] Resource Resource Instance ID[ora.diskmon 1 1]. Values:
STATE=OFFLINE
TARGET=OFFLINE
LAST_SERVER=node001
CURRENT_RCOUNT=0
LAST_RESTART=1341294297
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=2
STATE_CHANGE_VERS=5
LAST_FAULT=1341242723
DEGREE_ID=1
ID=ora.diskmon 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:START of [ora.cssd 1 1] on [node001] : 0x2aaab000ba10
ReadLocks:|STATE INITED| has failed!
2012-07-03 07:53:17.222: [ CRSPE][1357756736] Processing unplanned state change for [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [ CRSPE][1357756736] Target is not ONLINE, not recovering [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [ CRSPE][1357756736] PE Command [ Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0 ] has completed
2012-07-03 07:53:17.222: [ AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.222: [ AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.223: [ AGFW][1347250496] Agfw received reply from PE for resource state change for ora.diskmon 1 1
2012-07-03 07:53:17.223: [ CRSOCR][1351452992] Multi Write Batch done.
2012-07-03 07:54:56.523: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.523: [ AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [ CRSPE][1357756736]Created alert : (:CRSPE00111:) : Start action timed out!
2012-07-03 07:54:56.524: [ CRSPE][1357756736] Start action failed with error code: 3
2012-07-03 07:54:56.524: [ AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [ AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [ AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [ AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [ CRSPE][1357756736] CRS-2674: Start of 'ora.cssd' on 'node001' failed
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2674: Start of 'ora.cssd' on 'node001' failed]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
2012-07-03 07:54:56.525: [ CRSPE][1357756736] Sending message to agfw: id = 23555
2012-07-03 07:54:56.525: [ CRSPE][1357756736] CRS-2679: Attempting to clean 'ora.cssd' on 'node001'
2012-07-03 07:54:56.525: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:54:56.525: [ AGFW][1347250496] Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is being stopped. Will be restarted when agent stops completly
2012-07-03 07:54:56.525: [ AGFW][1347250496] Could not forward message [RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555] to agent. /opt/grid/product/11gR2/grid/bin/cssdagent_root is not running
2012-07-03 07:54:56.525: [ AGFW][1347250496] Can not start Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is waiting to be restarted.
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2679: Attempting to clean 'ora.cssd' on 'node001']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 14
2012-07-03 07:55:01.628: [ CRSCOMM][1340946752] IPC listener connection to member 14 has been removed
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [ CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:01.628: [ AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:01.628: [ AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:01.628: [ AGFW][1347250496] Restarting the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.628: [ AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10188] stopped!
2012-07-03 07:55:01.628: [ CRSCOMM][1347250496] removeConnection: Member 14 does not exist.
2012-07-03 07:55:01.628: [ AGFW][1347250496] Starting the agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id: root and incarnation:7
2012-07-03 07:55:01.632: [ AGFW][1347250496] Starting the HB [Interval = 30000, misscount = 10kill allowed=0] for agent: /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New IPC Member:{Relative|Node:0|Process:15|Type:3}:AGENT
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New process connected to us ID:{Relative|Node:0|Process:15|Type:3} Info:AGENT
2012-07-03 07:55:01.837: [ AGFW][1347250496] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.837: [ AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root with pid:10530 connected to server.
2012-07-03 07:55:01.837: [ AGFW][1347250496] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [ AGFW][1347250496] Agfw Proxy Server sending message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [ AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.839: [ AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.851: [ AGFW][1347250496] Received the reply to the message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.854: [ AGFW][1347250496] Received the reply to the message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.861: [ AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [ AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.862: [ AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [ CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.862: [ AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.863: [ AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [ CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.863: [ AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.863: [ AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [ CRSPE][1357756736] CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded
2012-07-03 07:55:01.863: [ CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
2012-07-03 07:55:01.863: [UiServer][1361959232] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
2012-07-03 07:55:01.864: [ CRSPE][1357756736] PE Command [ Start Resource : 0x2aaab0009d00 ] has completed
2012-07-03 07:55:01.864: [ AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [ CRSPE][1357756736] UI Command [Start Resource : 0x2aaab0009d00] is replying to sender.
2012-07-03 07:55:01.864: [ AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [UiServer][1361959232] Container [ Name: UI_DATA
ora.cssd:
TextMessage[215]
2012-07-03 07:55:01.864: [UiServer][1361959232] Done for ctx=0x2aaab004ece0
2012-07-03 07:55:01.869: [UiServer][1364060480] Closed: remote end failed/disc.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 15
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] IPC listener connection to member 15 has been removed
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.792: [ CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:03.792: [ AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:03.792: [ AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:03.792: [ AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10530] stopped!
2012-07-03 07:55:03.792: [ CRSCOMM][1347250496] removeConnection: Member 15 does not exist. -
RFS Process is not starting up
Hi,
DB_VERSION=10.2.0.4
OS_VERSION=Windows 2003 Server
I have a cascaded setup at our site and due to poor bandwidth , my Far DR is always lag in archive sync,However my Near DR remains always in Sync with my Primary.
Today i applied the incremental backup on Far DR ,and put it in the recovery mode but the RFS process is not starting up.
SQL> select message from v$dataguard_status;
MESSAGE
ARC0: Archival started
ARC1: Archival started
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC1: Becoming the heartbeat ARCH
Attempt to start background Managed Standby Recovery process
MRP0: Background Managed Standby Recovery process started
Managed Standby Recovery not using Real Time Apply
Media Recovery Waiting for thread 1 sequence 211777
Fetching gap sequence in thread 1, gap sequence 211777-211842
FAL[client]: Failed to request gap sequence
MESSAGE
GAP - thread 1 sequence 211777-211842
DBID 2578507424 branch 709654561
FAL[client]: All defined FAL servers have been attempted.
14 rows selected.
SQL> select process,status from v$managed_standby;
PROCESS STATUS
ARCH CONNECTED
ARCH CONNECTED
MRP0 WAIT_FOR_GAP
Alertlog_output:-
Completed: alter database recover managed standby database disconnect from session
Wed Oct 06 16:31:14 2010
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 211777-211842
DBID 2578507424 branch 709654561
FAL[client]: All defined FAL servers have been attempted.
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that is sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
Wed Oct 06 16:40:42 2010
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
mrpo_trace_output :-
Client logon and security negotiation successful!
*** 2010-10-06 16:28:10.486
Redo shipping client performing standby login
*** 2010-10-06 16:28:10.690 66535 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-10-06 16:28:40.972
Redo shipping client performing standby login
*** 2010-10-06 16:28:41.160 66535 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-10-06 16:29:11.427
Redo shipping client performing standby login
*** 2010-10-06 16:29:11.615 66535 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-10-06 16:29:41.929
Redo shipping client performing standby login
*** 2010-10-06 16:29:42.116 66535 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-10-06 16:30:12.415
Redo shipping client performing standby login
*** 2010-10-06 16:30:12.602 66535 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-10-06 16:30:43.557
Redo shipping client performing standby login
*** 2010-10-06 16:30:43.744 66535 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-10-06 16:31:14.011
Edited by: Girish on Oct 6, 2010 4:32 AM
Edited by: Girish on Oct 6, 2010 4:34 AM
Edited by: Girish on Oct 6, 2010 5:42 AMCheck that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that is sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps. -
Rdbms do not start with dependency after upgrade 11Gr2 under window 2003
Hi,
i finish to upgrade my 2 nodes cluster. I reboot the server just to be shure all the good dll are use.
At the reboot i check the window services. My OracleServiceTest1 did not start automaticly. I check the dependency and this service depend on the OracleAsmService+ASM1.
I even put a delay on the dependency, but the problem still real.
The ASM service start without problem. The only way i can start the RDBMS i to do it manually in the window "Computer management".
How i can generate a autostart off this service (RDBMS) when the service ASM start?
I check my registery variables and they are all OK.
Thanks for your help.I'm not shure if i will drop this service on a cluster.
I can use the comand
sc delete OracleServiceTest1
to delete the service in Window. This action won't affect the cluster.
But, how to recreate only this service in Window only?
Edited by: Ron_B on 2010-04-28 12:44
Maybe you are looking for
-
What dll do I use and where are the .h include files?
I recently downloaded the wavetek 271 drivers and have found that when imported (Lab View 6.0) that they need to be linked to a dll. But which one? And the includes call for the glib.h and other libraries where are they at. Are they compressed? thank
-
IPhone 4.0 and collaboration services
Did anyone catch or read anything about the new iPhone OS 4 supporting CardDAV or true PUSH for email, cals, and cards?
-
Reply and forward not working in full screen mail app
In the full screen version of the Mail application in Lion I can't reply or forward any messages. I have to go back to the old view in order to respond to mail. Is anyone else having this problem and if so, is there a fix? The full screen apps and
-
RFC's for SAP Performance data
I am in the process of developing a Performance Monitoring System for SAP in Java. The requirement is to fetch the performance metrics from SAP in a Java application. I know there are some RFC's available in CCMS module. But the customer is not inter
-
Downloading multiple files from a site with single request?
Interesting Scenario: User clicks on check boxes indicating which files they want to download from a site, then clicks a "Download" button. Is there a way (other than ZIPPING them) to download each file individually to the directory they specify? We