ASM : CSSD
Hi,
I am reading oracle 10g Rac-grid-service,Murali vallath book.
The page number 125 says that " When installing ASM on a single-instance configuration, ensure that the cluster synchronization services (CSS) module is installed."
Does it mean voting disk is mandatory for a single instance database.
and
Page number 48 says that "
Using a vacuous monitoring over the voting disk locations, CSS
performs state changes to bring the voting disk online. This is to
determine if CSS has a registered MASTER node already active. The
various states of the voting disk are
1 - Not configured and no thread has been spawned
2 - Threads are spawned
3 - Thread started and disk is offline
4 - The voting disk is online "
what is vacuous monitoring?
Is there any way to identify voting disk state?
Please help me.
Thanks & Regards,
If you are installing single instance ASM then CSSD daemon is used to communicate with ASM disks. In single instance environment, Voting disk is not needed. Instead, CSS will communicate with OLR which is oracle local registry. This daemon is mandatory to start ASM instance for non RAC environment. If you kill this daemon process then node will reboot so please be careful with this process.
Similar Messages
-
Cssd does not start in non-RAC environment Thus we can not bring up ASM
Non-RAC environment
ASM version = 11.1.0.7
HP-UX Itanium 11.23
After power outage, CSSD does not start on non-RAC environment
Running as root "/sbin/init.d/init.cssd start" does not start cssd
Oracle support tried "$ASM_HOME/bin/localconfig delete" and "$ASM_HOME/bin/localconfig add"
but it did not start CSS
Oracle support tried "$ASM_HOME/bin/localconfig reset $ASM_HOME"
It started the CSSD and the "crsctl check css" came back with CSS is healthy
But around 1 minute later it rebooted the server and when it came up again CSS does not start.
They checked /etc/inittab and it looked fine.
Before the reboot we saw this message in the /var/adm/syslog/OLDsyslog.log:
Cluster Ready Services completed waiting on dependencies
Again it is a NON-RAC environment. We only need CSSD for ASM. We do not have CRS installed on this server.
Our test system has been down for a week and we did not get the resolution from Oracle support yet !
Any pointers are greately appriciated.
Thanks,
DzungHere is the message in $ASM_HOME/log/<hostname>/alert<hostname>.log :
2010-07-16 09:42:02.956
[client(11930)]CRS-1006:The OCR location /db/app/oracle/product/11.1/cdata/localhost/local.ocr is inacce
ssible. Details in /db/app/oracle/product/11.1/log/rmodbd01/client/clscfg10.log.
2010-07-16 09:42:02.971
[client(11930)]CRS-1006:The OCR location /db/app/oracle/product/11.1/cdata/localhost/local.ocr is inacce
ssible. Details in /db/app/oracle/product/11.1/log/rmodbd01/client/clscfg10.log.
2010-07-16 09:42:03.054
[client(11930)]CRS-1013:The OCR at /db/app/oracle/product/11.1/cdata/localhost/local.ocr was successfull
y formatted using version 2. Ignore earlier CRS-1006 messages if any.
2010-07-16 09:42:46.379
[cssd(12297)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rmodbd01 .
Here is the message in $ASM_HOME/log/<hostname>/cssd/cssdOUT.log:
setsid: failed with -1/1
s0clssscGetEnvOracleUser: calling getpwnam_r for user oracle
s0clssscGetEnvOracleUser: info for user oracle complete
07/16/10 09:42:36: CSSD starting
Here is the message in $ASM_HOME/log/<hostname>/cssd/ocssd.log:
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 7, from 6, changes 6
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmSetVersions: properties common to all peers: 1,2,3,4,5,6,7
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmEstablishMasterNode: MASTER for 174732166 is node(0) birth(174732166)
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 8, from 7, changes 7
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 9, from 8, changes 8
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 10, from 9, changes 9
[ CSSD]CLSS-3000: reconfiguration successful, incarnation 174732166 with 1 nodes
[ CSSD]CLSS-3001: local node number 0, master node number 0
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssscSAGEInitFenceCompl: Completing kgzf fence initialization
[ CSSD]2010-07-16 09:42:46.394 [12] >TRACE: clssgmUpdateEventValue: Client listener incarn val 174732166, changes 1
[ CSSD]2010-07-16 09:42:46.395 [12] >TRACE: clssgmAllocProc: (60000000003c7120) allocated
[ CSSD]2010-07-16 09:42:46.395 [12] >TRACE: clssgmAllocProc: (60000000003c73a0) allocated
[ CSSD]2010-07-16 09:42:46.396 [14] >TRACE: Connect request from user oracle
[ CSSD]2010-07-16 09:42:46.396 [14] >TRACE: Connect request from user root
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c7120 - 1,2,3
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2810) proc(60000000003c7120) pid(12350) version 11:1:1:4
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c73a0 - 1,2,3
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2990) proc(60000000003c73a0) pid(12131) version 11:1:1:4
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(1/600000000096e7c0)
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 1 (600000000096e7c0)
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmJoinGrock: local grock CSS_INTERNAL_NODE_GROUP new client 600000000096e7c0 with con 60000000003b2b10, requested num 0
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmAddNodeGrpMember: member (60000000009e0030) added
[ CSSD]2010-07-16 09:42:46.403 [12] >TRACE: clssgmGroupState: requested group state of group localhost_NG, member count 0
[ CSSD]2010-07-16 09:42:46.403 [12] >TRACE: clssgmGroupState: requested group state of group localhost_NG, member count 0
[ CSSD]2010-07-16 09:42:46.404 [12] >TRACE: clssgmDeadProc: proc 60000000003c73a0
[ CSSD]2010-07-16 09:42:46.404 [12] >TRACE: clssgmDestroyProc: cleaning up proc(60000000003c73a0) con(60000000003b2990) skgpid ospid 12131 with 0 clients, refcount 0
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmGroupState: requested group state of unknown group MASTER#DISKMON#GROUP
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmGroupState: requested group state of group MASTER#DISKMON#GROUP, member count 0
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: KGZF: context successfully initialized, API version 1.4, using pipe default
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssscSAGEInitFenceCompl: kgzf fence initialization successfully completed
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmReconfigThread: CSS/GM open for global group registrations
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmReconfigThread: completed for reconfig(174732166), with status(1)
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmUpdateEventValue: Reconfig Event val 2, changes 2
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmWaitOnEventValue: after Reconfig Event val 2, eval 2 waited 47
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(2/600000000096e870)
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmUpdateEventValue: Reconfig Event val 0, changes 3
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmStartNMMon: previous reconfig complete, incarnation(174732166)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmJoinGrock: global grock MASTER#DISKMON#GROUP#MX new client 600000000096e870 with con 60000000003b2990, requested num -1
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddGrockMember: adding member to grock MASTER#DISKMON#GROUP#MX
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddMember: granted member(0) flags(0x2) node(0) grock (6000000000989e50/MASTER#DISKMON#GROUP#MX)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmQueueGrockEvent: lockName(MASTER#DISKMON#GROUP#MX) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmCommonAddMember: global lock grock MASTER#DISKMON#GROUP#MX member(0/Local) node(0) flags 0x2 0x2
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(3/600000000096e920)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 3 (600000000096e920)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmJoinGrock: global grock MASTER#DISKMON#GROUP new client 600000000096e920 with con 60000000003b2bd0, requested num 0
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddGrockMember: adding member to grock MASTER#DISKMON#GROUP
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddMember: new master 0 for group(MASTER#DISKMON#GROUP)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmAddMember: Adding fencing for member 0, group MASTER#DISKMON#GROUP, death 1, SAGE 0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmAddMember: member (0/60000000009e0230) added. pbsz(72) prsz(0) flags 0x0 to grock (600000000098a170/MASTER#DISKMON#GROUP)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmQueueGrockEvent: groupName(MASTER#DISKMON#GROUP) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x78, state 0x0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmCommonAddMember: global group grock MASTER#DISKMON#GROUP member(0/Local) node(0) flags 0x0 0x1e00
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmExitGrock: client 2 (600000000096e870), grock MASTER#DISKMON#GROUP#MX, member 0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock MASTER#DISKMON#GROUP#MX
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmRemoveMember: grock MASTER#DISKMON#GROUP#MX, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 3
[ CSSD]2010-07-16 09:42:48.405 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:48.405 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:42:52.444 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:52.444 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:42:56.484 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:56.484 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:00.516 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:00.516 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:04.563 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:04.563 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:08.603 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:08.603 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:12.643 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:12.643 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:16.676 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:16.676 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:20.723 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:20.723 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:24.762 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:24.762 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:28.802 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:28.802 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:32.842 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:32.842 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:36.882 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:36.882 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:40.922 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:40.922 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:44.964 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:44.964 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:49.002 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:49.002 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:53.043 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:53.043 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:57.085 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:57.085 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:58.354 [12] >TRACE: clssgmAllocProc: (60000000003c7ee0) allocated
[ CSSD]2010-07-16 09:43:58.355 [14] >TRACE: Connect request from user oracle
[ CSSD]2010-07-16 09:43:58.356 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c7ee0 - 1,2,3
[ CSSD]2010-07-16 09:43:58.356 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2990) proc(60000000003c7ee0) pid(13157) version 11:1:1:4
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmRegisterClient: proc(2/60000000003c7ee0), client(1/600000000096e870)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 1 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmJoinGrock: global grock CLSSSCHECK_GROUP new client 600000000096e870 with con 60000000003b2c90, requested num -1
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddGrockMember: adding member to grock CLSSSCHECK_GROUP
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: new master 0 for group(CLSSSCHECK_GROUP)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: Adding fencing for member 0, group CLSSSCHECK_GROUP, death 1, SAGE 0
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: member (0/60000000009e0130) added. pbsz(8) prsz(8) flags 0x0 to grock (6000000000989e50/CLSSSCHECK_GROUP)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmQueueGrockEvent: groupName(CLSSSCHECK_GROUP) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x0, state 0x0
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmCommonAddMember: global group grock CLSSSCHECK_GROUP member(0/Local) node(0) flags 0x0 0x0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 1 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExitGrock: client 1 (600000000096e870), grock CLSSSCHECK_GROUP, member 0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock CLSSSCHECK_GROUP
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRemoveMember: grock CLSSSCHECK_GROUP, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 2
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRegisterClient: proc(2/60000000003c7ee0), client(2/600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmJoinGrock: global grock CLSSSCHECK_LOCK new client 600000000096e870 with con 60000000003b2c90, requested num -1
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmAddGrockMember: adding member to grock CLSSSCHECK_LOCK
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmAddMember: granted member(0) flags(0x2) node(0) grock (6000000000989e50/CLSSSCHECK_LOCK)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmQueueGrockEvent: lockName(CLSSSCHECK_LOCK) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmCommonAddMember: global lock grock CLSSSCHECK_LOCK member(0/Local) node(0) flags 0x2 0x2
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExitGrock: client 2 (600000000096e870), grock CLSSSCHECK_LOCK, member 0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock CLSSSCHECK_LOCK
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRemoveMember: grock CLSSSCHECK_LOCK, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 3
[ CSSD]2010-07-16 09:43:58.362 [12] >TRACE: clssgmDeadProc: proc 60000000003c7ee0
[ CSSD]2010-07-16 09:43:58.362 [12] >TRACE: clssgmDestroyProc: cleaning up proc(60000000003c7ee0) con(60000000003b2990) skgpid ospid 13157 with 0 clients, refcount 0
[ CSSD]2010-07-16 09:44:01.125 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:01.125 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:05.164 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:05.164 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:09.196 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:09.196 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:13.236 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:13.236 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:17.284 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:17.284 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:21.322 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:21.322 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:25.316 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:25.316 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:30.333 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:30.333 [8] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[ CSSD]2010-07-16 09:44:34.324 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:34.324 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
Here is the message in $ASM_HOME/log/<hostname>/diskmon/client.log:
[ DISKMON] 07/16/2010 18:33:32.050 dskm_send_command: process 23246 sending command 8 to master diskmon listening on default pipe
[ DISKMON] 07/16/2010 18:33:32.080 dskm_send_command3: skgznp_connect failed with error 56815
[ DISKMON] 07/16/2010 18:33:32.080 dskm_send_command3: error 56815 at location skgznpcon6 - connect() - Connection refused
Here is the message in $ASM_HOME/log/<hostname>/diskmon/diskmonOUT.log:
setsid: failed with -1/1
dskm_getenv_oracle_user: calling getpwnam_r for user oracle
dskm_getenv_oracle_user: info for user oracle complete
07/16/10 18:33:31: Master Diskmon starting
Here is the message in $ASM_HOME/log/<hostname>/diskmon/diskmon.log:
[ DISKMON] 07/16/2010 09:42:35.573 dskm main: starting up
[ DISKMON] 07/16/2010 09:42:35.588 [12350:3] dskm_rac_thrd_main: running
[ DISKMON] 07/16/2010 09:42:35.588 [12350:1] dskm_rac_thrd_creat2: got the post from the css event handling thread
[ DISKMON] 07/16/2010 09:42:35.589 [12350:1] dskm main: startup complete
[ DISKMON] 07/16/2010 09:42:35.589 [12350:1] listening on -> default pipe
[ DISKMON] 07/16/2010 09:42:35.792 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:36.385 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:36.906 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:37.426 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:37.945 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:38.465 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:46.376 [12350:1] dskm_slave_thrd_creat: thread created
[ DISKMON] 07/16/2010 09:42:46.376 [12350:11] dskm_slave_thrd_main1: slave 0 running
[ DISKMON] 07/16/2010 09:42:46.376 [12350:11] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_proc_identify8: client kgzf/12297, version 0x01020000, slave 0, reid cid=3e0391f05e06cfafbf7419d7cf085a44,icin=174732166,nmn=0,lnid=174732166,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_send_version1:
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_send_version4: done
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_process_msg5: received msg type KGZM_KGZF_HANDSHAKE (0x0010)
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_proc_kgzf_handshake3: client kgzf/12297, kgzf version 0x00010004, slave 0
[ DISKMON] 07/16/2010 09:42:46.401 [12350:3] dskm_clss_ini2: successful clsssinit(), clssvers 2.1
[ DISKMON] 07/16/2010 09:42:46.402 [12350:3] dskm_clss_ini12: node rmodbd01 (0) registered in cluster
[ DISKMON] 07/16/2010 09:42:46.403 [12350:3] dskm_reid_ini12: diskmon reid cid=3e0391f05e06cfafbf7419d7cf085a44,icin=174732166,nmn=0,lnid=174732166,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=12350,opsn=1279291355,lvl=process
[ DISKMON] 07/16/2010 09:42:46.424 [12350:3] dskm_sage_config: CELL storage configuration file cellinit.ora not found
[ DISKMON] 07/16/2010 09:42:46.425 [12350:3] dskm_nfy_kgzf1: notified thread kgzf enabled
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_proc_kgzf_handshake5: got the post from the hb thread
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_proc_kgzf_handshake9: done, kgzf enabled
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
[ DISKMON] 07/16/2010 09:42:46.426 [12350:3] dskm_rac_ini22: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 0 (GSDGRPSZ 512) -
Switching from one ASM installation to another ASM installation
Hi,
We've a server Red Hat Enterprise Linux Server release 5.3 (Tikanga) with installed two 11.1.0.7.0 installation: one for ASM and the other for rdmbs.
Now, we want to install the 11.1.0.7.4 PSU.
For reducing downtime we'd like to create two new 11.1.0.7.0 installation; then patching them with the 11.1.0.7.4 PSU.
Then we'd like to stop ASM and all database running and switching the two old installation with the new one.
For the rdbms installation it's easy: just to modify the oratab and then starting the databases with the new patched installation and then follow the post installation steps (atbundle.sql, etc.)
For ASM instead, how can I do it ? How can I reconfigure the ocssd for using the new ASM installation ? And do I need to do some other steps ?
Thanks
massiHi,
Probably my post was not clear enough. Sorry.
Here our scenario:
- we have one linux server with two Oracle11g installations. We use one installation for ASM and the other one for the rdbms. We planned to install the PSU 4 on both the installations.
We could simply shutdown all databases and ASM. Then install PSU4 on both ASM and rdmbs installation. Then startup AMS and databases and proceed with post PSU installation tasks.
There is another possibility. To install in advance two new Oracle11g: one for ASM and the other one for the rdbms. Then apply PSU4 on these two new Oracle11g installations. Then shutdown the databases and the ASM. Then execute:
$ORACLE_HOME/bin/localconfig reset <new_ASM_oracle_home>
stop cssd (/etc/init.d/init.cssd stop)
start cssd (/etc/init.d/init.cssd start)
Then change in /etc/oratab so that databases will point to the new installation (the new one installed for rdbms)
Then copy spfile and password files to the new oracle_home/dbs (the new one installed for rdbms)
Then change in /etc/oratab so that ASM is using the new installation (the new one installed forASM)
Then copy spfile and password file to the new oracle_home/dbs used by ASM (the new one installed forASM)
Finally restart ASM and databases
This way, the ASM will point to the new installation - and the databases will use the other new installation. This way the downtime will be shorter. It will be only necessary to apply the post PSU installation tasks
I've already did it last weekend. It worked
Thanks
massi
Thanks
massi -
Can't create database using ASM (SOLVED)
Hi all
I'm trying to use ASM for the first time, on Oracle 10.2.0.1 on Solaris x64.
I have installed the ASM instance into /opt/oracle/asm/10.2.0 and created disk groups. I have cssd running OK. I am able to start and stop the ASM instance without problems, and I can select from v$asm_diskgroup to confirm that disks are mounted OK.
I have then installed Oracle EE separately into /opt/oracle/server/10.2.0. I first did a software only install, and now I am trying to create a DB.
The problems come when I try to use this ASM instance to host a new database. I first tried to use DBCA to create a new database, but on database creation I got the following errors:
ORA-00200: control file could not be created
ORA-00202: control file: '+DBLIVE1'
ORA-17502: ksfdcre:4 Failed to create file +DBLIVE1
ORA-15001: diskgroup "DBLIVE1" does not exist or is not mounted
ORA-15055: Message 15055 not found; No message file for product=RDBMS, facility=ORA
ORA-01031: insufficient privileges
I then told DBCA just to create the DB creation scripts, and I tried manually running these with SQL*Plus.
When doing it with SQL*PLus, I initially got the same error as shown above. But then something changed (sorry, not sure what), and now the error I get is:
CREATE DATABASE "NEONREL1"
ERROR at line 1:
ORA-01501: CREATE DATABASE failed
ORA-00349: failure obtaining block size for '+DBLIVE1'
ORA-01031: insufficient privileges
I've put some debug info below, showing me succesfully connecting to the ASM instance and then attempting to create the DB using the db creation scripts, showing the error at the end. You can see that the oracle OS user is able to connect fine to ASM, then I swithc ORACLE_SID and ORACLE_HOME to the EE install and try to create the DB, at which point it apparently can't connect to ASM any more.
I've tried the DB creation many times, and in between attempts I completely empty $ORACLE_HOME/admin/<dbname> and delete the files related to the attempted install from $ORACLE_HOME/dbs/ . I've also stopping/starting ASM, rebooting, and I've done the install of ASM and EE a couple of times over in case I made any mistakes in my earlier attempts.
Any help would be much appreciated!
Tom
##### CHECKING ASM
oracle@neonrcom-db1:~$ uname -a
SunOS neonrcom-db1 5.10 Generic_127128-11 i86pc i386 i86pc
# css is running
oracle@neonrcom-db1:~$ ps -ef | grep css
oracle 498 1 0 21:46:40 ? 0:01 /opt/oracle/asm/10.2.0/bin/ocssd.bin
# listener is running in the ASM instance
oracle@neonrcom-db1:~$ ps -ef | grep tnsl
oracle 1332 1 0 21:49:59 ? 0:00 /opt/oracle/asm/10.2.0/bin/tnslsnr LISTENER -inherit
# ASM is only entry in /var/opt/oracle/oratab
oracle@neonrcom-db1:~$ grep -v "^#" /var/opt/oracle/oratab
+ASM:/opt/oracle/asm/10.2.0:N
# I can connect to ASM fine, and it has diskgroups mounted.
oracle@neonrcom-db1:~$ export ORACLE_HOME=/opt/oracle/asm/10.2.0
oracle@neonrcom-db1:~$ export ORACLE_SID='+ASM'
oracle@neonrcom-db1:~$ sqlplus "sys as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Mon Jul 21 20:53:10 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL> set line 150
SQL> select name, block_size, state, type, total_mb, free_mb from v$asm_diskgroup;
NAME BLOCK_SIZE STATE TYPE TOTAL_MB FREE_MB
DBARCH1 4096 MOUNTED EXTERN 2096856 2096784
DBLIVE1 4096 MOUNTED EXTERN 4193904 4193812
#### Contents of init.ora for new DB
db_create_file_dest=+DBLIVE1
db_recovery_file_dest=+DBARCH1
db_recovery_file_dest_size=2147483648
##### DB INSTALLATION ATTEMPT
oracle@neonrcom-db1:~$ export ORACLE_HOME=/opt/oracle/server/10.2.0
oracle@neonrcom-db1:~$ export ORACLE_SID='NEONREL1'
oracle@neonrcom-db1:~$ export PATH=$ORACLE_HOME/bin:$PATH
oracle@neonrcom-db1:~$ /opt/oracle/server/10.2.0/admin/NEONREL1/scripts/NEONREL1.sh
You should Add this entry in the /var/opt/oracle/oratab: NEONREL1:/opt/oracle/server/10.2.0:Y
SQL*Plus: Release 10.2.0.1.0 - Production on Mon Jul 21 22:10:54 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
specify a password for sys as parameter 1
Enter value for 1: xxx
specify a password for system as parameter 2
Enter value for 2: xxx
specify a password for sysman as parameter 3
Enter value for 3: xxx
specify a password for dbsnmp as parameter 4
Enter value for 4: xxx
specify ASM SYS user password as parameter 6
Enter value for 6: xxx
Connected to an idle instance.
SQL> spool /opt/oracle/server/10.2.0/admin/NEONREL1/scripts/CreateDB.log
SQL> startup nomount pfile="/opt/oracle/server/10.2.0/admin/NEONREL1/scripts/init.ora";
ORACLE instance started.
Total System Global Area 1.9294E+10 bytes
Fixed Size 2054976 bytes
Variable Size 2264925376 bytes
Database Buffers 1.7012E+10 bytes
Redo Buffers 14721024 bytes
SQL> CREATE DATABASE "NEONREL1"
2 MAXINSTANCES 8
3 MAXLOGHISTORY 1
4 MAXLOGFILES 16
5 MAXLOGMEMBERS 3
6 MAXDATAFILES 100
7 DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
8 EXTENT MANAGEMENT LOCAL
9 SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
10 SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
11 SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
12 CHARACTER SET AL32UTF8
13 NATIONAL CHARACTER SET UTF8
14 LOGFILE GROUP 1 SIZE 51200K,
15 GROUP 2 SIZE 51200K,
16 GROUP 3 SIZE 51200K
17 USER SYS IDENTIFIED BY "&&sysPassword" USER SYSTEM IDENTIFIED BY "&&systemPassword";
CREATE DATABASE "NEONREL1"
ERROR at line 1:
ORA-01501: CREATE DATABASE failed
ORA-00349: failure obtaining block size for '+DBLIVE1'
ORA-01031: insufficient privileges
Message was edited by:
tjobbinsUpdate: I've worked out the difference between the two sets of errors I get.
The basic error is this:
ORA-00200: control file could not be created
ORA-00202: control file: '+DBLIVE1'
ORA-17502: ksfdcre:4 Failed to create file +DBLIVE1
ORA-15001: diskgroup "DBLIVE1" does not exist or is not mounted
ORA-15055: Message 15055 not found; No message file for product=RDBMS, facility=ORA
ORA-01031: insufficient privileges
However if my init.ora contains the line:
control_files=/opt/oracle/server/10.2.0/dbs/cntrlNEONREL1.dbf
then I instead get the second error:
CREATE DATABASE "NEONREL1"
ERROR at line 1:
ORA-01501: CREATE DATABASE failed
ORA-00349: failure obtaining block size for '+DBLIVE1'
ORA-01031: insufficient privileges
So basically these must be the same error, just in the second case I'm not trying to put the control file on the ASM so it fails at a different point.
But both errors must be because of the same cause, I suppose. -
ASM instance crash due to error ORA-27506: IPC error connecting to a port
Hi All,
Today the ASM instance goes down.
When i checked the alert log I found the below error.
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:sendmsg failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgxpsnd1
Please find the environment details.
OS : RHEL-5
DB: 11.1.0.7 2-node RAC
I want to know the root cause of this issue.
Please suggest.
Thanks and Regards,Hi,
Could you please upload cluster alert log and cssd.log?
regards,
Kishore -
Dear all,
I am trying to install clean Oracle 11.2.0.3 grid infrastructure on a two node cluster running on Solaris 5.10.
- Cluster verification was successfully on both nodes; No warning or issues;
- I am using 2 network cards for the public and 2 for the private interconnect;
- OCR is stored on ASM
- Firewall is disabled on both nodes
- SCAN is being configured on the DNS (not added in /etc/hosts)
- GNS is not used
- hosts file is identical (except the primary hostname)
The problem: root.sh fails on the 2nd (remote) node, because it fails to start the "ora.asm" resource. However, the root.sh has completed successfully on the 1st node.. Somehow, root.sh doesn't create +ASM2 instance on the remote (host2) node.
root.sh was executed first on the local node (host1) and after the successful execution was started on the remote (host2) node.
Output from host1 (working):
===================
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'host1'
CRS-2676: Start of 'ora.mdnsd' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'host1'
CRS-2676: Start of 'ora.gpnpd' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'host1'
CRS-2672: Attempting to start 'ora.gipcd' on 'host1'
CRS-2676: Start of 'ora.cssdmonitor' on 'host1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'host1'
CRS-2672: Attempting to start 'ora.diskmon' on 'host1'
CRS-2676: Start of 'ora.diskmon' on 'host1' succeeded
CRS-2676: Start of 'ora.cssd' on 'host1' succeeded
ASM created and started successfully.
Disk Group CRS created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 4373be34efab4f01bf79f6c5362acfd3.
Successful addition of voting disk 7fd725fa4d904f07bf76cecf96791547.
Successful addition of voting disk a9c85297bdd74f3abfd86899205aaf17.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE 4373be34efab4f01bf79f6c5362acfd3 (/dev/rdsk/c4t600A0B80006E2CC40000C6674E82AA57d0s4) [CRS]
2. ONLINE 7fd725fa4d904f07bf76cecf96791547 (/dev/rdsk/c4t600A0B80006E2CC40000C6694E82AADDd0s4) [CRS]
3. ONLINE a9c85297bdd74f3abfd86899205aaf17 (/dev/rdsk/c4t600A0B80006E2F100000C7744E82AC7Ad0s4) [CRS]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'host1'
CRS-2676: Start of 'ora.asm' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'host1'
CRS-2676: Start of 'ora.CRS.dg' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'host1'
CRS-2676: Start of 'ora.registry.acfs' on 'host1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Name Type Target State Host
ora.CRS.dg ora....up.type ONLINE ONLINE host1
ora....ER.lsnr ora....er.type ONLINE ONLINE host1
ora....N1.lsnr ora....er.type ONLINE ONLINE host1
ora....N2.lsnr ora....er.type ONLINE ONLINE host1
ora....N3.lsnr ora....er.type ONLINE ONLINE host1
ora.asm ora.asm.type ONLINE ONLINE host1
ora....SM1.asm application ONLINE ONLINE host1
ora....B1.lsnr application ONLINE ONLINE host1
ora....db1.gsd application OFFLINE OFFLINE
ora....db1.ons application ONLINE ONLINE host1
ora....db1.vip ora....t1.type ONLINE ONLINE host1
ora.cvu ora.cvu.type ONLINE ONLINE host1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE host1
ora.oc4j ora.oc4j.type ONLINE ONLINE host1
ora.ons ora.ons.type ONLINE ONLINE host1
ora....ry.acfs ora....fs.type ONLINE ONLINE host1
ora.scan1.vip ora....ip.type ONLINE ONLINE host1
ora.scan2.vip ora....ip.type ONLINE ONLINE host1
ora.scan3.vip ora....ip.type ONLINE ONLINE host1
Output from host2 (failing):
===================
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node billdb1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Start of resource "ora.asm" failed
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'host2'
CRS-2676: Start of 'ora.drivers.acfs' on 'host2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'host2'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
*. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".*
CRS-2674: Start of 'ora.asm' on 'host2' failed
CRS-2679: Attempting to clean 'ora.asm' on 'host2'
CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start ASM at /u01/11.2.0/grid/crs/install/crsconfig_lib.pm line 1272.
/u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib -I/u01/11.2.0/grid/crs/install /u01/11.2.0/grid/crs/install/rootcrs.pl execution failed
Contents of "/u01/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_host2.log"
=============================================
CRS-2672: Attempting to start 'ora.asm' on 'host2'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on 'host2' failed
CRS-2679: Attempting to clean 'ora.asm' on 'host2'
CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
CRS-4000: Command Start failed, or completed with errors.
2011-10-24 19:36:54: Failed to start Oracle Grid Infrastructure stack
2011-10-24 19:36:54: ###### Begin DIE Stack Trace ######
2011-10-24 19:36:54: Package File Line Calling
2011-10-24 19:36:54: --------------- -------------------- ---- ----------
2011-10-24 19:36:54: 1: main rootcrs.pl 375 crsconfig_lib::dietrap
2011-10-24 19:36:54: 2: crsconfig_lib crsconfig_lib.pm 1272 main::__ANON__
2011-10-24 19:36:54: 3: crsconfig_lib crsconfig_lib.pm 1171 crsconfig_lib::start_cluster
2011-10-24 19:36:54: 4: main rootcrs.pl 803 crsconfig_lib::perform_start_cluster
2011-10-24 19:36:54: ####### End DIE Stack Trace #######
Shortened output from "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log"
2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] clean {
2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop_option stop mode immediate option 1
2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop {
2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] InstAgent::stop original reason system do shutdown abort
2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00ab1948
2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection sid +ASM2 status 2
2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check OH /u01/11.2.0/grid SID +ASM2
2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,1,7 exists
2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] (:CLSN00006:)AsmAgent::check failed gimh state 0
2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 0
2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet { entry
2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: retval [0] configured [1] local only [0] error buffer []
2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: OCR loc [0], Disk Group : [+CRS]
2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet m_ocrDgpSet 015fba90 dgName CRS
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrret 0 found 1
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrDgpSet CRS
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet exit }
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::ocrDgCheck Entry {
2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool
2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv:
2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/11.2.0/grid, m_oracleSid:+ASM2, m_usrOraEnv:
2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] ConnectionPool::addConnection m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv: pConnxn:
01fcdf10
2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] Utils::getCrsHome crsHome /u01/11.2.0/grid
2011-10-24 19:35:51.969: [ora.asm][14] {0:0:224} [check] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/11.2.0/grid/bin/oracle)(ARGV0=o
racle+ASM2)(ENVS='ORACLE_HOME=/u01/11.2.0/grid,ORACLE_SID=+ASM2')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM2)))
2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] ConnectionPool::getConnection 260 pConnxn 013e40a0
2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] DgpAgent::getConnxn connected
2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt: server not attached
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt (2) Exception OCIException
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection:connect:excp OCIException OCI error 1034
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus excp ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus asm inst is down or going down
2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus dgName CRS ret 1
2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] (:CLSN00100:)DgpAgent::ocrDgCheck OCR dgName CRS state 1
2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] ConnectionPool::releaseConnection InstConnection 013e40a0
2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] AsmAgent::check ocrCheck 2 m_OcrOnline 0 m_OcrTimer 0
2011-10-24 19:35:52.193: [ora.asm][14] {0:0:224} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
2011-10-24 19:35:52.197: [ COMMCRS][23]clsc_connect: (1020d39d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
Please advice for any workaround or a metalink note.
Thanks in advance!Thanks for the fast reply!
- Yes, the shared storage is accessible.
- The alert log for the +ASM2 clearly shows that ASM instance has started normally using default parameters and at one point PMON process dumped.
- The system logs just shows that there is an error executing "crswrapexece.pl"
System Log
===================
*Oct 24 19:25:03 host2 root: [ID 702911 user.error] exec /u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib /u01/11.2.0/grid/bin/crswrapexece.pl /*
u01/11.2.0/grid/crs/install/s_crsconfig_host2_env.txt /u01/11.2.0/grid/bin/ohasd.bin "reboot"
Oct 24 19:26:33 host2 oracleoks: [ID 902884 kern.notice] [Oracle OKS] mallocing log buffer, size=10485760
Oct 24 19:26:33 host2 oracleoks: [ID 714332 kern.notice] [Oracle OKS] log buffer = 0x301780fcb50, size 10485760
Oct 24 19:26:33 host2 oracleoks: [ID 400061 kern.notice] NOTICE: [Oracle OKS] ODLM hash size 16384
Oct 24 19:26:33 host2 oracleoks: [ID 160659 kern.notice] NOTICE: OKSK-00004: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
IS.SPARC64_110803.1 2011/08/11 02:38:30
Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleadvm0
Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleadvm0 is /pseudo/oracleadvm@0
Oct 24 19:26:33 host2 oracleoks: [ID 141287 kern.notice] NOTICE: ADVMK-00001: Module load succeeded. Build information: (LOW DEBUG) - USM_11.2.0.3.0_SOL
ARIS.SPARC64_110803.1 built on 2011/08/11 02:40:17.
Oct 24 19:26:33 host2 oracleacfs: [ID 202941 kern.notice] NOTICE: [Oracle ACFS] FCB hash size 16384
Oct 24 19:26:33 host2 oracleacfs: [ID 671725 kern.notice] NOTICE: [Oracle ACFS] buffer cache size 511MB (79884 buckets)
Oct 24 19:26:33 host2 oracleacfs: [ID 730054 kern.notice] NOTICE: [Oracle ACFS] DLM hash size 16384
Oct 24 19:26:33 host2 oracleoks: [ID 617314 kern.notice] NOTICE: ACFSK-0037: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
IS.SPARC64_110803.1 2011/08/11 02:42:45
Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleacfs0
Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleacfs0 is /pseudo/oracleacfs@0
Oct 24 19:26:36 host2 oracleoks: [ID 621795 kern.notice] NOTICE: OKSK-00010: Persistent OKS log opened at /u01/11.2.0/grid/log/host2/acfs/acfs.log.0.
Oct 24 19:31:37 host2 last message repeated 1 time
Oct 24 19:33:05 host2 CLSD: [ID 770310 daemon.notice] The clock on host host2 has been updated by the Cluster Time Synchronization Service to be synchr
onous with the mean cluster time.
ASM alert log
====================================================================
<msg time='2011-10-24T19:35:48.776+01:00' org_id='oracle' comp_id='asm'
client_id='' type='UNKNOWN' level='16'
host_id='host2' host_addr='10.172.16.200' module=''
pid='26406'>
<txt>System state dump requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
</txt>
</msg>
<msg time='2011-10-24T19:35:48.778+01:00' org_id='oracle' comp_id='asm'
client_id='' type='UNKNOWN' level='16'
host_id='host2' host_addr='10.172.16.200' module=''
pid='26406'>
<txt>System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc
</txt>
</msg>
<msg time='2011-10-24T19:35:48.927+01:00' org_id='oracle' comp_id='asm'
type='UNKNOWN' level='16' host_id='host2'
host_addr='10.172.16.200' pid='26470'>
<txt>ORA-1092 : opitsk aborting process
</txt>
</msg>
<msg time='2011-10-24T19:35:49.128+01:00' org_id='oracle' comp_id='asm'
type='UNKNOWN' level='16' host_id='host2'
host_addr='10.172.16.200' pid='26472'>
<txt>ORA-1092 : opitsk aborting process
</txt>
</msg>
Output from "/u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc"
REQUEST:system state dump at level 10, requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
===================================================
SYSTEM STATE (level=10)
Orapids on dead process list: [count = 0]
PROCESS 1:
SO: 0x3df098b50, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x3df098b50, name=process, file=ksu.h LINE:12616 ID:, pg=0
(process) Oracle pid:1, ser:0, calls cur/top: 0x0/0x0
flags : (0x20) PSEUDO
flags2: (0x0), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 0
last post received-location: No post
last process to post me: none
last post sent: 0 0 0
last post sent-location: No post
last process posted by me: none
(latch info) wait_event=0 bits=0
O/S info: user: , term: , ospid: (DEAD)
OSD pid info: Unix process pid: 0, image: PSEUDO
SO: 0x38000cef0, type: 5, owner: 0x3df098b50, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x0, name=kss parent, file=kss2.h LINE:138 ID:, pg=0
PSO child state object changes :
Dump of memory from 0x00000003DF722AC0 to 0x00000003DF722CC8
3DF722AC0 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
3DF722CC0 00000000 00000000 [........]
PROCESS 2: PMON
SO: 0x3df099bf8, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x3df099bf8, name=process, file=ksu.h LINE:12616 ID:, pg=0
(process) Oracle pid:2, ser:1, calls cur/top: 0x3db6c8d30/0x3db6c8d30
flags : (0xe) SYSTEM
flags2: (0x0), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 136
last post received-location: kjm.h LINE:1228 ID:kjmdmi: pmon to attach
last process to post me: 3df0a2138 1 6
last post sent: 0 0 137
last post sent-location: kjm.h LINE:1230 ID:kjiath: pmon attached
last process posted by me: 3df0a2138 1 6
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 0x3debbbf40
O/S info: user: grid, term: UNKNOWN, ospid: 26396
OSD pid info: Unix process pid: 26396, image: oracle@host2 (PMON)
SO: 0x3d8800c18, type: 30, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x3df099bf8, name=ges process, file=kji.h LINE:3669 ID:, pg=0
GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=0
outq=0 sndq=0 opid=0 prmb=0x0
mbg=(0 0) mbg=(0 0) mbg[r]=(0 0)
fmq=(0 0) fmq=(0 0) fmq[r]=(0 0)
mop[s]=0 mop[q]=0 pendq=0 zmbq=0
nonksxp_recvs=0
------------process 3d8800c18--------------------
proc version : 0
Local inst : 2
pid : 26396
lkp_inst : 2
svr_mode : 0
proc state : KJP_FROZEN
Last drm hb acked : 0
flags : x50
ast_rcvd_svrmod : 0
current lock op : 0
Total accesses : 1
Imm. accesses : 0
Locks on ASTQ : 0
Locks Pending AST : 0
Granted locks : 0
AST_Q:
PENDING_Q:
GRANTED_Q:
SO: 0x3d9835198, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
(broadcast handle) 3d9835198 flag: (2) ACTIVE SUBSCRIBER,
owner: 3df099bf8 - ospid: 26396
event: 1, last message event: 1,
last message waited event: 1,
next message: 0(0), messages read: 0
channel: (3d9934df8) PMON actions channel [name: 2]
scope: 7, event: 1, last mesage event: 0,
publishers/subscribers: 0/1,
messages published: 0
heuristic msg queue length: 0
SO: 0x3d9835008, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
(broadcast handle) 3d9835008 flag: (2) ACTIVE SUBSCRIBER,
owner: 3df099bf8 - ospid: 26396
event: 1, last message event: 1,
last message waited event: 1,
next message: 0(0), messages read: 0
channel: (3d9941e40) scumnt mount lock [name: 157]
scope: 1, event: 12, last mesage event: 0,
publishers/subscribers: 0/12,
messages published: 0
heuristic msg queue length: 0
SO: 0x3de4a2b80, type: 4, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x3df099bf8, name=session, file=ksu.h LINE:12624 ID:, pg=0
(session) sid: 33 ser: 1 trans: 0x0, creator: 0x3df099bf8
flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x409) -/-/INC
DID: , short-term DID:
txn branch: 0x0
oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
ksuxds FALSE at location: 0
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 0.666415 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=0x0/-1
Session Wait History:
elapsed time of 0.666593 sec since last wait
0: waited for 'pmon timer'
duration=0x12c, =0x0, =0x0
wait_id=63 seq_num=64 snap_id=1
wait times: snap=3.000089 sec, exc=3.000089 sec, total=3.000089 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.002067 sec of elapsed time
1: waited for 'pmon timer'
duration=0x12c, =0x0, =0x0
wait_id=62 seq_num=63 snap_id=1
wait times: snap=3.010111 sec, exc=3.010111 sec, total=3.010111 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.001926 sec of elapsed time
2: waited for 'pmon timer'
duration=0x12c, =0x0, =0x0
wait_id=61 seq_num=62 snap_id=1
wait times: snap=3.125286 sec, exc=3.125286 sec, total=3.125286 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.003361 sec of elapsed time
3: waited for 'pmon timer'
duration=0x12c, =0x0, =0x0
wait_id=60 seq_num=61 snap_id=1
wait times: snap=3.000081 sec, exc=3.000081 sec, total=3.000081 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.002102 sec of elapsed time
4: waited for 'pmon timer'
duration=0x12c, =0x0, =0x0 -
The Script root.sh problem - ora.asm and ASM and Clusterware Stack failed
Folks,
Hello. I am installing Oracle 11gR2 RAC using 2 VMs (rac1 and rac2) whose OS are Oracle Linux 5.6 in VMPlayer according to the website http://appsdbaworkshop.blogspot.com/2011/10/11gr2-rac-on-linux-56-using-vmware.html
I am installing Grid infrastructure. On step 9 of 10 - execute script /u01/app/grid/root.sh for 2 VMs rac1 and rac2.
After run root.sh in rac1 successfully. I run root.sh in rac2 and get an error as below:
[root@rac2 grid]# ./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= ora11g
ORACLE_HOME= /u01/app/grid
Enter the full pathname of the local bin directory: [usr/local/bin]: /usr/local/bin
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-03-05 16:32:52: Parsing the host name
2012-03-05 16:32:52: Checking for super user privileges
2012-03-05 16:32:52: User has super user privileges
Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
Start action for octssd aborted
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2664: Resource 'ora.ctssd' is already running on 'rac2'
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl start resource ora.asm -init
Start of resource "ora.asm -init" failed
Failed to start ASM
Failed to start Oracle Clusterware stack
[root@rac2 grid]#
As we see the output above, at the end of the output
1) Start of resource ora.asm -init failed
2) Failed to start ASM
3) Failed to start Oracle Clusterware stack
The runInstaller is in the first VM rac1. My question is:
Do any folk understand how to solve the script root.sh in rac2 problem ( 3 fails of ora.asm, ASM and Clusterware stack as above) ?
Thanks.Please check there is no firewall exist:
try this like:
root.sh fails on second node
MOS note:
11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network [ID 981357.1]
Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] (Most probabily this issue) -
Failed to restart the CSSD during the interconnect failure
Hi all,
I run a small ATP on my LAB where i have
- 2x nodes RAC 11.2.0.2 & ASM (my OCR & Voting files are stored on ASM)
- 1 public interface <> eth0
- 1 private interface <> eth1
- 1 SCAN IP defined in the /etc/hosts file (i'm not using DNS or GNS)
The test i run was to shutdown the private interface (eth1) on node 1 and i saw that
1) all cluster services and cluster daemons on node 2 were killed and node 2 was evicted from the cluster by node 1
2) all new connections were redirected to the survived node
3) Oracle OHASD daemon was restarted on node 2 and tried to start the cluster services without success because private network between cluster nodes was down
Up to here everything worked as expected but once i turn on eth1 it took ~ 9 minutes for the CSSD to startup and bring all the components up & running.
The node2 alert logs showes
[ctssd(12949)]CRS-2402:The Cluster Time Synchronization Service aborted on host node2. Details at (:ctss_css_init1:) in /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/ctssd/octssd.log.
2011-04-13 08:09:40.978
[ohasd(5058)]CRS-2765:Resource 'ora.cssd' has failed on server 'node2'.
2011-04-13 08:09:40.985
[/u01/oracle/installed/oracle_cluster-11.2.0.2-1/bin/oraagent.bin(5764)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/agent/ohasd/oraagent_oracle/oraagent_oracle.log";
2011-04-13 08:09:41.169
[ohasd(5058)]CRS-2765:Resource 'ora.asm' has failed on server 'node2'.
2011-04-13 08:09:50.337
[cssd(13103)]CRS-1713:CSSD daemon is started in clustered mode
2011-04-13 08:10:05.833
[cssd(13103)]CRS-1707:Lease acquisition for node node2 number 2 completed
2011-04-13 08:10:07.119
[cssd(13103)]CRS-1605:CSSD voting file is online: ORCL:CRS_DISK1_2G; details in /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/cssd/ocssd.log.
2011-04-13 08:10:07.121
[cssd(13103)]CRS-1605:CSSD voting file is online: ORCL:CRS_DISK2_2G; details in /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/cssd/ocssd.log.
2011-04-13 08:10:07.143
[cssd(13103)]CRS-1605:CSSD voting file is online: ORCL:CRS_DISK1_2G; details in /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/cssd/ocssd.log.
2011-04-13 08:19:49.386
[/u01/oracle/installed/oracle_cluster-11.2.0.2-1/bin/cssdagent(13091)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:6:7} in /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2011-04-13 08:19:49.387
[cssd(13103)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/cssd/ocssd.log
2011-04-13 08:19:49.387
[cssd(13103)]CRS-1603:CSSD on node node2 shutdown by user.
2011-04-13 08:19:54.501
[ohasd(5058)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'node2'.
2011-04-13 08:19:57.723
[cssd(17068)]CRS-1713:CSSD daemon is started in clustered mode
2011-04-13 08:20:01.177
[ohasd(5058)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node2'.
2011-04-13 08:20:13.167
[cssd(17068)]CRS-1707:Lease acquisition for node node2 number 2 completed pay attention at the timestamp 08:10:07.143 & 08:19:49.386
The error in the oracssdagent_root.log is
2011-04-13 08:09:49.286: [CLSFRAME][3014212592] New Framework state: 2
2011-04-13 08:09:49.286: [CLSFRAME][3014212592] M2M is starting...
2011-04-13 08:09:49.288: [ CRSCOMM][3014212592] Ipc: Starting send thread
2011-04-13 08:09:49.288: [ CRSCOMM][1092061504] Ipc: sendWork thread started.
2011-04-13 08:09:49.289: [ CRSCOMM][1105643840] IpcC: IPC Client thread started listening
2011-04-13 08:09:49.289: [ CRSCOMM][1105643840] IpcC: Received member number of 10
2011-04-13 08:09:49.290: [CLSFRAME][3014212592] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:node2
2011-04-13 08:09:49.290: [CLSFRAME][3014212592] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:node2
2011-04-13 08:09:49.291: [CLSFRAME][3014212592] Tints initialized with nodeId: 0 procId: 10
2011-04-13 08:09:49.291: [CLSFRAME][3014212592] Starting thread model named: MultiThread
2011-04-13 08:09:49.292: [CLSFRAME][3014212592] Starting thread model named: TimerSharedTM
2011-04-13 08:09:49.293: [CLSFRAME][3014212592] New Framework state: 3
2011-04-13 08:09:49.293: [ AGFW][3014212592] Agent Framework started successfully
2011-04-13 08:09:49.293: [ AGFW][1116150080] {0:10:2} Agfw engine module has enabled...
2011-04-13 08:09:49.293: [CLSFRAME][1116150080] {0:10:2} Module Enabling is complete
2011-04-13 08:09:49.293: [CLSFRAME][1116150080] {0:10:2} New Framework state: 6
2011-04-13 08:09:49.294: [CLSFRAME][3014212592] M2M is now powered by a doWork() thread.
2011-04-13 08:09:49.294: [ AGFW][1116150080] {0:10:2} Agent is started with userid: root , expected user: root
2011-04-13 08:09:49.294: [ AGENT][1116150080] {0:10:2} Static Version 11.2.0.2.0
2011-04-13 08:09:49.294: [ AGFW][1116150080] {0:10:2} Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:11
2011-04-13 08:09:49.302: [ AGFW][1116150080] {0:10:2} Agent received the message: RESTYPE_ADD[ora.cssd.type] ID 8196:12358
2011-04-13 08:09:49.302: [ AGFW][1116150080] {0:10:2} Added new restype: ora.cssd.type
2011-04-13 08:09:49.303: [ AGFW][1116150080] {0:10:2} Agent sending last reply for: RESTYPE_ADD[ora.cssd.type] ID 8196:12358
2011-04-13 08:09:49.305: [ AGFW][1116150080] {0:10:2} Agent received the message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:12359
2011-04-13 08:09:49.305: [ AGFW][1116150080] {0:10:2} Added new resource: ora.cssd 1 1 to the agfw
2011-04-13 08:09:49.306: [ AGFW][1116150080] {0:10:2} Agent sending last reply for: RESOURCE_ADD[ora.cssd 1 1] ID 4356:12359
2011-04-13 08:09:49.308: [ AGFW][1116150080] {0:6:7} Agent received the message: RESOURCE_START[ora.cssd 1 1] ID 4098:12360
2011-04-13 08:09:49.308: [ AGFW][1116150080] {0:6:7} Preparing START command for: ora.cssd 1 1
2011-04-13 08:09:49.308: [ AGFW][1116150080] {0:6:7} ora.cssd 1 1 state changed from: UNKNOWN to: STARTING
2011-04-13 08:09:49.309: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_cssdstart: Start action called
2011-04-13 08:09:49.309: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_getattr: attr OMON_INITRATE, value 1000
2011-04-13 08:09:49.309: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_getattr: attr OMON_POLLRATE, value 500
2011-04-13 08:09:49.309: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_getattr: attr ORA_OPROCD_MODE, value
2011-04-13 08:09:49.310: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_getattr: attr PROCD_TIMEOUT, value 1000
2011-04-13 08:09:49.310: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_getattr: attr LOGGING_LEVEL, value 1
2011-04-13 08:09:49.310: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_cssdstart: loglevels CSSD=2,GIPCNM=2,GIPCGM=2,GIPCCM=2,CLSF=0,SKGFD=0,GPNP=1,OLR=0
2011-04-13 08:09:49.313: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_cssdstart: START action for resource /u01/oracle/installed/oracle_cluster-11.2.0.2-1/bin/ocssd: SUCCESS
2011-04-13 08:09:49.313: [ora.cssd][1114048832] {0:6:7} [start] clsncssd_waitomon: start waiting
2011-04-13 08:09:49.313: [ CSSCLNT][1098377536]clsssInitNative: Init for agent
2011-04-13 08:09:50.317: [ CSSCLNT][1098377536]clsssInitNative: Init for agent
2011-04-13 08:09:51.319: [ CSSCLNT][1098377536]clsssInitNative: Init for agent
2011-04-13 08:09:51.322: [ CSSCLNT][1098377536]clssnsqueryfatal: css is fatal = 0
2011-04-13 08:09:51.322: [ USRTHRD][1098377536] clsncssd_thrdspawn: spawn OPROCD succ
2011-04-13 08:09:51.322: [ USRTHRD][1098377536] clsncssd_thrdspawn: spawn POLLMSG succ
2011-04-13 08:09:51.323: [ USRTHRD][1099954496] clsnpollmsg_main: starting pollmsg thread
2011-04-13 08:09:51.323: [ USRTHRD][1107745088] clsnproc_main: timeout of procd cannot be 0, now we set to default 1000.
2011-04-13 08:09:51.323: [ USRTHRD][1117727040] clsnwork_main: starting worker thread
2011-04-13 08:09:51.323: [ USRTHRD][1098377536] clsncssd_thrdspawn: spawn WORKER succ
2011-04-13 08:09:51.323: [ USRTHRD][1107745088] clsnproc_main: starting oprocd
2011-04-13 08:09:51.323: [ USRTHRD][1098377536] clsncssd_thrdspawn: spawn KILL succ
2011-04-13 08:10:07.151: [ USRTHRD][1098377536] clsnomon_init: css init done, nodenum 2
2011-04-13 08:10:07.151: [ USRTHRD][1098377536] clsnomon_WaitToRegister: waiting for first reconfiguration and kgzf initialization
2011-04-13 08:19:49.385: [CLSFRAME][3014212592] TM [MultiThread] is changing desired thread # to 3. Current # is 2
2011-04-13 08:19:49.387: [ AGFW][1111947584] {0:6:7} Created alert : (:CRSAGF00113:) : Aborting the command: start for resource: ora.cssd 1 1
2011-04-13 08:19:49.387: [ora.cssd][1111947584] {0:6:7} [start] clsncssd_cssdabort: sending shutdown abort to CSS with new ctx
2011-04-13 08:19:49.387: [ CSSCLNT][1098377536]clsssRecvMsg: wrong type request (0) on 0xc9 ret 0
2011-04-13 08:19:49.387: [ CSSCLNT][1098377536]clssnskgzfdone: RPC failed rc 1
2011-04-13 08:19:49.387: [ USRTHRD][1098377536] clsnomon_WaitToRegister: exadata initialization completed with rc=1
2011-04-13 08:19:49.387: [ USRTHRD][1098377536] clsnomon_init: problems in the CSS to allow OMON registration 2
2011-04-13 08:19:49.387: [ USRTHRD][1098377536] clsnomon_cleanup: to exit status = 2
2011-04-13 08:19:49.387: [ USRTHRD][1098377536] clsnomon_cleanup: failure, sending shutdown immediate to CSS
2011-04-13 08:19:49.387: [ USRTHRD][1098377536] CHECK action is in progress, Rejecting the check action requested by entry point for ora.cssd
2011-04-13 08:19:49.426: [ AGFW][2008402928] Starting the agent: /u01/oracle/installed/oracle_cluster-11.2.0.2-1/log/node2/agent/ohasd/oracssdagent_root/
2011-04-13 08:19:49.426: [ AGENT][2008402928] Agent framework initialized, Process Id = 17013
2011-04-13 08:19:49.426: [ USRTHRD][2008402928] to enter agent main
2011-04-13 08:19:49.426: [ USRTHRD][2008402928] clsscssd_main: New soft limit for stack size is 1572864, hard limit is 4294967295
2011-04-13 08:19:49.434: [ USRTHRD][2008402928] clsncssd_main: setting priority to 4
2011-04-13 08:19:49.434: [ USRTHRD][2008402928] *** Agent Framework Started *** Do you have any idea why it took so long to bring all the components up & running?
Thanks a lot!!
GHi,
there is an internal timer for the clusterware ressources regarding restarting the ressources.
In case of a node eviction or clusterstack reboot the clusterware tries to startup again.
If the issue still persists, CRS will wait for some time to start the stack again. This "restart" try is based on a timer, which is set to 600 seconds (note this is not the ORA_CHECK_TIMEOUT) but the STARTUP_TIMEOUT.
Since a missing interconnect does have some implications (not only on the network but on the whole stack) it is expected, that the cluster does not start so fast automatically (because it still has the first start running.
There is even another "issue" connected to this - Oracle will only try several times (FAILURE_COUNT/FAILURE_THRESHOLD) to restart ressources. If he cannot restart cssd/crsd for several times, OCW will not try to startup automatically, but expects the administrator to solve the error and then startup again.
But actually this does make sense:
We have to give some time for an error to be resolved, before we start automatically. It does not matter if the restart of the node is delayed by this, because
=> If the error is fixed automatically, it will normally be fixed after a cluster/node reboot and hence cluster will come up
=> If the error is not fixed automatically, but manually, it can be expected that the administrator tells clusterware the issue is resolved. He does that by simply starting the stack (crsctl start crs)
=> If the error is fixed automaticall, but fixing took a while (lets say 15 minutes), it does not really matter if clusterware needs 10 more minutes to come up.
So what you see is expected, and wanted.
It would cost way too much to monitor all ressources regarding cluster problems and trigger a startup....
Sebastian -
Asm diskgroup does not exist or not mounted
Hi,
iam have configured a 2 node cluster and now when i rebooted and again started my rac database so all the applications are offline so i stop them with crs_stop and then restarted but my databases asm and instance service is not up so in that case i tried to connect to database or to instance but i was not able to so then i thought of connecting to asm instance and then the status is as follows
SQL> select name ,state ,mount_status from v$asm_disk ;
NAME STATE MOUNT_S
NORMAL CLOSED
NORMAL CLOSED
FG_0000 NORMAL CACHED
SQL> alter database open ;
alter database open
ERROR at line 1:
ORA-15000: command disallowed by current instance type
SQL> alter diskgroup all mount ;
alter diskgroup all mount
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15130: diskgroup "DG1" is being dismounted
ORA-15066: offlining disk "DG1_0001" may result in a data loss
SQL>
SQL>
so in that cse the contents are as follows
on Jul 11 21:58:46 2011
Dirty detach reconfiguration started (old inc 2, new inc 2)
List of nodes:
0 1
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 1 invalid = TRUE
0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon Jul 11 21:58:46 2011
freeing rdom 1
Mon Jul 11 21:58:47 2011
WARNING: dirty detached from domain 1
Mon Jul 11 21:58:47 2011
ERROR: diskgroup DG1 was not mounted
Mon Jul 11 21:58:47 2011
WARNING: PST-initiated MANDATORY DISMOUNT of group DG1 not performed - group not mounted
Mon Jul 11 21:58:47 2011
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_b000_18030.trc:
ORA-15001: diskgroup "DG1" does not exist or is not mounted
ORA-15001: diskgroup "DG1" does not exist or is not mounted
so iam not able to figure out please help me in solving this
Thanks in Advance
Regards
KavitaHi,
Sir i wait for many times but the major issue is iam facing that all aplications when i start my machine are offline or unknown........ then when i exec crsctl stop crs and crsctl start crs i wait a lot and then i exec crrs_stat t that time all applications are either unknown or offline............ then i stop forcily all applications which are unknown finally bring all to offline status and then restart but in this case i exec srvctl start nodeapps - n rac1 and nrac2 these things work but tried with database or service or instnace that ownt work it give me the following error
and then i sart instance or database through sqlplus and if i do any transcations then i get the follwoing error .........
PRKP-1001 : Error starting instance devdb1 on node rac1
CRS-0215: Could not start resource 'ora.devdb.devdb1.inst'
and now if iam trying to start my asm instance iam have already posted through what iam getting and this i have tried for 2 diffemt 2 node cluster but facing the same issue
2011-07-12 22:02:02.500: [ CRSRES][3772144560]0startRunnable: setting CLI values
2011-07-12 22:02:03.235: [ CRSRES][3772144560]0Attempting to start `ora.prod1.LISTENER_PROD1.lsnr` on member `prod1`
2011-07-12 22:02:06.197: [ CRSRES][3772144560]0Start of `ora.prod1.LISTENER_PROD1.lsnr` on member `prod1` succeeded.
Sir this from
(0x8364680) pid() proto(10:2:1:1)
[ CSSD]2011-07-12 22:46:09.669 [91069360] >TRACE: clssgmClientConnectMsg: Connect from con(0x8350880) proc(0x8364680) pid() proto(10:2:1:1)
[ CSSD]2011-07-12 22:47:03.826 [91069360] >TRACE: clssgmClientConnectMsg: Connect from con(0x8352a08) proc(0x8364680) pid() proto(10:2:1:1)
[ CSSD]2011-07-12 22:47:04.385 [91069360] >TRACE: clssgmClientConnectMsg: Connect from con(0x835bf18) proc(0x835ed88) pid() proto(10:2:1:1)
[ CSSD]2011-07-12 22:47:17.938 [91069360] >TRACE: clssgmClientConnectMsg: Connect from con(0x835bf18) proc(0x8364680) pid() proto(10:2:1:1)
[ CSSD]2011-07-12 22:48:27.641 [91069360] >TRACE: clssgmClientConnectMsg: Connect from con(0x8350880) proc(0x8364680) pid() proto(10:2:1:1)
Thanks in Advance
Regards
Kavita -
Asm 11.1.0.6 root.sh fails to start css
We have Oracle ASM and installing 11.1.0.6 on a new machine (eventually will apply 11.1.0.7 patchset). So we are on 11g R1 in production environments.
When installing Oracle ASM 11.1.0.6, the root.sh fails
I checked various metalink notes, all settings are OK. We have ASMLib and i have configured ASMLib with the new disks prior to the ASM installation
Just to rule out the rootcause is with ASMLib , I have also disabled it. Still same problem. I ran the usual localconfig delete and localconfig add too.
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.+
Oracle CSS service would not start as installed+
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started+
Finished product-specific root actions.+I applied the 11.1.0.7 patchset. Still same problem.
Strangely, there are no logfiles in $ASM_HOME/log/<hostname>/cssd/ - This directory is empty
The below are the final messages in root.sh
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started -
Not able to start the oracle database with ASM
HI all,
WE are using oracle10.2.0.1 on OEL5.
I have mounted ASM but when i try to start the oracle database, it is asking "db is already started shut it down first"
[root@localhost ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
[root@localhost ~]#
[root@localhost ~]# su - oracle
-bash-3.2$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Mon May 31 02:12:06 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> shutdown
ORA-01012: not logged on
SQL> startp mount
SP2-0734: unknown command beginning "startp mou..." - rest of line ignored.
SQL> conn / as sysdba
Connected.
SQL> shutdown
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup mount
ASM instance started
Total System Global Area 130023424 bytes
Fixed Size 2019032 bytes
Variable Size 102838568 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
-bash-3.2$ export ORACLE_SID=dbtest
-bash-3.2$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Mon May 31 02:13:30 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> exit
-bash-3.2$ cd /u01/app/oracle/admin/dbtest/pfile/
-bash-3.2$ ls -l
total 8
-rw-r----- 1 oracle oinstall 2442 May 27 23:07 init.ora.427201023913
-rw-r--r-- 1 oracle oinstall 1406 May 31 01:44 sqlnet.log
-bash-3.2$ pwd
/u01/app/oracle/admin/dbtest/pfile
-bash-3.2$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Mon May 31 02:14:00 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> exit
-bash-3.2$ sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Mon May 31 02:14:09 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
SQL> startup pfile="/u01/app/oracle/admin/dbtest/pfile/init.ora.427201023913';
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL>
-bash-3.2$ ps -ef|grep ora
root 5501 5483 0 00:32 ? 00:00:00 hald-addon-storage: polling /dev/hdc
root 6333 1 0 00:32 ? 00:00:00 /bin/su -l oracle -c sh -c 'cd /u01/app/oracle/product/10.1.0/db_1/log/localhost/cssd; ulimit -c unlimited; exec /u01/app/oracle/product/10.1.0/db_1/bin/ocssd '
oracle 6527 6333 0 00:33 ? 00:00:00 /u01/app/oracle/product/10.1.0/db_1/bin/ocssd.bin
oracle 6849 1 0 00:49 ? 00:00:00 /u01/app/oracle/product/10.1.0/db_1/bin/tnslsnr LISTENER -inherit
root 7637 7617 0 01:53 pts/2 00:00:00 su - oracle
oracle 7638 7637 0 01:53 pts/2 00:00:00 -bash
root 7751 6763 0 02:11 pts/1 00:00:00 su - oracle
oracle 7752 7751 0 02:11 pts/1 00:00:00 -bash
oracle 7773 1 0 02:12 ? 00:00:00 oracledbtest (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 7777 1 0 02:13 ? 00:00:00 asm_pmon_dbtest
oracle 7779 1 0 02:13 ? 00:00:00 asm_psp0_dbtest
oracle 7781 1 0 02:13 ? 00:00:00 asm_mman_dbtest
oracle 7783 1 0 02:13 ? 00:00:00 asm_dbw0_dbtest
oracle 7785 1 0 02:13 ? 00:00:00 asm_lgwr_dbtest
oracle 7787 1 0 02:13 ? 00:00:00 asm_ckpt_dbtest
oracle 7789 1 0 02:13 ? 00:00:00 asm_smon_dbtest
oracle 7791 1 0 02:13 ? 00:00:00 asm_rbal_dbtest
oracle 7793 1 0 02:13 ? 00:00:00 asm_gmon_dbtest
oracle 7805 7752 0 02:15 pts/1 00:00:00 ps -ef
oracle 7806 7752 0 02:15 pts/1 00:00:00 grep ora
-bash-3.2$
{code}
so anybody can help me how to start the oracle database....-bash-3.2$ sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Mon May 31 02:26:11 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
SQL> show parameter instance
NAME TYPE VALUE
active_instance_count integer
cluster_database_instances integer 1
instance_groups string
instance_name string dbtest
instance_number integer 0
instance_type string asm
open_links_per_instance integer 4
parallel_instance_group string
parallel_server_instances integer 1
SQL>
-bash-3.2$ sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Mon May 31 02:26:11 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
SQL> show paramete instance_name
SP2-0158: unknown SHOW option "paramete"
instance "local"
SP2-0158: unknown SHOW option "_name"
SQL> show parameter instance
NAME TYPE VALUE
active_instance_count integer
cluster_database_instances integer 1
instance_groups string
instance_name string dbtest
instance_number integer 0
instance_type string asm
open_links_per_instance integer 4
parallel_instance_group string
parallel_server_instances integer 1
SQL>
-bash-3.2$ ps -ef|grep pmon
oracle 7777 1 0 02:13 ? 00:00:00 asm_pmon_dbtest
oracle 7851 7752 0 02:32 pts/1 00:00:00 grep pmon
-bash-3.2$
-bash-3.2$ -
Installing ASM in it's own home (non-RAC environment)
I am installing ASM
When I run root.sh I get the following error:Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started.
The install is on Centos 5 (aka RHEL 5). I have set up a user and a group specifically for this install (user: oracleasm group: asmdba) all of which are the owners of the disks (dev/oracleasm/disks) and the home which i am installing it on. But cannot get past the CSS stack appears not be running.
Any assistance would be greatly appreciated...I have been all over OTN and metalink but no luck.
Regards,
MattCheck if daemon ocssd.bin is running by executing:
ps -ef|grep ocssd
If it not running then execute
/etc/init.d/init.cssd run
sleep 60
/etc/init.d/init.cssd start
sleep 120
/etc/init.d/dbora start
If that works you had problem because of Oracle bug and the solution in metalink note id 264235.1
You need to fix the /etc/inittab file moving the line that had init.cssd run to line before level 3 scripts executed. -
Found the errors in CSSD logs of RAC node
Found the below error in CSSD logs in One of RAC nodes from 5:15 to 5:18 PM, after this the error got disappeared. Could anyone please have an idea what could be the reason of this error.
Also, at that time we didn't find any errors in the alert log.
[ CSSD]2009-07-19 17:15:51.048 [3600] >TRACE: Authorization failed (112bd2a70), timed out, start 17:13:51.041, duration 120009
[ CSSD]2009-07-19 17:15:51.048 [3600] >TRACE: Authorization prepare time: 2 ms
[ CSSD]2009-07-19 17:15:51.233 [3086] >TRACE: clssgmClientConnectMsg: Connect from con(112b67930) proc(112b680b0) pid(1049540) proto(10:2:1:1)
[ CSSD]2009-07-19 17:15:51.268 [3600] >TRACE: Authorization failed (112bd4a10), timed out, start 17:13:51.268, duration 120003
[ CSSD]2009-07-19 17:15:51.268 [3600] >TRACE: Authorization prepare time: 3 ms
[ CSSD]2009-07-19 17:15:52.544 [3086] >TRACE: clssgmClientConnectMsg: Connect from con(112b67930) proc(112b680b0) pid(786918) proto(10:2:1:1)
[ CSSD]2009-07-19 17:15:53.297 [3600] >TRACE: Authorization failed (112c38af0), timed out, start 17:13:53.290, duration 120009
[ CSSD]2009-07-19 17:15:53.297 [3600] >TRACE: Authorization prepare time: 3 ms
[ CSSD]2009-07-19 17:15:53.317 [3600] >TRACE: Authorization failed (112d356f0), timed out, start 17:13:53.320, duration 120000
[ CSSD]2009-07-19 17:15:53.317 [3600] >TRACE: Authorization prepare time: 2 ms
[ CSSD]2009-07-19 17:16:02.342 [3086] >TRACE: clssgmClientConnectMsg: Connect from con(112b932b0) proc(112b67d10) pid(1336252) proto(10:2:1:1)
[ CSSD]2009-07-19 17:16:02.977 [3600] >TRACE: Authorization failed (112d04f70), timed out, start 17:14:02.978, duration 120001
[ CSSD]2009-07-19 17:16:02.977 [3600] >TRACE: Authorization prepare time: 2 ms
[ CSSD]2009-07-19 17:16:03.007 [3600] >TRACE: Authorization failed (112d38210), timed out, start 17:14:03.006, duration 120002
[ CSSD]2009-07-19 17:16:03.007 [3600] >TRACE: Authorization prepare time: 2 ms
[ CSSD]2009-07-19 17:16:10.447 [3600] >TRACE: Authorization failed (112bd7e30), timed out, start 17:14:10.441, duration 120007
[ CSSD]2009-07-19 17:16:10.447 [3600] >TRACE: Authorization prepare time: 2 ms
[ CSSD]2009-07-19 17:16:10.847 [3600] >TRACE: Authorization failed (112d3ee70), timed out, start 17:14:10.840, duration 120008
[ CSSD]2009-07-19 17:16:10.847 [3600] >TRACE: Authorization prepare time: 2 ms
Thanks,
MahiCheck the metalink note:
6996694-OCSSD.BIN CONSUMING 100% CPU AND ASM/DB HANGING -
Unable to create ASM instance in Sol 10 with oracle 10g
Hi
I am trying to create ASM instance in oracle 10g, getting an error will try to add localconfig add command
"bash-3.00# /export/home/oracle/oracle/product/10.2.0/db_1/bin/localconfig add reset
Failure at scls_scr_create with code 1
Internal Error Information:
Category: 1234
Operation: scls_scr_create
Location: mkdir
Other: Unable to make user dir
Dep: 2
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Adding to inittab
/etc/init.d/init.cssd: /var/opt/oracle/scls_scr/Sun/root/cssrun: cannot create
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started "
initcssd has been installed and unable to start also getting an error
# svcs -x svc:/system/initcssd:default
svc:/system/initcssd:default (system activity reporting package)
State: maintenance since Wed Nov 16 10:39:29 2011
Reason: Start method failed repeatedly, last exited with status 2.
See: http://sun.com/msg/SMF-8000-KS
See: sar(1M)
See: /var/svc/log/system-initcssd:default.log
Impact: This service is not running.
Can some please help me to create this instance, alos need a initcssd.zip file for 10gHi thanks,
I have passed more steps, CSS is started after changing the hostname, after that I created two drive and mounted properly
when I try to create a ASM disk its failing with following error and idea
SQL> CREATE DISKGROUP DB_DATA NORMAL REDUNDANCY
2 FAILGROUP controller1 DISK '/dev/dsk/c0d1s0'
3 FAILGROUP controller2 DISK '/dev/dsk/c1d1s0';
CREATE DISKGROUP DB_DATA NORMAL REDUNDANCY
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/dsk/c1d1s0' matches no disks
ORA-15025: could not open disk '/dev/dsk/c1d1s0'
ORA-15056: additional error message
Intel SVR4 UNIX Error: 13: Permission denied
Additional information: 42
Additional information: 134497888
Additional information: -809278080
ORA-15031: disk specification '/dev/dsk/c0d1s0' matches no disks
ORA-15025: could not open disk '/dev/dsk/c0d1s0'
ORA-27037: unable to obtain file status
Intel SVR4 UNIX Error: 25: Inappropriate ioctl for device
Additional information: 16
Additional information: 134497888
Additional information: -809278080 -
Hi,
I have noticed following 'strange' behaviour of Oracle Restart and ASM.
starting position:
-bash-3.2 $ crsctl status resource -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.DATA.dg
ONLINE ONLINE oracle-restart
ora.LISTENERASM.lsnr
ONLINE ONLINE oracle-restart
ora.asm
ONLINE ONLINE oracle-restart Started
Cluster Resources
ora.cssd
1 ONLINE ONLINE oracle-restart
ora.diskmon
1 ONLINE ONLINE oracle-restartstep 1:
-bash-3.2 $ srvctl stop asm
-bash-3.2 $ srvctl stop diskgroup -g data
-bash-3.2 $ srvctl disable diskgroup -g datastep 2:
via sqlplus start ASM instance
SQL> startup
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2212656 bytes
Variable Size 256552144 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL> select * from v$asm_diskgroup;
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE
ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
COMPATIBILITY
DATABASE_COMPATIBILITY V
1 DATA 512 4096
1048576 MOUNTED EXTERN 10236 10177 0
59 0 10177 0
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE
ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB
COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
COMPATIBILITY
DATABASE_COMPATIBILITY V
11.2.0.0.0
10.1.0.0.0 N
-bash-3.2 $ crsctl status resource -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.DATA.dg
OFFLINE OFFLINE oracle-restart <== funny !!!
ora.LISTENERASM.lsnr
ONLINE ONLINE oracle-restart
ora.asm
ONLINE ONLINE oracle-restart Started
Cluster Resources
ora.cssd
1 ONLINE ONLINE oracle-restart
ora.diskmon
1 ONLINE ONLINE oracle-restartIs this behaviour a 'feature' or bug?
Anyone had similar experience?
thanks,
goranHi,
asm resource is depending on diskgroup resource ... if diskgroup res. is not available, crsctl status shows offline, I would expect asm should be also shown as 'offline' (and brought offline) as they are dependent.
What is the point of managing resources via srvctl when it doesn't take care of dependencies? For me it's wrong.ora.asm : is ASM Instance
ora.*.dg : is Diskgroup
ora.*.dg is dependent of ora.asm, not to the contrary.
I can have more than one diskgroup and want only one diskgroup disabled, so I need the ASM Instance (ora.asm) online.
Important:
If you shut down the database with SQL*Plus, Oracle Restart does not interpret this as a database failure and does not attempt to restart the database.
Similarly, if you shut down the Oracle ASM instance with SQL*Plus or ASMCMD, Oracle Restart does not attempt to restart it.
An important difference between starting a component with SRVCTL and starting it with SQL*Plus (or another utility) is the following:
When you start a component with SRVCTL, any components on which this component depends are automatically started first, and in the proper order.
When you start a component with SQL*Plus (or another utility), other components in the dependency chain are not automatically started; you must ensure that any components on which this component depends are started.
Oracle Restart also manages the weak dependency between database instances and the Oracle Net listener (the listener): When a database instance is started, Oracle Restart attempts to start the listener. If the listener startup fails, then the database is still started. If the listener later fails, Oracle Restart does not shut down and restart any database instances.
It makes no sense Oracle Restart to shut down all environment (databases) because the listener down.
Regards,
Levi Pereira
Maybe you are looking for
-
Clearing of Balance in GR/IR clearing account due to date problem
>Hi to All, >The scenario is like this :- >The client is making back dated entries for quarter between 01.04.2008 to 30.06.2008 in SAP in July 2008 as the system is getting implemented now. He wants to regularise all the transactions in SAP from 01/0
-
Tds line item not simulating while posting as reference with FBR2
Dear All When I am posting an Invoice using previous document as reference with Tcode FBR2 WTax line Item not simulating, but while I am posting without reference i.e. with Tcode FB60 it is simulating, this problem i am facing in production server
-
Is it possible to create reports in 1 month without any experience?
I'm on a project where we need to make some custom reports in the span of about a month. I'm trying to figure out what is feasible for a person, with no reporting background but plenty of SQL experience,to handle in that time frame. The format of all
-
I can't log into FaceTime any suggestions?
Having trouble logging into facetime. I get a message that I cannot connect with the FaceTime server. I can connect with everything else .
-
Find thread that opened a session in AIX
Hello. As stated in "Determining which Oracle database session is being used by a specific Siebel user session [ID 863771.1]": On the Oracle database side, you can query the V$SESSION table (make sure you are logged in as the tableowner) using the fo