Rac node crash

Question

Hi,
I am using 2 node rac enviroment.I am doing some tests for availability.When i poweroff one node of cluster the other node of cluster reboot itself. The voting disk is online. I think this is caused from network heartbeat of private network. So i wonder when one node of cluster goes down , does the other node have to reboot itself or not ?
Thanks.

Answer 1

Hi,
This is 10gr1 rac.I could not find alertlog.But taken log from ocssd logs from both nodes.I have rebooted node2 and node1 reboot itself.
Node2
CLSS-3001: local node number 2, master node number 1
2012-05-22 08:45:57.247 [773] >TRACE:   clssgmClientConnectMsg: Connect from con(111cdebb0), proc(111ce38b0)  pid()
2012-05-22 08:45:57.247 [773] >TRACE:   clssgmClientConnectMsg: Connect from con(111ce1650), proc(111ce3b70)  pid()
2012-05-22 08:47:20.865 [773] >TRACE:   clsc_receive: (111ce0e10) Connection failed, transport error (507, 0, 0)
2012-05-22 08:47:20.865 [773] >TRACE:   clscreceive: (111ce1650) Physical connection (111ce0e10) not active, rc 11
2012-05-22 08:47:20.865 [773] >TRACE:   clscreceive: (111ce3fb0) Physical connection (111ce0e10) not active, rc 11
2012-05-22 08:47:20.865 [773] >TRACE:   clssgmDeleteClientListener: cleanup for proc(111ce3b70) con(111ce1650) pid()
2012-05-22 08:47:20.879 [1287] >TRACE:   clscsendx: (111ce3fb0) Connection not active
2012-05-22 08:47:20.934 [773] >TRACE:   clsc_receive: (111cde370) Connection failed, transport error (507, 0, 0)
2012-05-22 08:47:20.934 [773] >TRACE:   clscreceive: (111cdebb0) Physical connection (111cde370) not active, rc 11
2012-05-22 08:47:20.934 [773] >TRACE:   clssgmDeleteClientListener: cleanup for proc(111ce38b0) con(111cdebb0) pid()
2012-05-22 08:47:51.100 [773] >TRACE:   clssgmClientConnectMsg: Connect from con(111cdebb0), proc(111ce0e10)  pid()
Node1
CLSS-3001: local node number 1, master node number 1
2012-05-22 08:47:51.112 [516] >TRACE:   clsc_receive: (111ce6c50) Connection failed, transport error (507, 0, 0)
2012-05-22 08:47:51.112 [516] >WARNING: clssnmeventhndlr: Receive failure with node 2, rc=11
2012-05-22 08:47:51.112 [1287] >TRACE:   clsc_receive: (111bd8ab0) Connection failed, transport error (507, 0, 0)
2012-05-22 08:47:51.157 [1544] >WARNING: clssnmPollingThread: Eviction started for node 2, flags 0x0001, state 3, wt4c 0
2012-05-22 08:47:56.157 [1544] >TRACE:   clssnmDoSyncUpdate: Initiating sync 3
2012-05-22 08:47:56.157 [516] >TRACE:   clssnmHandleSync: Acknowledging sync: src[1] seq[9] sync[3]
2012-05-22 08:47:56.203 [1] >USER:    NMEVENT_SUSPEND [00][00][00][02]
2012-05-22 08:48:00.165 [1544] >TRACE:   clssnmEvict: Evicting node 2, birth 2, death 0, killme 1
2012-05-22 08:48:00.165 [1544] >TRACE:   clssnmWaitOnEvictions: Waiting for node 2 to die, missed HB 0 of 22503
2012-05-22 08:48:01.165 [1544] >TRACE:   clssnmWaitOnEvictions: Waiting for node 2 to die, missed HB 1 of 22503
2012-05-22 08:48:02.166 [1544] >TRACE:   clssnmWaitOnEvictions: Waiting for node 2 to die, missed HB 2 of 22503While node2 rebooting node1 has begun to reboot . Below log is OS log from node1
LABEL:          CORE_DUMP
IDENTIFIER:     C69F5C9B
Date/Time:       Tue May 22 08:48:05 2012
Sequence Number: 26098
Machine Id:      00C2F7704C00
Node Id:         kopstest1
Class:           S
Type:            PERM
Resource Name:   SYSPROC
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SOFTWARE PROGRAM
User Causes
USER GENERATED SIGNAL
        Recommended Actions
        CORRECT THEN RETRY
Failure Causes
SOFTWARE PROGRAM
        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
SIGNAL NUMBER
           6
USER'S PROCESS ID:
                909402
FILE SYSTEM SERIAL NUMBER
          12
INODE NUMBER
      109027
CORE FILE NAME
/oracle/product/10.1.0/crs/css/init/core
PROGRAM NAME
ocssd.bin
STACK EXECUTION DISABLED
           0
COME FROM ADDRESS REGISTER
PROCESSOR ID
  hw_fru_id: N/A
  hw_cpu_id: N/A
ADDITIONAL INFORMATION
pthread_k 88
Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/ocssd.bin SIG/6 FLDS/pthread_k VALU/88Thanks.
Edited by: ecer on May 21, 2012 11:11 PM