Oracle RAC 10.2G reboots node every 45 minutes

Question

Hello:
- We have installed Oracle RAC 10.2G for Solaris X86 ( 64 bit ).
- On one node, there are no issues. But the other node ( I think )
is being rebooted by CRS every 45 minutes or so.
- Is this issue caused by some misconfiguration I did during the install ?
- Or is there a patch available to fix this ?
- Has anyone else encountered this problem ?
Thanks
jlem

Answer 1

Hello:
- I re-installed Oracle RAC. The nodes were only rebooted once so far.
So, the second install may be ok. If not, I have provided answers to the first email reply.
- Any help given is most welcome. In meantime, I will continue searching the oracle forums
for solutions.
- My environment is:
- both nodes are running under vmware ESX server version 3.0.1
- the shared storage for OCR and Voting Disk is a raw shared device under vmware
- both nodes are using Solaris X86 5.10 update 5
- Oracle version is: 10.2.0.3 ( patched from version 10.2.0.1 )
- My public network configuration is:
node 1:
e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.20.1.74 netmask ffff0000 broadcast 10.20.255.255
ether 0:c:29:3a:45:a9
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.20.1.77 netmask ffff0000 broadcast 10.20.255.255
node 2:
e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.20.1.75 netmask ffff0000 broadcast 10.20.255.255
ether 0:c:29:2b:db:90
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.20.1.78 netmask ffff0000 broadcast 10.20.255.255
- My private network configuration is:
node 1:
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.0.1 netmask ffffff00 broadcast 192.168.0.255
ether 0:c:29:3a:45:b3
node 2:
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.0.2 netmask ffffff00 broadcast 192.168.0.255
ether 0:c:29:2b:db:9a
- My storage solution is:
- 3 virtual shared SCSI hard disks ( each 500 MB in size )
- My log files are:
- /var/adm/messages
- doesn't report much only the following:
Nov 12 10:57:05 saucer nfs4cbd[328]: [ID 867284 daemon.notice] nfsv4 cannot determine local hostname binding for transport
tcp6 - delegations will not be available on this transport
Nov 12 10:57:21 saucer savecore: [ID 570001 auth.error] reboot after panic: forced crash dump initiated at user requestNov 12 10:57:21 saucer savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/saucer/*.2Nov 12 10:57:41 saucer root: [ID 702911 user.error] Oracle Cluster Ready Services disabled by administrator.Nov 12 10:57:54 saucer rootnex: [ID 349649 kern.info] xsvc0 at rootNov 12 10:57:54 saucer genunix: [ID 936769 kern.info] xsvc0 is /xsvc
- ocssd.log file for node1 indicates that node2 was evicted for impeding a reconfig. Details are:
[    CSSD]2008-11-12 10:55:43.700 [15] >TRACE: clssnmPollingThread: node saucer (2) is impending reconfig
[    CSSD]2008-11-12 10:55:43.700 [15] >WARNING: clssnmPollingThread: node saucer (2) at 90% heartbeat fatal, eviction in 0
.973 seconds
[    CSSD]2008-11-12 10:55:44.679 [15] >TRACE: clssnmPollingThread: node saucer (2) is impending reconfig
[    CSSD]2008-11-12 10:55:44.679 [15] >TRACE: clssnmPollingThread: Eviction started for node saucer (2), flags 0x000d, s
tate 3, wt4c 0
[    CSSD]2008-11-12 10:55:44.690 [17] >TRACE: clssnmDoSyncUpdate: Initiating sync 3
[    CSSD]2008-11-12 10:55:44.690 [17] >TRACE: clssnmDoSyncUpdate: diskTimeout set to (27000)ms
[    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSetupAckWait: Ack message type (11)
[    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
[    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSetupAckWait: node(2) is ALIVE
[    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSendSync: syncSeqNo(3)
- node2 ocssd.log does not indicate the problem. See below for details:
[    CSSD]2008-11-12 10:52:34.731 [11] >TRACE: clssgmClientConnectMsg: Connect from con(da8410) proc(dab900) pid() proto(
10:2:1:1)
[    CSSD]2008-11-12 10:53:37.305 [11] >TRACE: clssgmClientConnectMsg: Connect from con(da8410) proc(dab900) pid() proto(
10:2:1:1)
[    CSSD]2008-11-12 10:54:40.515 [11] >TRACE: clssgmClientConnectMsg: Connect from con(da8410) proc(dab900) pid() proto(
10:2:1:1)
[    CSSD]2008-11-12 11:18:09.997 >USER: Oracle Database 10g CSS Release 10.2.0.3.0 Production Copyright 1996, 2004 Orac
le. All rights reserved.
[    CSSD]2008-11-12 11:18:09.997 >USER: CSS daemon log for node saucer, number 2, in cluster crs
[  clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=saucerDBG_CSSD))
[    CSSD]2008-11-12 11:18:10.016 [1] >TRACE: clssscmain: local-only set to false
[    CSSD]2008-11-12 11:18:10.031 [1] >TRACE: clssnmReadNodeInfo: added node 1 (flying) to cluster
[    CSSD]2008-11-12 11:18:10.042 [1] >TRACE: clssnmReadNodeInfo: added node 2 (saucer) to cluster
[    CSSD]2008-11-12 11:18:10.057 [5] >TRACE: clssnm_skgxnmon: skgxn init failed
[    CSSD]2008-11-12 11:18:10.057 [1] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
- ORACLE VERIFY: cluvfy was run on node2 resulting with the following:
bash-3.00$ ./cluvfy comp ocr -n all -verbose
Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
Data integrity check for OCR passed.
OCR integrity check passed.
Verification of OCR integrity was successful.
bash-3.00$
Thanks
jlem