Link down on Oracle 10R2 - RAC

Question

Dear All..
This is my first post, I have question related with RAC environment.
I setting up RAC on spesifications bellow:
- 3 HP server using intel xeon and 4G RAM each.
- Hitachi SAN
- Linux Centos 4.2
- Oracle 10g Release 2
- ASM 2 and OCFS 2
In installation steps, I found no problems. And the IP configuration used are:
# Public Network - (eth0)
172.20.141.1 linux1
172.20.141.2 linux2
172.20.141.3 linux3
# Private Interconnect - (eth1)
192.168.0.1 int-linux1
192.168.0.2 int-linux2
192.168.0.3 int-linux3
# Public Virtual IP (VIP) addresses for - (eth0)
172.20.142.1 vip-linux1
172.20.142.2 vip-linux2
172.20.142.3 vip-linux3
==========================================
Hangcheck 30 seconds, margin is 180 seconds
==========================================
[root@linux2 ~]# grep Hangcheck /var/log/messages |
tail -5
Apr 24 13:09:59 linux2 kernel: Hangcheck: starting
hangcheck timer 0.5.0 (tick is 30 seconds, margin is
180 seconds).
Apr 25 13:24:42 linux2 kernel: Hangcheck: starting
hangcheck timer 0.5.0 (tick is 30 seconds, margin is
180 seconds).
==========================================
setup O2CB_HEARTBEAT_THRESHOLD=301 pada file
/etc/sysconfig/o2cb
==========================================
But when I am trying to test the server using very hard
environment test (transactions and reporting using application
build by Delphi), just likes production ones, I found an error
occurred, eth0 : link down. The times it happen was random
(1 month 3 times - averages values).
And this is the error log.
==========================================
linux2 error log
==========================================
Apr 22 07:55:01 linux2 crond(pam_unix)[4288]: session
closed for user root
Apr 22 07:55:18 linux2 kernel: eth0: link down
Apr 22 07:55:19 linux2 kernel: eth0: link up, 100Mbps,
full-duplex, lpa 0x45E1
Apr 22 07:55:24 linux2 kernel:
(0,0):o2net_idle_timer:1330 connection to node li
nux3 num 1 at 172.20.141.3:7777 has been idle for 10
seconds, shutting it down.
Apr 22 07:55:24 linux2 kernel:
(0,0):o2net_idle_timer:1341 here are some times t
hat might help debug the situation: (tmr
1145667314.798878 now 1145667324.796791
dr 1145667314.798856 adv
1145667314.798892:1145667314.798893 func (c9943f19:505
) 1145429443.839194:1145429443.839197)
Apr 22 07:55:24 linux2 kernel:
(2248,1):o2net_set_nn_state:421 no longer connect
ed to node linux3 at 172.20.141.3:7777
Apr 22 07:55:26 linux2 kernel:
(0,0):o2net_idle_timer:1330 connection to node li
nux1 num 2 at 172.20.141.1:7777 has been idle for 10
seconds, shutting it down.
Apr 22 07:55:26 linux2 kernel:
(0,0):o2net_idle_timer:1341 here are some times t
hat might help debug the situation: (tmr
1145667316.353964 now 1145667326.352424
dr 1145667316.353945 adv
1145667316.353965:1145667316.353966 func (c9943f19:505
) 1145429355.325673:1145429355.325679)
Apr 22 07:55:26 linux2 kernel:
(2248,1):o2net_set_nn_state:421 no longer connect
ed to node linux1 at 172.20.141.1:7777
============================================
linux1 error log
============================================
Apr 22 07:55:02 linux1 crond(pam_unix)[10395]: session
closed for user root
Apr 22 07:57:58 linux1 kernel:
(2319,0):o2net_set_nn_state:421 no longer connect
ed to node linux2 at 172.20.141.2:7777
===========================================
linux3 error log
===========================================
Apr 22 07:55:03 linux3 crond(pam_unix)[14329]: session
closed for user root
Apr 22 07:56:50 linux3 kernel:
(2512,0):o2net_set_nn_state:421 no longer connect
ed to node linux2 at 172.20.141.2:7777
Any body have an advice or solution to overcome this problem.
Thanks for helping.
Best Regards,
Car Elcaro.

Answer 1

$cat opmn/conf/ons.config
localport=6101
remoteport=6200
loglevel=9
useocr=on
2007-02-09 13:43:16.441: [    RACG][1] [15217][1][ora.cgwdb02.ons]: Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = cgwdb01, port = 6200}
Adding remote host cgwdb01:6200
onscfg[1]
{node = cgwdb02, port = 6200}
Adding remote host cgwdb02:6200
ons is not running ...
2007-02-09 13:43:16.454: [    RACG][1] [15217][1][ora.cgwdb02.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/oracle/product/10.2.0/crs
2007-02-09 13:43:16.454: [    RACG][1] [15217][1][ora.cgwdb02.ons]: clsrcexecut: cmd = /u01/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /u01/oracle/product/10.2.0/crs/bin/onsctl ping
2007-02-09 13:43:16.454: [    RACG][1] [15217][1][ora.cgwdb02.ons]: clsrcexecut: rc = 1, time = 0.531s
2007-02-09 13:43:16.454: [    RACG][1] [15217][1][ora.cgwdb02.ons]: end for resource = ora.cgwdb02.ons, action = check, status = 1, time = 0.743s
2007-02-09 13:43:17.421: [    RACG][1] [15239][1][ora.cgwdb02.ons]: onsctl: shutting down ons daemon ...
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = cgwdb01, port = 6200}
Adding remote host cgwdb01:6200
onscfg[1]
{node = cgwdb02, port = 6200}
Adding remote host cgwdb02:6200