Solaris RAC nodes re-booting

Question

I have a pre-production 2-node cluster running on Solaris 10, Oracle 10.2.0.3 with the Oracle CRS, and using a NetApp filer as the shared storage.
I also have a separate Solaris server running Grid Control 10.2.0.3, with the repository as one of the databases on the RAC (don't know if this is relevant to my problem).
Periodically both RAC nodes reboot, with no trace of why (the GC server is fine). There is nothing logged in the Solaris logs (messages file), CRS logs, Oracle logs or the NetApp logs.
All that is shown is the relevant service starting up following the shutdown.
Has anyone any experience of this, or any thoughts on which component may cause such an issue?
Thanks in advance
Bob

Answer 1

What type of Sun hardware are you using?
Below is the Action Plan Oracle support sent me on my SR on this issue, not sure if any of this was provided to you or would be of help.
ACTION PLAN
============
1. there is nothing on the files at all that sheds any light on the issue
agian 3 sperate sets of clusters all losing all nodes at the same tiem is a very strange occurance. Please be sure to have the admin look for
anything in common wiht all custers.
2. advice placing oswatcher on the systems Note.301137.1 Ext/Pub OS Watcher User Guide
if we should have another occurances we will want the oswatcher logs for 1 hr before issue thru issue
also see if the unix admin perhaps has any os stats from this occurance
3. advice settign ntpd to run with -x option I do see that you are having negative time changes
at times
-x will give us a skew rather then an abbrupt time change
4. advice setting this when you can
Please do the following
set the diagwait parameter:
crsctl set css diagwait N [-force]
Where N is the number of seconds to wait for a filesystem sync to
complete (after this wait the node will reboot regardless of whether the
sync has completed). This change must be made with the clusterware
down, which will require the '-force', or with the stack up on just 1
node, after which the stack on that node must be restarted before the
stack starts up on any of the other nodes.
N should be set to 25 (25 seconds)
5. advice that you have with pcw mlr#6 Patch 5980915 on the systems as well
but I do not believe that this was an oracle bug the reason for placing the patch on is for advanced diagnostics that is in that patchset
6. the two issues sun is workking on
Sun is working to resolve a time skew issue and a Solaris 10 kernel SIGALRM Sun#6292092 in addition to Sun#6595936.
7. we do have a diagnostic oprocd that soem sites have used but on thier test systems. It stops reboots adn dumps information but I have
been hesitant to place it on production boxes if you continue to have issues we may consider download the oprocd_skewfix_noreboot fro
m Bug 6279879 but at this time I do not belvve that is warrented