RAC Linux DB Servers Shutdown When Patching from 11.1.0.6 to 11.1.0.7

Question

Hello.
We have been successfully running a RAC 11.1.0.6.0 64-bit DB on Linux Redhat DB servers. We have had this DB running RHEL version 4 at kernel version 2.6.9-78.0.1.ELsmp. We also had successfully installed the January 2009 family CPU patch. A few months ago we began building a new 64-bit 11.1.0.6.0 RAC cluster set. This new cluster was on a different OS (a patch level up - kernel version is 2.6.9.78.ELsmp (Update 7). To our knowledge, no other OS/HW differences existed between the older, first RAC cluster and the new (2nd) one.
After the SAs handed over the new RAC DB servers to us, we installed the 11.1.0.6.0 clusterware without problem. We then began to install the clusterware 11.1.0.7.0 patch. Here is the problem. When we patch the clusterware up to 11.1.0.7.0, the last thing we are supposed to do is run the root shell script. When we run this script, it starts and never retunrs. In fact, by initiating this script, the node on which it is running actually crashes/shuts down. It does not reboot. It shuts down and we actually have to literlaly go to that server and push the on-off button. We can repeat this on any of the nodes and all of the machines respond the same. We have repeated this cluster build three different times. The first time, the root script finished, but each of the nodes later began shutting down selectively afterwards and we'd have to restart them. On the 2nd and 3rd attempts, the root script would not finish and the node would crash.
Oracle support hinted that we needed to go to the install glibc-2.3.4-2.40 (or above) or upgrade to EL4u7. Well, we are at EL4u7 now and the version of glibc is glibc-2.3.4-2.41. We look fine on that front. They also indicated that we need to install both the 32-bit and 64-bit versions of glibc. When we run "rpm -qa | grep -i glibc", we see:
glibc-2.3.4-2.41:
glibc-devel-2.3.4-2.41
glibc-devel-2.3.4-2.41
glibc-2.3.4-2.41:
We wonder as to why we see TWO (2) entries repeated. Does anyone know why? Also, how can one tell if both the 32 and 64-bit versions of glibc exist on our 64-bit machines? Does the fact that the above duplicate entries imply this?
MOE IMPORTANTLY, has anyone experienced this node/DB server crash when patching up from 11.1.0.6.0 to 11.1.0.7.0? We are out of answers on or end. We cannot even look at the system logs or the ocssd logs because the servers crash immediately upon restarting them.
Thank you.
Matt
Matt

Answer 1

Matt,
When I had this issue and the first node was going in to reboot loop, I remember I saw a message related to 'tick' of some kind. Based on that I thought the cluster is rebooting because of quorum issue with vote disk. So I thought if clusterware don't find vote disk then it will not start and no possibility of quorum issue that was leading to reboot. So I renamed the vote disk and OCR (which was not necessary though) and uninstalled and installed clusterware again.
In /etc/inittab OCCSD daemon is started at run level 3,5 with fatal option. That means if Cluster Synchronization Service (OCSSD daemon) gets terminated it reboot the server. If I had option of starting server to init 1 level, I would open /etc/inittab and comment out following lines and then start server in init 3 level and uninstall clusterware and install again.
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
Edited by: jatin1 on Apr 27, 2009 11:30 PM