Fresh cluster inst., servers reboot, cannot restart clusters or ASM

Question

Hello all,
I just installed 11Gr2 cluster over 5 nodes. I used ASM in the installer, to hold the voting disk, etc for it.
I installed the RDBMS binairies successfully across all nodes. NO INSTANCES YET.
A few days went by....
I was getting ready to do post installing patches...and found things looking strange. I found the (working on node1), the clustering systems was not running.
I looked, and the servers (all 5 of them) for some reason had rebooted since install.
I tried starting the cluster:
crsctl start cluster -all.
Took while to return, and then errored with a timeout msg.
I checked to see if it was up:
./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
It then dawned on me...maybe ASM wasn't up either?
Nope...not running.
I tried to start it locally on node 1
I set the SID, and tried using sqlplus
I got:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/init+ASM1.ora'
I looked...nothing in that directory at all but a simple init.ora file.
I tried shutting down the cluster with
crsctl stop cluster -all
I got a ton of messages for each node like:
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'node1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
I'm trying to get through to Oracle support, but they're running slow.
Any ideas here?
I used OUI to create ASM for the cluster...why would it not put an init file there to point to the spfile in ASM?
I'm guessing this is the reason the nodes couldn't talk or sync. Trouble is...how do I start ASM without an init file? I seem to recall there might be a way to create a file to point to the ASM for the spfile, but I'm new to this too...and not sure where to point or the syntax to use.
Will have starting cluster up with no ASM have done any damage...if so, how to fix it?
As you can tell, learning about clusters/RAC and ASM....and I'm not finding good reference materials on troubleshooting. Heck, the install docs are bad enough....
Thank you in advance for any advice or links...
cayenne
ps. this is on RHEL5
Edited by: cayenne on Aug 10, 2010 12:33 PM

Answer 1

ssolbach wrote:
Hi,
Try doing crsctl stop crs -f 2 times. Sometimes this is needed.
crsctl check crs should report you that nothing is running anymore.
If it is... then only way I can think of is disable the automatic startup of crs
crsctl disable crs
and restart the node... This will definitely bring everything down (and does not start it up after restarting).
You can then start the crs stack with crsctl start crs.
Furthermore don't get confused with oracleasm.
Oracleasm is for asmlib. Which may be used in preparating the storage for ASM but has nothing to do later with if ASM is running or not.
To check the status of ASM first see if clusterstack was started successfully (you need to wait a little till it is started):
crsctl check crs.
Everything should be online... if it is, you can do a crsctl stat res -t which will show you all ressources including if ASM is running.
If there has been a problem starting up the stack (crsctl check crs) then we have to find out why.
Check $CRS_HOME/log/<hostname>/alert*.log for error messages.
If something indicates a problem with ASM do a:
adrci
show alert
and choose the ASM alert.log.
Search for error messages.
GL.
SebastianThank you, I'd not known of adrci before!! Once I set the ORACLE_HOME, it came right up!
Ok, it does look like ASM problem...and clustering can't find the voting disc.
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_25186.trc:
ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:sendmsg failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgxpsnd1
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_25186.trc:
ORA-27508: IPC error sending a message
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_o000_22504.trc (incident=4801):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 192.168.100.1 not found. Check output from ifconfig command
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_4801/+ASM1_o000_22504_i4801.trc
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_gmon_25212.trc:
ORA-29746: Cluster Synchronization Service is being shut down.
ORA-29702: error occurred in Cluster Group Service operation
GMON (ospid: 25212): terminating the instance due to error 29746
opidrv aborting process O000 ospid (22504) as a result of ORA-603
Ok, I've been going through the alert log, and the .trc files it indicates. I started by checking the private address 192.168.100.1...it seems to be up with ifconfig..and pingable.
I'm looking through errors...the one : ORA-27300 returning a value 22 got a hit on Oracle Support...but wasn't the same errors as I got.
Whew...still plowing through all the files and logs...not seeing anything out there so far that matches my problem...dunno what could have caused this to just all go BANG. I mean...no one but me using the machines, no databases on them yet...all that was installed was clustering with its voting disk and (ocr?) on a single ASM disk group....and RDBMS binaries installed across all 5 nodes.
Please let me know if you see anything that stands out here...I'm still searching myself...
Again, thanks for the adrcpi hint!!
cayenne