Could Not start CRS or HAS in RAC 11.2.0.1

Question

Dear Legends,
We have the following environment in SOLARIS 2 Node Database. Not sure why the 2nd node went down today morning but 1st Node is still Up and Running.
My Tries so for
1. Changed the following line as per Doc 1368382.1 from
   h1:3:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
To
   h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
2. Provided a Restart of Entire Linux Box
3. Tried issuing the following command ./crsctl start crs but Error out as follows
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
4. Tried checking in Database, ASM alert logs nothing related with OHASD.
5. Also checked in ohasd.log and ohasdOUT.log not able to find any thing.
Please help me. This is our Production Environment.
Thanks,
Karthik

Answer 1

Karthik,
I encountered the same error few days back and below is what I did to resolve the issue as mentioned in metalink 1368382.1. Did you see anything in the crs logs?
did you follow all the steps in the solution section below like killiing remaining rc3 scripts?
checks :
1. Command '$GRID_HOME/bin/crsctl check crs' returns error:
     CRS-4639: Could not contact Oracle High Availability Services
2. Command 'ps -ef | grep init' does not show a line similar to:
     root 4878 1 0 Sep12 ? 00:00:02 /bin/sh /etc/init.d/init.ohasd run
3. Command 'ps -ef | grep d.bin' does not show a line similar to:
     root 21350 1 6 22:24 ? 00:00:01 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
    Or it may only show "ohasd.bin reboot" process without any other processes
4. ohasd.log report:
       2013-11-04 09:09:15.541: [ default][2609911536] Created alert : (:OHAS00117:) :  TIMED OUT WAITING FOR OHASD MONITOR
5. ohasOUT.log report:
       2013-11-04 08:59:14
       Changing directory to /u01/app/11.2.0/grid/log/lc1n1/ohasd
       OHASD starting
       Timed out waiting for init.ohasd script to start; posting an alert
Solutions:
1. Add the following line to /etc/inittab
    h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
   and then run "init q" as the root user.
2. Run command 'ps -ef | grep rc' and kill any remaining rc3 scripts that appear to be stuck.
3. Remove the bad entry before init.ohasd. Consult with OS vendor if "init q" does not spawn "init.ohasd run" process. As a workaround,
   start the init.ohasd manually, eg: as root user, run "/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null &"
4. Enable CRS autostart:
   # crsctl enable crs
   # crsctl start crs
5. Restore OLR from backup, as root user:
   # touch $GRID_HOME/cdata/<node>.olr
  # chown root:oinstall $GRID_HOME/cdata/<node>.olr
  # ocrconfig -local -restore$GRID_HOME/cdata/<node>/backup_<date>_<num>.olr
  # crsctl start crs
If OLR backup does not exist for any reason, perform deconfig and rerun root.sh is required to recreate OLR, as root user:
   # $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
   # $GRID_HOME/root.sh
6. If above does not help, check OS messages for ohasd.bin logger message and manually execute crswrapexece.pl command mentioned in the OS message with LD_LIBRARY_PATH set to <GRID_HOME/lib to continue debug.