CRS not starting after Oracle 11g Grid was succesuflly installed

Question

Hi:
Have 2 node RAC. My Shared storage is on openfiler.
Oracle 11g Grid was successfully installed and all the services was working fine. When i restarted the system I found CRS was not coming up. Remaining processes were up and running.
Let me explain what I did: (O/s RHEL 5.4 & oracle 11g R2 grid)
I have a partition in shared disk /dev/sdc1 (I have other partitions, but thought of using only one for OCR and voting disk). (did not define it as raw partition as I used to do in 10g RAC).
While installing Oracle grid and when it prompted to enter the Diskgroup for OCR and Voting disk I gave a name DGDATA and choose only one (external redundancy) disk “/dev/sdc1″ (it does not ask for one for OCR and one for Voting as it used to do in 10g clusterware installation). As per my assumption both OCR and Voting disk were created within /dev/sdc1 (after installation when I queried I got the same info)
The Oracle grid installation has gone through successfully. when I restart the system I find CRS not starting.
The log says: “Error PROC:26: Error while accessing the physical storage ASM……..
ORA-01034: oracle not available...
Could not init OCR, code:26….
Linux permission denied".
For your information….ASM instance is up (check in both the nodes) and I find the diskgroup mounted.
Can anyone help on this.
Also let me know how to uninstall Oracle 11g Grid software and configuration. If I have to reinstall the Oracle 11g grid software and config once again is it possible without reinstalling RHEL. Let me know the steps or any link please. don't want to get into a mess again.
Thanks in advance.
Regards
Dinesh

Answer 1

Jorg, I even tried to nomount the DB instance using initDBRAC.ora pfile. Then tried to create spfile still got the same error.
The second trace file which I have pasted looks like some memory dump.
ASM_DISKSTRING= /dev/sdc*
***********************************************************************1st trace file
Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/crs/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_dbrac)(ENVS='ORACLE_HOME=/crs/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@skyit6)(HOST=skyit6)(USER=oracle))))
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 01-JUN-2011 01:27:07
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 01-JUN-2011 01:27:07
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
TNS-00517: Lost contact
nt secondary err code: 32
nt OS err code: 0
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/crs/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_dbrac)(ENVS='ORACLE_HOME=/crs/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
Wed Jun 01 01:27:07 2011
ERROR: slave communication error with ASM; terminating process 16772
Errors in file /oraeng/app/oracle/product/diag/rdbms/dbrac/DBRAC/trace/DBRAC_ora_16772.trc:
----======================2nd trace file
Trace file /oraeng/app/oracle/product/diag/rdbms/dbrac/DBRAC/trace/DBRAC_ora_16772.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters option
ORACLE_HOME = /oraeng/app/oracle/product/11.2.0
System name: Linux
Node name: skyit6
Release: 2.6.18-164.el5
Version: #1 SMP Tue Aug 18 15:51:54 EDT 2009
Machine: i686
Instance name: DBRAC
Redo thread mounted by this instance: 0 <none>
Oracle process number: 25
Unix process pid: 16772, image: oracle@skyit6 (TNS V1-V3)
*** 2011-06-01 01:27:07.010
*** SESSION ID:(1.3) 2011-06-01 01:27:07.010
*** CLIENT ID:() 2011-06-01 01:27:07.010
*** SERVICE NAME:() 2011-06-01 01:27:07.010
*** MODULE NAME:(sqlplus@skyit6 (TNS V1-V3)) 2011-06-01 01:27:07.010
*** ACTION NAME:() 2011-06-01 01:27:07.010
ERROR: slave communication error with ASM; terminating process 16772
*** 2011-06-01 01:27:07.035
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=2, mask=0x0)
----- Error Stack Dump -----
----- Current SQL Statement for this session (sql_id=bff5sku8phfgx) -----
create spfile='+dgasmspfile/spfileDBRAC.ora' from pfile
*** 2011-06-01 01:27:07.070
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
skdstdst()+41 call kgdsdst() BF9E05FC ? 2 ?
ksedst1()+77 call skdstdst() BF9E05FC ? 0 ? 1 ? AB80E70 ?
852F42A ? AB80E70 ?
ksedst()+33 call ksedst1() 0 ? 1 ?
dbkedDefDump()+2699 call ksedst() 0 ? 0 ? BF9E0D40 ? 0 ? 0 ?
0 ?
ksedmp()+47 call dbkedDefDump() 2 ? 0 ?
kfTerminateMe()+75 call ksedmp() 2 ? 3 ? F1170AC ? F149234 ?
5 ? BF9E0A6C ?
kfnCheckCommError() call kfTerminateMe() F149234 ? BF9E0C1C ?
+32 B033D83 ? 29392378 ?
F149234 ? EB00601 ?
kfncSlaveSubmit()+5 call kfnCheckCommError() 29392378 ? F149234 ?
83
kfncFileDelete()+11 call kfncSlaveSubmit() BF9E0E5C ? 0 ? F147B24 ?
81
kfioDelete()+126 call kfncFileDelete() 29DEE0 ? 1C ?
ksfddel1()+582 call kfioDelete() 29DEE0 ? 1C ? D ?
ksfddel()+187 call ksfddel1() 29DEE0 ? 1C ? D ? 0 ?
BF9E0F2C ? 9BA2 ?
ksp_spfile_create() call ksfddel() 29DEE0 ? 1C ? D ? 0 ?
+677
kspcspfp()+791 call ksp_spfile_create() BF9E49D4 ? 0 ? 29DEE0 ? 1C ?
7D02 ? 0 ?
kspocte()+67 call kspcspfp() 29DEE0 ? 1C ? F074CC4 ? F ?
61,1 1%
Regards
Dinesh