Unable to start all 3 instances in RAC

Question

Hello All,
I got the following error while trying to restart a 3 node RAC database. OEM reported two out of three nodes to be down.
We also noticed that the cluster_database parameter was set to "False" and the cluster_database_instances integer was set to 1.We changed the parameters to 'True' and '3' respectively and while trying to restart the database we get the error message as shown below. Any suggestions/help to resolve this would be appreciated.
srvctl start database -d abc123
PRCR-1079 : Failed to start resource ora.abc123.db
CRS-5017: The resource action "ora.abc123.db start" encountered the following error:
ORA-01102: cannot mount database in EXCLUSIVE mode
. For details refer to "(:CLSN00107:)" in "/oradba/app/grid/11.2.0.3_a/log/abcde104/agent/crsd/oraagent_oradba/oraagent_oradba.log".
CRS-2674: Start of 'ora.abc123.db' on 'abcde104' failed
CRS-2632: There are no more servers to try to place resource 'ora.abc123.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.abc123.db start" encountered the following error:
ORA-01102: cannot mount database in EXCLUSIVE mode
. For details refer to "(:CLSN00107:)" in "/oradba/app/grid/11.2.0.3_a/log/abcde103/agent/crsd/oraagent_oradba/oraagent_oradba.log".
CRS-2674: Start of 'ora.abc123.db' on 'abcde103' failed

Answer 1

Hi srsatya, thanks for your reply!
I used the alter system set cluster_database=true scope=spfile SID='*' command to change my parameter from node1.After that the restart failed. The DB is now running from node 3.
My spfile is in a shared location.
SQL> show parameter spfile;
NAME                                 TYPE        VALUE
spfile                               string      +DATA/abc123/spfileabc123.ora
1.Login to the database and run "create pfile from spfile" , that will generate pfile in dbs folder on the server where you executed the command.
2.Shutdown all cluster database instance.Can I do this using SQLPLUS individually on each node? Or Srvctl?
3.From the node where you create pfile, open the pfile and edit cluster parameters
4.startup to nomount  with new modified pfile
5.Run create spfile <+DATA/abc123/spfileabc123.ora> from pfile='Give new pfile path & file name';
My current init.ora(pfile) file points to the shared location specified above.This will overwrite my spfile with the new cluster parameters. After that can I remove the new pfile that was created?
6.shutdown the database
7.run srvctl to start all database.