CRS-0223: Resource has placement error, when starting DB instance

Question

Hello All,
Env: 2 node 10.2.0.4 RAC on Solaris 10 SPARC
All of a sudden the DB instance on Node 2 crashed. The alert.log says "Instance terminated by MMAN".
There were no changes recently.
I tried to start it up using srvctl and it failed with
[SHCL1N02]$ srvctl start instance -d SHCL1_PRMY -i SHCL1N02
PRKP-1001 : Error starting instance SHCL1N02 on node myjpsuolicdbd02
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.SHCL1_prmy.SHCL1N02.inst' has placement error.
[SHCL1N02]$The nodeapps and crs processes are all running happily, the instance alone has a problem.
I tried
(1)Stop nodeapps, Stop CRS, Start CRS => Still instance on node2 didn't come up.
(2)sqlplus / as sysdba, startup => Instance came up, but the ora.SHCL1_prmy.SHCL1N02.inst is still offline.
Below is the error in the crsd.log file.
2009-11-04 10:18:30.627: [  CRSRES][2970821] CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.SHCL1_prmy.SHCL1N02.inst'
2009-11-04 10:21:23.269: [  CRSRES][2970843] StopResource: setting CLI values
2009-11-04 10:21:23.340: [  CRSRES][2970843] Attempting to stop `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02`
2009-11-04 10:21:30.478: [  CRSAPP][2970843] StopResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:21:30.502: [  CRSRES][2970843] Stop of `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02` succeeded.
2009-11-04 10:21:49.867: [  CRSRES][2970861] startRunnable: setting CLI values
2009-11-04 10:21:49.895: [  CRSRES][2970861] Attempting to start `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02`
2009-11-04 10:21:55.019: [  CRSAPP][2970861] StartResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:22:00.583: [  CRSAPP][2970861] StopResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:22:00.592: [  CRSRES][2970861] X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1803
2009-11-04 10:22:00.593: [  CRSRES][2970861][ALERT] `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02` has experienced an unrecoverable failure.
2009-11-04 10:22:00.593: [  CRSRES][2970861] Human intervention required to resume its availability.Any clue would be much helpful.
TIA,
JJ

Answer 1

The srvctl failed 'coz of my problem(over security).
/u01/app/oracle/product/10.2.0/db_1/log/<node_name>/racg/imon_SHCL1_prmy.log
(thanks to support guy to taking me to this file.)
In this file i saw the below error and it struck me.
SQL*Plus: Release 10.2.0.4.0 - Production on Wed Nov 4 18:59:49 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
Enter user-name: ERROR:
ORA-01031: insufficient privileges
Enter user-name: SP2-0306: Invalid option.
2009-11-04 18:59:50.737: [    RACG][176] [28813][176][ora.SHCL1_prmy.SHCL1N02.inst]: Usage: CONN[ECT] [logon] [AS {SYSDBA|SYSOPER}]
where <logon>  ::= <username>[/<password>][@<connect_identifier>] | /
Enter user-name: Enter password:
ERROR:
ORA-01005: null password given; logon denied
2009-11-04 18:59:50.738: [    RACG][176] [28813][176][ora.SHCL1_prmy.SHCL1N02.inst]: SP2-0157: unable to CONNECT to ORACLE after 3 attempt
s, exiting SQL*PlusI had SQLNET.AUTHENTICATION_SERVICES=(NONE) in the sqlnet.ora, and this stopped the srvctl from logging into the db.
1)Commented the entry in sqlnet.ora
2)crs_stop -f ora.SHCL1_prmy.SHCL1N02.inst
3)crs_stat -u => to confirm that the TARGET and STATE are OFFLINE for ora.SHCL1_prmy.SHCL1N02.inst
4)srvctl start instance -d SHCL1_PRMY -i SHCL1N02
and the instance came up.
They would investigate on regarding the ORA-7445 error.
Regards,
J J