PRKO-2015 : Error in checking condition of instance on node:

Question

I am using Oracle 10.2.0.4 2 node RAC on Solaris 10 x86_86.
I have renamed our test database using nid and have restarted the RAC instance.
The cluster is not able to identify the database eventhough I've tried adding the database and instances manually.
oracle@slsmtdrdbs01$srvctl add database -d SMTTSTDR -o /u01/app/oracle/product/10.2.0/db_1 -p /u02/app/oracle/smttstdr/spfilesmttstdr.ora
oracle@slsmtdrdbs01$srvctl add instance -d smttstdr -i smttstdr1 -n slsmtdrdbs01
oracle@slsmtdrdbs01$srvctl add instance -d smttstdr -i smttstdr2 -n slsmtdrdbs02
oracle@slsmtdrdbs01$srvctl status database -d smttstdr
PRKO-2015 : Error in checking condition of instance on node: slsmtdrdbs01
PRKO-2015 : Error in checking condition of instance on node: slsmtdrdbs02I see these messages in the CRS log which confirm tht the instances have been successfully regstered .
2009-06-24 15:47:49.823: [  CRSRES][781664] Resource Registered: ora.SMTTSTDR.db
2009-06-24 15:48:16.256: [  CRSRES][781666] Resource Registered: ora.smttstdr.smttstdr1.inst
2009-06-24 15:48:28.436: [  CRSRES][781672] Resource Registered: ora.smttstdr.smttstdr2.inst
But still I get the error messages with srvctl.
Think so this would need CRS reboot.
Wanted to confirm if there is any other way to fix this issue as there is a production database in the same box and rebooting the CRS would require downtime.
Any suggestions ?

Answer 1

oracle@slsmtdrdbs01$crs_stat -t
Name           Type           Target    State     Host
ora....STDR.db application    OFFLINE   OFFLINE
ora....01.lsnr application    ONLINE    ONLINE    slsm...bs01
ora....s01.gsd application    OFFLINE   OFFLINE
ora....s01.ons application    ONLINE    ONLINE    slsm...bs01
ora....s01.vip application    ONLINE    ONLINE    slsm...bs01
ora....02.lsnr application    ONLINE    ONLINE    slsm...bs02
ora....s02.gsd application    OFFLINE   OFFLINE
ora....s02.ons application    ONLINE    ONLINE    slsm...bs02
ora....s02.vip application    ONLINE    ONLINE    slsm...bs02
ora.smtdgsl.db application    OFFLINE   OFFLINE
ora....l1.inst application    OFFLINE   OFFLINE
ora....l2.inst application    ONLINE    OFFLINE
ora....rddr.db application    ONLINE    ONLINE    slsm...bs02
ora....1a.inst application    ONLINE    ONLINE    slsm...bs01
ora....1b.inst application    ONLINE    ONLINE    slsm...bs02
ora....r1.inst application    OFFLINE   OFFLINE
ora....r2.inst application    OFFLINE   OFFLINESMTTSTDR is the test database that was created by renaming SMTDGSL. SMTPRDDR is the production database.
I don't see imon files at all.
I went through the thread
Not able to start instance using srvctl
I checked the permissions for the CRS directories
oracle@slsmtdrdbs01$pwd
/u01/app/oracle/product/10.2.0/crs/log
oracle@slsmtdrdbs01$ls -l
total 4
drwxrwx---   2 oracle   dba          512 Nov 24  2008 crs
drwxr-xr-t   8 root     dba          512 Nov 24  2008 slsmtdrdbs01
oracle@slsmtdrdbs01$pwd
/u01/app/oracle/product/10.2.0/crs/log/slsmtdrdbs01
oracle@slsmtdrdbs01$ls -l
total 88
drwxr-x---   2 oracle   dba          512 Nov 24  2008 admin
-rw-rw-r--   1 root     dba        37122 Mar  9 09:23 alertslsmtdrdbs01.log
drwxr-x---   2 oracle   dba         2048 Apr 30 13:14 client
drwxr-x---   2 root     dba          512 Nov 24  2008 crsd
drwxr-x---   4 oracle   dba          512 Nov 26  2008 cssd
drwxr-x---   2 oracle   dba          512 Nov 24  2008 evmd
drwxrwxr-t   5 oracle   dba          512 Jun 25 10:00 racgslsmtdrdbs01 directory is owned by root. Is that an issue?
But racg is owned by oracle.
Output of the trace
srvctl status database -d smttstdr
432  [main] [10:5:53:34] [HAOperationImpl.runCommand:1223]  CRS cmd is: /u01/app/oracle/product/10.2.0/crs/bin/crs_stat -u
ora.SMTTSTDR.smttstdr1.inst ora.SMTTSTDR.smttstdr2.inst
   433  [main] [10:5:53:36] [RuntimeExec.runCommand:74]  Calling Runtime.exec() with the command
   434  [main] [10:5:53:37] [RuntimeExec.runCommand:76]  /u01/app/oracle/product/10.2.0/crs/bin/crs_stat
   435  [main] [10:5:53:38] [RuntimeExec.runCommand:76]  -u
   436  [main] [10:5:53:39] [RuntimeExec.runCommand:76]  ora.SMTTSTDR.smttstdr1.inst
   437  [main] [10:5:53:41] [RuntimeExec.runCommand:76]  ora.SMTTSTDR.smttstdr2.inst
   438  [main] [10:5:53:47] [RuntimeExec.runCommand:131]  runCommand: Waiting for the process
   439  [Thread-1] [10:5:53:48] [StreamReader.run:61]  In StreamReader.run
   440  [Thread-0] [10:5:53:48] [StreamReader.run:61]  In StreamReader.run
   441  [Thread-0] [10:5:53:96] [StreamReader.run:65]  OUTPUT>CRS-0210: Could not find resource ora.SMTTSTDR.smttstdr1.inst.
   442  [Thread-0] [10:5:53:97] [StreamReader.run:65]  OUTPUT>
   443  [Thread-0] [10:5:53:98] [StreamReader.run:65]  OUTPUT>CRS-0210: Could not find resource ora.SMTTSTDR.smttstdr2.inst.
   444  [Thread-0] [10:5:53:100] [StreamReader.run:65]  OUTPUT>
   445  [main] [10:5:53:103] [RuntimeExec.runCommand:133]  runCommand: process returns 210
   446  [main] [10:5:53:104] [RuntimeExec.runCommand:147]  RunTimeExec: output>
   447  [main] [10:5:53:105] [RuntimeExec.runCommand:150]  CRS-0210: Could not find resource ora.SMTTSTDR.smttstdr1.inst.
   448  [main] [10:5:53:108] [RuntimeExec.runCommand:150]
   449  [main] [10:5:53:109] [RuntimeExec.runCommand:150]  CRS-0210: Could not find resource ora.SMTTSTDR.smttstdr2.inst.
   450  [main] [10:5:53:110] [RuntimeExec.runCommand:150]
   451  [main] [10:5:53:111] [RuntimeExec.runCommand:155]  RunTimeExec: error>
   452  [main] [10:5:53:113] [RuntimeExec.runCommand:175]  Returning from RunTimeExec.runCommand
   453  [main] [10:5:53:114] [HAOperationImpl.runCommand:1232]  signed exit value = 210
   454  [main] [10:5:53:115] [HAOperationImpl.runCommand:1257]  set status HA_RES_NOT_EXIST_ERR
   455  [main] [10:5:53:117] [HAStatusOperation.run:127]  Returned from executing the HA Operation
   456  [main] [10:5:53:118] [HAStatusOperation.run:132]  OUTPUT> CRS-0210: Could not find resource ora.SMTTSTDR.smttstdr1.in
st.
   457  [main] [10:5:53:119] [HAStatusOperation.run:132]  OUTPUT>
   458  [main] [10:5:53:120] [HAStatusOperation.run:132]  OUTPUT> CRS-0210: Could not find resource ora.SMTTSTDR.smttstdr2.in
st.
   459  [main] [10:5:53:121] [HAStatusOperation.run:132]  OUTPUT>
   460  [main] [10:5:53:122] [HAStatusOperation.run:138]  ERROR>
   461  [main] [10:5:53:124] [LocalCommand.execute:56]  LocalCommand.execute: Returned from run method
   462  [main] [10:5:53:125] [ParallelServerHA.isRunningInstances:581]  Stat operation failed. it could be a partial failure.
no exception thrown.
   463  [main] [10:5:53:127] [StatusAction.internalDisplayInstanceStatus:708]  displaying status for instance smttstdr1
   464  [main] [10:5:53:129] [Instance.isEnabled:197]  enabled is true
   465  PRKO-2015 : Error in checking condition of instance on node: slsmtdrdbs01
   466  [main] [10:5:53:143] [StatusAction.internalDisplayInstanceStatus:708]  displaying status for instance smttstdr2
   467  [main] [10:5:53:144] [Instance.isEnabled:197]  enabled is true
   468  PRKO-2015 : Error in checking condition of instance on node: slsmtdrdbs02