PRCR-1079 Failed to start resource ora.rac.db - during installation

Hi
After successful installation of Grid Infrasturcture I proceeded with database installation on clusterware and at the stage when the installer was creating clone database I got the following errors ( this was my 2nd attempt and I got the same errors both the time ) :
Errors:
PRCR-1079 : Failed to start resource ora.rac.db
ORA-01092 : ORACLE instance terminated. Disconnection forced
ORA-00704 : bootstrap process failure
ORA-00604 : error occurred at resursiive SQL level 2
ORA-01578 : ORACLE data block corrupted (file # 1, block # 5505)
ORA-01110 : data file 1:'+DATA/rac/datafile/system.256.799676855'
Process ID : 23498
Session ID : 63 Serial number 3
CRS-2674 Start of 'ora.rac.db' on 'rac2' failed
CRS-2632 There are no more servers to try to place resource 'ora.rac.db' on that would satisfy its placement policy
There are no logs on that node (rac2)
I am running Oracle Linux 5.4 64 bit
As mentioned above this was my 2nd attempt afresh and I got the same errors both the times, please let me know what is the problem as the rac2 is replica of rac2 in VMWare.
Thanks for your help
Rgds
T

Hi
I tried again for the 3rd time and go the same error again, this time I rebuilt the node 2 - Can someone ple ase help me with this issue why it keeps failing on node 2 at the same stage for the 2rd time in a row.
Also please help me clone the database manually from node 1 to node 2 so I don't have to try to reinstall it again, there must be ways to do it
Thanks for your help in advance
Rgds
T

Similar Messages

  • Facing prcr-1079 failed to start resource ora.orcl.db  problem

    Hi all,
    I am about installing an oracle  database with asm
    I have installed ASM libraries and created asm instance with diskgroup successfully mounted. I started the ASM instance in the oragrid user.
    Know I runned the DBCA command to create a database named ORCL from the oracle user,
    I followed up the procedure and during creation phase (at 80 %) I got multiple errors,
    :  prcr-1079 failed to start resource ora.orcl.db 
       ORA-01031: insufficient privileges
    is there anything wrong, or I need to grant system privileges for oracle over directories?
    Thanks for help

    HI,
    Please check following link: PRCR-1079: Failed to start resource ora.test.db, ORA-01031,ORA-2674 | Just Innovation
    # usermod -G asmadmin,asmdba,asmoper,dba grid
    Now you will notice the Privileges are changed
    # id grid
    uid=501(grid) gid=1000(oinstall) groups=1000(oinstall),1100(dba),1300(asmadmin),1400(asmdba),1500(asmoper)
    Now try to runInstaller again!
    Thank you

  • Failed to start resource: Name: ora.racdb.db, node: null, filter: null, ms

    Hi DBA's.
    Im, running
    Finalizing Installation 96% the following Warning:
    [Thread-288] [ 2010-01-21 14:28:57.456 ARST ] [CRSNative.internalStartResource:352] Failed to start resource: Name: ora.racdb.db, node: null, filter: null, msg CRS-2674:
    Start of 'ora.racdb.db' on 'linux2' failed
    CRS-2678: 'ora.racdb.db' on 'linux2' has experienced an unrecoverable failure
    CRS-0267: Human intervention required to resume its availability.
    CRS-5807: Agent failed to process the message
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    [Thread-288] [ 2010-01-21 14:28:57.457 ARST ] [PostDBCreationStep.executeImpl:828] Exception while Starting with HA Database Resource PRCR-1079 : Failed to start resourc
    e ora.racdb.db
    CRS-2674: Start of 'ora.racdb.db' on 'linux2' failed
    CRS-2678: 'ora.racdb.db' on 'linux2' has experienced an unrecoverable failure
    CRS-0267: Human intervention required to resume its availability.
    CRS-5807: Agent failed to process the message
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    oracle$ dbca

    Hi...
    Now is Ok.
    I did:
    srvctl start instance -d racdb -i racdb2
    [oracle@linux1 oracle]$ su - grid -c "crsctl status resource -w \"TYPE co 'ora'\" -t"
    Password:
    NAME TARGET STATE SERVER STATE_DETAILS
    Local Resources
    ora.CRS.dg
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    ora.FRA.dg
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    ora.LISTENER.lsnr
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    ora.RACDB_DATA.dg
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    ora.asm
    ONLINE ONLINE linux1 Started
    ONLINE ONLINE linux2 Started
    ora.eons
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    ora.gsd
    OFFLINE OFFLINE linux1
    OFFLINE OFFLINE linux2
    ora.net1.network
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    ora.ons
    ONLINE ONLINE linux1
    ONLINE ONLINE linux2
    Cluster Resources
    ora.LISTENER_SCAN1.lsnr
    1 ONLINE ONLINE linux1
    ora.linux1.vip
    1 ONLINE ONLINE linux1
    ora.linux2.vip
    1 ONLINE ONLINE linux2
    ora.oc4j
    1 OFFLINE OFFLINE
    ora.racdb.db
    1 ONLINE ONLINE linux1 Open
    2 ONLINE ONLINE linux2 Open
    ora.scan1.vip
    1 ONLINE ONLINE linux1
    Thanks.

  • Root.sh failed with CRS-2800:   Can't start resource 'ora.ctssd' as it is

    Cannot start resource 'ora.ctssd' as it is already in the INTERMEDIATE state.
    I was trying to install two node RAC on linux REDHAT, I have encountered following errors:
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.ctssd' as it is already in the INTERMEDIATE state on server
    CRS-4000: Command Start failed, or completed with errors.
    Clusterware exclusive mode start of Clusterware Ready Services failed at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6475.
    /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
    How to fix this issue?
    runCluvfy ran successfully.
    I found out on the second node(oracle02), there is no any daemon running. But on the local installation node(oracle01):
    [[email protected] bin]# ./crsctl stat res -t -init
    NAME TARGET STATE SERVER STATE_DETAILS
    Cluster Resources
    ora.asm
    1 ONLINE ONLINE oracle01 Started
    ora.cluster_interconnect.haip
    1 ONLINE ONLINE oracle01
    ora.crf
    1 OFFLINE OFFLINE
    ora.crsd
    1 ONLINE OFFLINE
    ora.cssd
    1 ONLINE ONLINE oracle01
    ora.cssdmonitor
    1 ONLINE ONLINE oracle01
    ora.ctssd
    1 ONLINE INTERMEDIATE oracle01 Wrong check return.
    ora.diskmon
    1 ONLINE ONLINE oracle01
    ora.drivers.acfs
    1 ONLINE ONLINE oracle01
    ora.evmd
    1 OFFLINE OFFLINE
    ora.gipcd
    1 ONLINE ONLINE oracle01
    ora.gpnpd
    1 ONLINE ONLINE oracle01
    ora.mdnsd
    1 ONLINE ONLINE oracle01
    [[email protected] bin]#
    How to proceed next step? what to fix in order for my installation to be successful.
    Thanks in advance.

    OK, please post the css last lines in the log.
    Also, if you have alert log (for crs, it should be under $GI_HOME/log/<hostname>) post it as well.
    It's a bit complicated to trace it here, I'll do my best, but you might want to open an SR on this.
    Liron

  • Database failed to start up - ORA-03113: end-of-file on communication chann

    Hi,
    We had power failure, and i had to restore the database from offline backup.
    after restore i tried to start the database and i received following errors.
    SQL> startup
    ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
    ORACLE instance started.
    Total System Global Area 663961600 bytes
    Fixed Size 2184056 bytes
    Variable Size 394267784 bytes
    Database Buffers 260046848 bytes
    Redo Buffers 7462912 bytes
    Database mounted.
    ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
    SQL>
    SQL> alter database open resetlogs;
    alter database open resetlogs
    ERROR at line 1:
    ORA-03113: end-of-file on communication channel
    Process ID: 7179
    Session ID: 227 Serial number: 3
    SQL>
    I checked the alert.log
    Tue Jun 19 08:06:33 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 3
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up:
    Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options.
    Using parameter settings in server-side spfile /oracle/PRD/112_64/dbs/spfilePRD.ora
    System parameters with non-default values:
    processes = 900
    sessions = 1800
    event = "10027"
    event = "10028"
    event = "10142"
    event = "10183"
    event = "10191"
    event = "38068 level 100"
    event = "38085"
    event = "38087"
    event = "44951 level 1024"
    shared_pool_size = 352M
    filesystemio_options = "setall"
    control_files = "/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf"
    control_files = "/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf"
    control_files = "/oracle/PRD/saparch/cntrl/cntrlPRD.dbf"
    control_file_record_keep_time= 30
    db_block_size = 8192
    db_cache_size = 248M
    compatible = "11.2.0"
    log_archive_dest_1 = "LOCATION=/oracle/PRD/saparch/PRDarch"
    log_archive_dest_2 = "LOCATION=/oracle/PRD/saparch1/PRDarch"
    log_archive_format = "%t_%s_%r.dbf"
    db_files = 1024
    log_checkpoints_to_alert = TRUE
    replication_dependency_tracking= FALSE
    undo_tablespace = "PSAPUNDO"
    inmemory_undo = FALSE
    undo_retention = 16000
    recyclebin = "off"
    remote_os_authent = TRUE
    service_names = "PRD.WORLD"
    parallel_max_servers = 80
    parallel_execution_message_size= 16384
    fixcontrol = "5099019:ON"
    fixcontrol = "5705630:ON"
    fixcontrol = "6399597:ON"
    fixcontrol = "6430500:ON"
    fixcontrol = "6440977:ON"
    fixcontrol = "6626018:ON"
    fixcontrol = "6972291:ON"
    fixcontrol = "9495669:ON"
    fixcontrol = "9196440:ON"
    fixcontrol = "8937971:ON"
    audit_file_dest = "/oracle/PRD/saptrace/audit"
    hpux_sched_noage = 178
    db_name = "PRD"
    open_cursors = 900
    star_transformation_enabled= "TRUE"
    optimizerextended_cursor_sharing_rel= "NONE"
    optimizeradaptive_cursor_sharing= FALSE
    parallel_threads_per_cpu = 1
    optimizer_index_cost_adj = 20
    query_rewrite_enabled = "FALSE"
    optimpeek_user_binds = FALSE
    pga_aggregate_target = 4G
    optimizeruse_feedback = FALSE
    diagnostic_dest = "/oracle/PRD/saptrace"
    max_dump_file_size = "20000"
    Deprecated system parameters with specified values:
    remote_os_authent
    End of deprecated system parameter listing
    Tue Jun 19 08:06:34 2012
    PMON started with pid=2, OS id=7143
    Tue Jun 19 08:06:34 2012
    PSP0 started with pid=3, OS id=7145
    Tue Jun 19 08:06:35 2012
    VKTM started with pid=4, OS id=7147
    VKTM running at (100ms) precision
    Tue Jun 19 08:06:35 2012
    GEN0 started with pid=5, OS id=7149
    Tue Jun 19 08:06:35 2012
    DIAG started with pid=6, OS id=7151
    Tue Jun 19 08:06:35 2012
    DBRM started with pid=7, OS id=7153
    Tue Jun 19 08:06:35 2012
    DIA0 started with pid=8, OS id=7155
    Tue Jun 19 08:06:35 2012
    MMAN started with pid=9, OS id=7157
    Tue Jun 19 08:06:35 2012
    DBW0 started with pid=10, OS id=7159
    Tue Jun 19 08:06:35 2012
    LGWR started with pid=11, OS id=7161
    Tue Jun 19 08:06:35 2012
    CKPT started with pid=12, OS id=7163
    Tue Jun 19 08:06:35 2012
    SMON started with pid=13, OS id=7165
    Tue Jun 19 08:06:35 2012
    RECO started with pid=14, OS id=7167
    Tue Jun 19 08:06:35 2012
    MMON started with pid=15, OS id=7169
    Tue Jun 19 08:06:35 2012
    MMNL started with pid=16, OS id=7171
    ORACLE_BASE from environment = /oracle
    Tue Jun 19 08:06:36 2012
    ALTER DATABASE MOUNT
    Successful mount of redo thread 1, with mount id 1849974924
    Database mounted in Exclusive Mode
    Lost write protection disabled
    Completed: ALTER DATABASE MOUNT
    Tue Jun 19 08:06:40 2012
    ALTER DATABASE OPEN
    Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_7179.trc:
    ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
    ORA-1589 signalled during: ALTER DATABASE OPEN...
    Tue Jun 19 08:08:01 2012
    alter database open resetlogs
    RESETLOGS after complete recovery through change 23408973429
    Exception [type: SIGSEGV, Invalid Permissions for object] [ADDR:0x4] [PC:0x40000000083B8941, $cold_kcrf_val_log_bsz()+929] [flags: 0x0, count: 1]
    Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_7179.trc (incident=583361):
    ORA-07445: exception encountered: core dump [kcrf_val_log_bsz()+929] [SIGSEGV] [ADDR:0x4] [PC:0x40000000083B8941] [Invalid Permissions for object] []
    ORA-27041: unable to open file
    HPUX-ia64 Error: 13: Permission denied
    Additional information: 9
    ORA-27041: unable to open file
    HPUX-ia64 Error: 13: Permission denied
    Additional information: 9
    Incident details in: /oracle/PRD/saptrace/diag/rdbms/prd/PRD/incident/incdir_583361/PRD_ora_7179_i583361.trc
    Use ADRCI or Support Workbench to package the incident.
    See Note 411.1 at My Oracle Support for error and packaging details.
    Tue Jun 19 08:08:05 2012
    Dumping diagnostic data in directory=[cdmp_20120619080805], requested by (instance=1, osid=7179), summary=[incident=583361].
    Tue Jun 19 08:08:06 2012
    Sweep [inc][583361]: completed
    Sweep [inc2][583361]: completed
    I noticed the in spfile that archive log was still being used, but i remember that we disabled the archiving.
    oraprd> more spfilePRD.ora
    ^G^D^D[.M-^D"^K
    PRD.__db_cache_size=241591910
    PRD.__oracle_base='/oracle'#ORACLE_BASE set from environment
    PRD.__shared_pool_size=365743308
    *._fix_control='5099019:ON','5705630:ON','6399597:ON','6430500:ON','6440977:ON','6626018:ON','6972291:ON','9495669:ON','9196440:ON','8937971:ON'
    *._in_memory_undo=FALSE#RECOMMENDED BY ORACLE/SAP FOR 11.2.0 - SAP note 1431798
    *._optim_peek_user_binds=FALSE#RECOMMENDED BY ORACLE/SAP FOR 11.2.0 - SAP note 1431798
    *._optimizer_adaptive_cursor_sharing=FALSE#RECOMMENDED BY ORACLE/SAP FOR 11.2.0 - SAP note 1431798
    *._optimizer_extended_cursor_sharing_rel='NONE'#RECOMMENDED BY ORACLE/SAP FOR 11.2.0 - SAP note 1431798
    *._optimizer_use_feedback=FALSE#RECOMMENDED BY ORACLE/SAP FOR 11.2.0 - SAP note 1431798
    *.audit_file_dest='/oracle/PRD/saptrace/audit'
    *.compatible='11.2.0'#Parameter setting for SAP on Oracle Release 11.2 (see SAP note 1431798)
    *.control_file_record_keep_time=30
    *.control_files='/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf','/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf','/oracle/PRD/saparch/cntrl/cntrlPRD.dbf'
    *.db_block_size=8192
    *.db_cache_size=257698037
    *.db_files=1024
    *.db_name='PRD'
    *.diagnostic_dest='/oracle/PRD/saptrace'
    *.event='10027','10028','10142','10183','10191','38068 level 100','38085','38087','44951 level 1024'#RECOMMENDED BY ORACLE/SAP FOR 11.2.0 - SAP note 1431798
    *.filesystemio_options='setall'
    *.hpux_sched_noage=178
    *.log_archive_dest_1='LOCATION=/oracle/PRD/saparch/PRDarch'
    *.log_archive_dest_2='LOCATION=/oracle/PRD/saparch1/PRDarch'
    *.log_archive_format='%t_%s_%r.dbf'
    *.log_checkpoints_to_alert=TRUE
    *.max_dump_file_size='20000'
    *.open_cursors=900
    *.optimizer_index_cost_adj=20
    *.parallel_execution_message_size=16384
    *.parallel_max_servers=80
    *.parallel_threads_per_cpu=1
    *.pga_aggregate_target=4294967296
    *.processes=900
    *.query_rewrite_enabled='FALSE'
    *.recyclebin='off'
    *.remote_os_authent=TRUE#Parameter setting for SAP on Oracle Release 11.2 (see SAP note 1431798)
    *.replication_dependency_tracking=FALSE
    *.service_names='PRD.WORLD'
    *.sessions=1800
    *.shared_pool_size=365743308
    *.star_transformation_enabled='TRUE'
    *.undo_retention=16000
    *.undo_tablespace='PSAPUNDO'
    I reviewed the /oracle/PRD/saptrace/diag/rdbms/prd/PRD/incident/incdir_583361/PRD_ora_7179_i583361.trc, it hugh file with alot of Jargon.
    Any idea what could be causing this error. Thanks in advance

    ALTER DATABASE OPEN RESETLOGS
    RESETLOGS after complete recovery through change 23408973429
    Exception [type: SIGSEGV, Invalid Permissions for object] [ADDR:0x4] [PC:0x40000000083B8941, $cold_kcrf_val_log_bsz()+929] [flags: 0x0, count: 1]
    Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_8363.trc (incident=583362):
    ORA-07445: exception encountered: core dump [kcrf_val_log_bsz()+929] [SIGSEGV] [ADDR:0x4] [PC:0x40000000083B8941] [Invalid Permissions for object] []
    ORA-27041: unable to open file
    HPUX-ia64 Error: 13: Permission denied
    Additional information: 9
    ORA-27041: unable to open file
    HPUX-ia64 Error: 13: Permission denied
    Additional information: 9
    Incident details in: /oracle/PRD/saptrace/diag/rdbms/prd/PRD/incident/incdir_583362/PRD_ora_8363_i583362.trc
    Use ADRCI or Support Workbench to package the incident.
    See Note 411.1 at My Oracle Support for error and packaging details.
    Tue Jun 19 09:16:24 2012
    Dumping diagnostic data in directory=[cdmp_20120619091624], requested by (instance=1, osid=8363), summary=[incident=583362].
    Tue Jun 19 09:16:26 2012
    Sweep [inc][583362]: completed
    Sweep [inc2][583362]: completed

  • Failed to load resource file sqlcmd.rll- during EHP1 installation

    Hello all,
    We are installing EHP1 on Netweaver with MSSQL server 2005 SP2
    I am getting the error
    failed to load resource file sqlcmd.rll
    Please help
    Rohit

    It's always helpful to give as much information as possible when asking question. You did e. g. not specify
    - in which phase/when this error occurs
    - where this error message is coming
    - what has been done prior to this
    etc.
    so everything leading to answers could only be wild guesses such as "are the harddisks all mounted and available in the resource groups" or "search the filesystem for that DLL" etc.
    All we know is that "a DLL is missing".
    Markus

  • Dbconsole failed to start on one RAC node

    Hi
    I have 2 RAC nodes (RHEL 4) and 10.2.0.1. On one dbconsole is running and on other I get the following. Earlier dbconsole
    on both the nodes used to run perfectly fine. I will appreacite any suggestions to rectify this problem.
    Regards
    oracle@rac01<18>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> emctl start dbconsole
    TZ set to Canada/Newfoundland
    Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
    Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
    http://rac01:1158/em/console/aboutApplication
    Agent Version : 10.1.0.4.1
    OMS Version : Unknown
    Protocol Version : 10.1.0.2.0
    Agent Home : /u01/app/oracle/product/10.2/db_1/rac01_RACDB1
    Agent binaries : /u01/app/oracle/product/10.2/db_1
    Agent Process ID : 23329
    Parent Process ID : 21132
    Agent URL : http://rac01:3938/emd/main
    Started at : 2007-07-25 11:37:32
    Started by user : oracle
    Last Reload : 2007-07-25 11:37:32
    Last successful upload : (none)
    Last attempted upload : (none)
    Total Megabytes of XML files uploaded so far : 0.00
    Number of XML files pending upload : 371
    Size of XML files pending upload(MB) : 7.66
    Available disk space on upload filesystem : 44.78%
    Agent is already started. Will restart the agent
    Stopping agent ... stopped.
    Starting Oracle Enterprise Manager 10g Database Control ............................................................................................. failed.
    Logs are generated in directory /u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log
    oracle@rac01<19>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log>
    ON OTHER NODE:
    oracle@rac02<2>:/u01/app/oracle> emctl start dbconsole
    TZ set to Canada/Newfoundland
    Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
    Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
    http://rac01:1158/em/console/aboutApplication
    Starting Oracle Enterprise Manager 10g Database Control .................................... started.
    Logs are generated in directory /u01/app/oracle/product/10.2/db_1/rac02_RACDB2/sysman/log
    oracle@rac02<3>:/u01/app/oracle>

    Thanks for your time and reply .
    Well, here is what I got, couldn't make out from here.
    Regards
    oracle@rac01<19>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> ls -lart
    total 13500
    drwxr----- 7 oracle dba 4096 Jul 14 10:48 ..
    -rw-r----- 1 oracle dba 0 Jul 14 10:48 emdctl.log
    drwxrwx--- 2 oracle dba 4096 Jul 14 10:54 nmcRACDB11521
    -rw-r----- 1 oracle dba 4655792 Jul 24 23:01 emoms.trc
    -rw-r----- 1 oracle dba 4655792 Jul 24 23:01 emoms.log
    drwxr----- 3 oracle dba 4096 Jul 25 11:35 .
    -rw-r----- 1 oracle dba 4096 Jul 25 12:05 emdb.nohup.lr
    -rw-r----- 1 oracle dba 1074 Jul 25 12:05 emagent_perl.trc
    -rw-r----- 1 oracle dba 1731 Jul 25 12:06 emagent.log
    -rw-r----- 1 oracle dba 1080 Jul 25 12:07 emagentfetchlet.trc
    -rw-r----- 1 oracle dba 1080 Jul 25 12:07 emagentfetchlet.log
    -rw-r----- 1 oracle dba 81089 Jul 25 13:28 emdctl.trc
    -rw-r----- 1 oracle dba 3309143 Jul 25 13:28 emdb.nohup
    -rw-r----- 1 oracle dba 1044518 Jul 25 13:28 emagent.trc
    oracle@rac01<20>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> cat emagent.log
    2007-07-14 10:50:44 Thread-3086936288 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
    2007-07-14 10:51:16 Thread-3086936288 EMAgent started successfully (00702)
    2007-07-14 14:38:21 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
    2007-07-14 14:39:00 Thread-3086935744 EMAgent started successfully (00702)
    2007-07-24 07:05:06 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
    2007-07-24 07:07:11 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
    2007-07-24 07:07:14 Thread-3086935744 EMAgent started successfully (00702)
    2007-07-24 12:06:27 Thread-3086935744 EMAgent normal shutdown (00703)
    2007-07-24 12:08:26 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
    2007-07-24 12:08:51 Thread-3086935744 EMAgent started successfully (00702)
    2007-07-25 11:35:35 Thread-3086935744 EMAgent normal shutdown (00703)
    2007-07-25 11:37:32 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
    2007-07-25 11:39:29 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
    2007-07-25 11:39:30 Thread-3086935744 EMAgent started successfully (00702)
    2007-07-25 12:03:36 Thread-3086935744 EMAgent normal shutdown (00703)
    2007-07-25 12:05:15 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
    2007-07-25 12:06:23 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
    2007-07-25 12:06:24 Thread-3086935744 EMAgent started successfully (00702)
    oracle@rac01<21>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> cat emagentfetchlet.log
    2007-07-14 11:01:44,208 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-14 14:40:29,096 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-24 07:10:44,123 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-24 12:12:48,187 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-25 11:41:25,628 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-25 12:07:30,335 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    oracle@rac01<22>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log>
    oracle@rac01<22>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -40 emagentfetchlet.trc
    2007-07-14 11:01:44,208 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-14 14:40:29,096 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-24 07:10:44,123 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-24 12:12:48,187 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-25 11:41:25,628 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    2007-07-25 12:07:30,335 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
    oracle@rac01<25>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -10 emdctl.trc
    2007-07-25 13:01:02 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:04:41 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:07:12 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:10:50 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:14:32 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:18:09 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:20:40 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:24:27 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:28:06 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:31:43 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    oracle@rac01<28>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -10 emagent.trc
    2007-07-25 13:31:44 Thread-43162528 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:31:44 Thread-43162528 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
    2007-07-25 13:32:14 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:32:14 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
    2007-07-25 13:32:14 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:32:14 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
    2007-07-25 13:32:44 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:32:44 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
    2007-07-25 13:32:44 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
    2007-07-25 13:32:44 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
    Message was edited by:
    Singh

  • OPMN failed to start  (but it seems running) during setup

    I got a warning during setup of Grid Control on new database )10g rel 2 (Linux).
    I tried to set on contnue and after setup OPMN starts (as during setup) but ./emctl
    start oms fails with unexpected error.
    Any suggestion?
    Thanks
    Fabio

    Hi...
    I have the same problem.
    listener for db10g is started
    I set LD_LIBRARY_PATH to lib dirs with libmodapi.so and others and made
    "opmnctl startall" succefull
    I can connect to EM on 7777, 4489 ports
    no any messages in <oms10g_home>/ompn/log don't tell me anything about problems, <oms10g_home>/sysman/log is empty
    emctl start oms failed with unespected error :((((
    what exactly I have to see on emoms.properties?
    OEM 10.2.0.1 on SLES 10

  • Unable to start agent on RAC target after installation

    Hi All,
    I have just finised installing agent on solaris 10 RAC to be monitored by grid control 11g,but the agent will not start.
    I installed the agent as the oracle user
    I get this error when I try starting the agent.
    -bash-3.00$ pwd
    */u01/app/oracle/agent11g*
    -bash-3.00$ ./bin/emctl start agent
    EM Configuration issue. /u01/app/oracle/agent11g/node1 not found.
    -bash-3.00$
    Please help

    Hi,
    Make sure that you have set all your environment variables properly before starting your agent service.
    thanks,
    X A H E E R

  • ORA-12546 error during installation of 9.2.0.1 on Windows 2003 Server

    I have encountered an error whilst installing oracle 9.2.0.1 on a Windows 2003 Server. The error appears in the final part of the installation, when the Database Configuration Assistant is performing some work - it reaches the part where it creates the database instance - here we get the "ORA-12546: TNS Permission denied" error - this part of the installation is marked as failed - i can retry it as many times as i want but the same thing happens. If i finish the installation (even though the error has occurred) the oracle installation doesn't really work . I can't open the Database Configuration Assistant and don't really have a sysdba user. However all the services seem to be running fine (even the tns listener)
    I was wondering whether it could be a problem that would be solved with the 9.2.0.3 patch - but this patch assumes that you have performed a proper working installation of 9.2.0.1, so I'm not quite convinced that the problem is related to the oracle version.
    I've read other threads where the problem has been read/write privileges on some files and folders, but this shouldn't be an issue when I'm logged onto the machine as a local administrator.
    The install.log files don't really provide any extra information.

    I have exactly the same problem installing Oracle on a Windows Server 2003 X64 Enterprise Edition. I have Intel Xeon processors (EM64T).
    I tried to install Oracle for 64 bits, but it didn't work. Now I'm trying to install 32 bits version of Oracle, and I get the same error and the same problem, mentioned above.
    Can anyone help us, I can't figure out a way to solve it. I don't know if Oracle for 32 bits is not compatible with that version of Windows, even if it is supposed to be.
    Thanks!!!!

  • Not able to start databse in rac env

    Hi all,
    I am using oracle 11g r2 on rhel5, when i am trying to start the database it is throwing the below error:
    PRCR-1079 : Failed to start resource ora.dev.db
    ORA-12547: TNS:lost contact
    ORA-12547: TNS:lost contact
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-00845: MEMORY_TARGET not supported on this system
    CRS-2674: Start of 'ora.dev.db' on 'rac1' failed
    ORA-01034: ORACLE not availableResource Status
    [grid@rac2 ~]$ crs_stat -t
    Name           Type           Target    State     Host       
    ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1       
    ora.DATA1.dg   ora....up.type ONLINE    ONLINE    rac1       
    ora.FRA.dg     ora....up.type ONLINE    ONLINE    rac1       
    ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1       
    ora....R1.lsnr ora....er.type ONLINE    ONLINE    rac1       
    ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1       
    ora....N2.lsnr ora....er.type ONLINE    ONLINE    rac2       
    ora....N3.lsnr ora....er.type ONLINE    ONLINE    rac2       
    ora.asm        ora.asm.type   ONLINE    ONLINE    rac1     
    *ora.dev.db*     ora....se.type ONLINE    OFFLINE                Oracle is not able to find these resource
    ora.eons       ora.eons.type  ONLINE    ONLINE    rac1       
    ora.gsd        ora.gsd.type   OFFLINE   OFFLINE              
    ora....network ora....rk.type ONLINE    ONLINE    rac1       
    ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE              
    ora.ons        ora.ons.type   ONLINE    ONLINE    rac1       
    ora....SM1.asm application    ONLINE    ONLINE    rac1       
    ora....C1.lsnr application    ONLINE    ONLINE    rac1       
    ora....C1.lsnr application    ONLINE    ONLINE    rac1       
    ora.rac1.gsd   application    OFFLINE   OFFLINE              
    ora.rac1.ons   application    ONLINE    ONLINE    rac1       
    ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1       
    ora....SM2.asm application    ONLINE    ONLINE    rac2       
    ora....C2.lsnr application    ONLINE    ONLINE    rac2       
    ora....C2.lsnr application    ONLINE    ONLINE    rac2       
    ora.rac2.gsd   application    OFFLINE   OFFLINE              
    ora.rac2.ons   application    ONLINE    ONLINE    rac2       
    ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2       
    ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac1       
    ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1       
    ora.scan2.vip  ora....ip.type ONLINE    ONLINE    rac2       
    ora.scan3.vip  ora....ip.type ONLINE    ONLINE    rac2       
    [grid@rac2 ~]$
    [grid@rac2 ~]$ crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    [grid@rac2 ~]$
    [grid@rac2 ~]$ srvctl status database -d dev
    Instance dev1 is not running on node rac1
    Instance dev2 is not running on node rac2
    [grid@rac2 ~]$ When trying to start database, it throw some error like
    [grid@rac2 ~]$ srvctl start database -d dev
    PRCR-1079 : Failed to start resource ora.dev.db
    ORA-12547: TNS:lost contact
    ORA-12547: TNS:lost contact
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-00845: MEMORY_TARGET not supported on this system
    CRS-2674: Start of 'ora.dev.db' on 'rac1' failed
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    CRS-2632: There are no more servers to try to place resource 'ora.dev.db' on that would satisfy its placement policy
    ORA-12547: TNS:lost contact
    ORA-12547: TNS:lost contact
    CRS-2674: Start of 'ora.dev.db' on 'rac2' failed
    ORA-12547: TNS:lost contact
    ORA-12547: TNS:lost contact
    ORA-12547: TNS:lost contact
    [grid@rac2 ~]$ Can anybody suggest me how to resolve the same.
    Edited by: user13389425 on Jun 10, 2012 11:32 PM

    I have alos configrured /dev/shm....
    [root@rac2 pfile]# mount -t tmpfs shmfs -o size=1300m /dev/shm
    [root@rac2 pfile]# df -kh
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2              19G   13G  5.1G  73% /
    /dev/sda5             874M  829M     0 100% /tmp
    /dev/sda1              99M   14M   81M  15% /boot
    tmpfs                 1.3G     0  1.3G   0% /dev/shm
    shmfs                 1.3G     0  1.3G   0% /dev/shm
    [root@rac2 pfile]# But still getting the error:
    [grid@rac2 ~]$ srvctl start database -d dev
    PRCR-1079 : Failed to start resource ora.dev.db
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    CRS-2674: Start of 'ora.FRA.dg' on 'rac2' failed
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    CRS-2674: Start of 'ora.DATA1.dg' on 'rac2' failed
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    CRS-2632: There are no more servers to try to place resource 'ora.dev.db' on that would satisfy its placement policy
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    ORA-01078: failure in processing system parameters
    CRS-2674: Start of 'ora.dev.db' on 'rac1' failed
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    [grid@rac2 ~]$

  • 11gR2 Cluster node does not start, start of ora.gipcd fails

    Hi
    I have a clusert node which just does not start, and i do know understand what the problem is.
    Does anyone have an idea what the problem cound be?
    I am using Oracle 11.2.0.1.0 on OEL.
    I appreciate any help, thank you.
    Regards Bernhard
    NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
    Cluster Resources
    ora.asm
          1        ONLINE  OFFLINE                                                  
    ora.crsd
          1        ONLINE  INTERMEDIATE node001                             
    ora.cssd
          1        ONLINE  OFFLINE                                                  
    ora.cssdmonitor
          1        ONLINE  OFFLINE                                                  
    ora.ctssd
          1        ONLINE  OFFLINE                                                  
    ora.diskmon
          1        ONLINE  OFFLINE                                                  
    ora.drivers.acfs
          1        ONLINE  ONLINE       node001                             
    ora.evmd
          1        ONLINE  OFFLINE                                                  
    ora.gipcd
          1        ONLINE  OFFLINE                                                  
    ora.gpnpd
          1        ONLINE  OFFLINE                                                  
    ora.mdnsd
          1        ONLINE  ONLINE       node001                              alertnode001.log
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2012-06-29 08:43:52.511
    [/opt/grid/product/11gR2/grid/bin/cssdmonitor(5144)]CRS-5822:Agent '/opt/grid/product/11gR2/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
    2012-06-29 08:48:49.713
    [ohasd(4346)]CRS-2112:The OLR service started on node node001.
    2012-06-29 08:48:50.010
    [ohasd(4346)]CRS-8017:location: /etc/oracle/lastgasp has 50 reboot advisory log files, 0 were announced and 0 errors occurred
    2012-06-29 08:48:55.134
    [ohasd(4346)]CRS-2772:Server 'node001' has been assigned to pool 'Free'.
    [client(4634)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(4652)]CRS-10001:ACFS-9322: done.
    2012-06-29 08:50:59.362
    [ohasd(4346)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.oracssdmonitor_root.log
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2012-06-29 08:48:51.003: [    AGFW][1810312288] Starting the agent: /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/
    2012-06-29 08:48:51.003: [   AGENT][1810312288] Agent framework initialized, Process Id = 4414
    2012-06-29 08:48:51.003: [ USRTHRD][1810312288] to enter agent main
    2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: setting priority to 4
    2012-06-29 08:48:54.939: [ USRTHRD][1810312288]  *** Agent Framework Started ***
    2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_vmondisv: Compatible vendor clusterware not in use
    2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_thrdspawn: spawn OMON succ
    2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: spawn omon succ
    2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: default values for omon, initrate 1000, pollrate 500.
    2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: starting omon
    2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
    2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
    2012-06-29 08:48:55.041: [    AGFW][1810312288] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
    2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Inited lsf context 0x1ae1cdd0
    2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Initing CLS Framework messaging
    2012-06-29 08:48:55.041: [CLSFRAME][1810312288] New Framework state: 2
    2012-06-29 08:48:55.041: [CLSFRAME][1810312288] M2M is starting...
    2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] m_pClscCtx=0x1ae21d70m_pUgblm=0x1ae754b0
    2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] Starting send thread
    2012-06-29 08:48:55.052: [ CRSCOMM][1119893824] clsIpc: sendWork thread started.
    2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] IPC Client thread started listening
    2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] init data sent from server
    2012-06-29 08:48:55.057: [CLSFRAME][1810312288] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:node001
    2012-06-29 08:48:55.058: [CLSFRAME][1810312288] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:node001
    2012-06-29 08:48:55.066: [CLSFRAME][1810312288] Starting thread model named: MultiThread
    2012-06-29 08:48:55.073: [CLSFRAME][1810312288] Starting thread model named: SingleThread
    2012-06-29 08:48:55.076: [CLSFRAME][1810312288] Starting thread model named: SingleThreadT
    2012-06-29 08:48:55.080: [CLSFRAME][1810312288] New Framework state: 3
    2012-06-29 08:48:55.080: [    AGFW][1810312288] Agent Framework started successfully
    2012-06-29 08:48:55.080: [    AGFW][1182832960] Agfw engine module has enabled...
    2012-06-29 08:48:55.080: [CLSFRAME][1182832960] Module Enabling is complete
    2012-06-29 08:48:55.080: [CLSFRAME][1182832960] New Framework state: 6
    2012-06-29 08:48:55.080: [    AGFW][1182832960] Agent is started with userid: root , expected user: root
    2012-06-29 08:48:55.081: [    AGFW][1182832960] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
    2012-06-29 08:48:55.088: [    AGFW][1182832960] Agent received the message: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
    2012-06-29 08:48:55.089: [    AGFW][1182832960] Added new restype: ora.cssdmonitor.type
    2012-06-29 08:48:55.089: [    AGFW][1182832960] Agent sending last reply for: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
    2012-06-29 08:48:55.090: [    AGFW][1182832960] Agent received the message: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
    2012-06-29 08:48:55.090: [    AGFW][1182832960] Added new resource: ora.cssdmonitor 1 1 to the agfw
    2012-06-29 08:48:55.091: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
    2012-06-29 08:48:55.091: [    AGFW][1182832960] Agent received the message: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
    2012-06-29 08:48:55.091: [    AGFW][1182832960] Preparing CHECK command for: ora.cssdmonitor 1 1
    2012-06-29 08:48:55.096: [    AGFW][1172343104] Executing command: check for resource: ora.cssdmonitor 1 1
    2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_INITRATE, value 1000
    2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_POLLRATE, value 500
    2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr ORA_OPROCD_MODE, value
    2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr PROCD_TIMEOUT, value 1000
    2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr LOGGING_LEVEL, value 1
    2012-06-29 08:48:55.096: [    AGFW][1172343104] check for resource: ora.cssdmonitor 1 1 completed with status: OFFLINE
    2012-06-29 08:48:55.096: [CRSTIMER][1203812672] Timer Thread Starting.
    2012-06-29 08:48:55.096: [    AGFW][1182832960] ora.cssdmonitor 1 1 state changed from: UNKNOWN to: OFFLINE
    2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
    2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
    2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:28
    2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent received the message: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
    2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
    2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
    2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:34
    2012-06-29 08:48:55.101: [    AGFW][1182832960] ora.cssdmonitor 1 1 marked as deleted.
    2012-06-29 08:48:55.101: [    AGFW][1182832960] Deleting the resource: ora.cssdmonitor 1 1
    2012-06-29 08:48:55.101: [    AGFW][1182832960] Agent is commiting suicide.
    2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: CSSD Agent was asked to exit with exit code 1
    2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_becomeactive: Signaling active state
    2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_term: still threads up (1)
    2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: Agent is done.
    2012-06-29 08:48:55.101: [    AGFW][1182832960] Agent is exiting with exit code: 1
    2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
    2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
    2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
    2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29Edited by: Bernhard W on 03-Jul-2012 02:14

    I am really getting desperate about this.
    The ASM persmissions are on both nodes the same, so this should not be the problem and i can not find any problem on the network.
    The log files of the healthy node do not say anything about the node which does not startup now, the last log entry was from 6 months ago, when i shut that node down. There seems to be no communication between the two nodes now.
    I tried to manually start the ora.cssd but somehow ora.gipcd just does not start and i can not find the problem.
    [oracle@dmrs11srvcla001 ~]$ crsctl stat res -t -init
    NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
    Cluster Resources
    ora.asm
          1        ONLINE  OFFLINE                                                  
    ora.crsd
          1        ONLINE  INTERMEDIATE dmrs11srvcla001                             
    ora.cssd
          1        ONLINE  OFFLINE                                                  
    ora.cssdmonitor
          1        ONLINE  ONLINE       dmrs11srvcla001                             
    ora.ctssd
          1        ONLINE  OFFLINE                                                  
    ora.diskmon
          1        OFFLINE OFFLINE                                                  
    ora.drivers.acfs
          1        ONLINE  ONLINE       dmrs11srvcla001                             
    ora.evmd
          1        ONLINE  OFFLINE                                                  
    ora.gipcd
          1        ONLINE  OFFLINE                                                  
    ora.gpnpd
          1        ONLINE  OFFLINE                                                  
    ora.mdnsd
          1        ONLINE  ONLINE       dmrs11srvcla001                             
    [oracle@dmrs11srvcla001 ~]$ crsctl start res ora.cssd -init
    CRS-2672: Attempting to start 'ora.gipcd' on 'dmrs11srvcla001'
    Start action for daemon aborted
    CRS-2674: Start of 'ora.gipcd' on 'dmrs11srvcla001' failed
    CRS-2679: Attempting to clean 'ora.gipcd' on 'dmrs11srvcla001'
    CRS-2681: Clean of 'ora.gipcd' on 'dmrs11srvcla001' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'dmrs11srvcla001'
    Start action for daemon aborted
    CRS-2674: Start of 'ora.gpnpd' on 'dmrs11srvcla001' failed
    CRS-2679: Attempting to clean 'ora.gpnpd' on 'dmrs11srvcla001'
    CRS-2681: Clean of 'ora.gpnpd' on 'dmrs11srvcla001' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'dmrs11srvcla001'
    CRS-2672: Attempting to start 'ora.diskmon' on 'dmrs11srvcla001'
    CRS-2676: Start of 'ora.diskmon' on 'dmrs11srvcla001' succeededalertnode001.log
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-02 17:11:40.789
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-02 17:11:41.654
    [ohasd(4351)]CRS-2765:Resource 'ora.mdnsd' has failed on server 'node001'.
    2012-07-02 17:11:43.404
    [cssd(4997)]CRS-1713:CSSD daemon is started in clustered mode
    2012-07-02 17:11:46.634
    [cssd(4997)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-07-02 17:11:46.745
    [ohasd(4351)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'node001'.
    2012-07-02 17:12:42.384
    [/opt/grid/product/11gR2/grid/bin/orarootagent.bin(4488)]CRS-5818:Aborted command 'start for resource: ora.diskmon 1 1' for resource 'ora.diskmon'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/orarootagent_root/orarootagent_root.log.
    2012-07-02 17:12:46.390
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.diskmon'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-02 17:14:50.768
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-02 17:16:55.819
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-02 17:16:58.354
    [cssd(5137)]CRS-1713:CSSD daemon is started in clustered mode
    2012-07-02 17:16:58.425
    [cssd(5137)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-07-02 17:25:23.194
    [ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
    2012-07-02 17:25:23.200
    [ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-07-02 17:26:57.384
    [/opt/grid/product/11gR2/grid/bin/cssdagent(5116)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-07-02 17:27:02.514
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-02 17:37:11.971
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.evmd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 02:13:57.821
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 05:14:37.915
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 05:44:44.630
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 06:14:51.303
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 06:44:57.981
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 07:15:04.655
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 07:42:45.015
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-03 07:44:50.068
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
    2012-07-03 07:44:52.369
    [cssd(10209)]CRS-1713:CSSD daemon is started in clustered mode
    2012-07-03 07:44:52.434
    [cssd(10209)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-07-03 07:45:11.331
    [ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
    2012-07-03 07:53:17.222
    [ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
    2012-07-03 07:53:17.222
    [ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-07-03 07:54:51.401
    [/opt/grid/product/11gR2/grid/bin/cssdagent(10188)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-07-03 07:54:56.524
    [ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.ohasd.log
    2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:24919
    2012-07-03 07:45:11.330: [    AGFW][1347250496] Received state change for ora.crsd 1 1 [old state = PARTIAL, new state = OFFLINE]
    2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23258]
    2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:24919
    2012-07-03 07:45:11.330: [   CRSPE][1357756736] State change received from node001 for ora.crsd 1 1
    2012-07-03 07:45:11.331: [   CRSPE][1357756736] Processing PE command id=139. Description: [Resource State Change (ora.crsd 1 1) : 0x1c284d40]
    2012-07-03 07:45:11.331: [   CRSPE][1357756736] RI [ora.crsd 1 1] new external state [OFFLINE] old value: [INTERMEDIATE] on node001 label = []
    2012-07-03 07:45:11.331: [   CRSPE][1357756736] Resource Resource Instance ID[ora.crsd 1 1]. Values:
    STATE=OFFLINE
    TARGET=ONLINE
    LAST_SERVER=node001
    CURRENT_RCOUNT=1
    LAST_RESTART=1341292505
    FAILURE_COUNT=0
    FAILURE_HISTORY=
    STATE_DETAILS=
    INCARNATION=29
    STATE_CHANGE_VERS=29
    LAST_FAULT=1341292504
    DEGREE_ID=1
    ID=ora.crsd 1 1
    CARDINALITY_ID=1
    Lock Info:
    Write Locks:none
    ReadLocks:|STATE INITED||INITIAL CHECK DONE| has failed!
    2012-07-03 07:45:11.331: [   CRSPE][1357756736] Processing unplanned state change for [ora.crsd 1 1]
    2012-07-03 07:45:11.331: [   CRSPE][1357756736] ora.crsd 1 1: uptime exceeds uptime threshold , resetting restart count
    2012-07-03 07:45:11.331: [   CRSPE][1357756736] Local Recovery picked for [ora.crsd 1 1]
    2012-07-03 07:45:11.333: [   CRSPE][1357756736] Sending message to agfw: id = 23260
    2012-07-03 07:45:11.333: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260
    2012-07-03 07:45:11.333: [   CRSPE][1357756736] CRS-2672: Attempting to start 'ora.crsd' on 'node001'
    2012-07-03 07:45:11.333: [    AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260 to the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
    2012-07-03 07:45:12.349: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
    2012-07-03 07:45:12.350: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
    2012-07-03 07:45:12.350: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23260
    2012-07-03 07:45:12.350: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
    2012-07-03 07:45:12.350: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
    2012-07-03 07:45:12.350: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23260
    2012-07-03 07:45:12.351: [   CRSPE][1357756736] RI [ora.crsd 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on node001 label = []
    2012-07-03 07:45:12.351: [   CRSPE][1357756736] CRS-2676: Start of 'ora.crsd' on 'node001' succeeded
    2012-07-03 07:45:12.351: [   CRSPE][1357756736] PE Command [ Resource State Change (ora.crsd 1 1) : 0x1c284d40 ] has completed
    2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23269
    2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23269
    2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw received reply from PE for resource state change for ora.crsd 1 1
    2012-07-03 07:53:17.220: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:25353
    2012-07-03 07:53:17.221: [    AGFW][1347250496] Received state change for ora.diskmon 1 1 [old state = ONLINE, new state = PLANNED_OFFLINE]
    2012-07-03 07:53:17.221: [    AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23497]
    2012-07-03 07:53:17.221: [    AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:25353
    2012-07-03 07:53:17.221: [   CRSPE][1357756736] State change received from node001 for ora.diskmon 1 1
    2012-07-03 07:53:17.221: [   CRSPE][1357756736] Processing PE command id=140. Description: [Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0]
    2012-07-03 07:53:17.221: [   CRSPE][1357756736] RI [ora.diskmon 1 1] new external state [OFFLINE] old value: [ONLINE] on node001 label = []
    2012-07-03 07:53:17.221: [   CRSPE][1357756736] RI [ora.diskmon 1 1] new target state: [OFFLINE] old value: [ONLINE]
    2012-07-03 07:53:17.222: [  CRSOCR][1351452992] Multi Write Batch processing...
    2012-07-03 07:53:17.222: [   CRSPE][1357756736] Resource Resource Instance ID[ora.diskmon 1 1]. Values:
    STATE=OFFLINE
    TARGET=OFFLINE
    LAST_SERVER=node001
    CURRENT_RCOUNT=0
    LAST_RESTART=1341294297
    FAILURE_COUNT=0
    FAILURE_HISTORY=
    STATE_DETAILS=
    INCARNATION=2
    STATE_CHANGE_VERS=5
    LAST_FAULT=1341242723
    DEGREE_ID=1
    ID=ora.diskmon 1 1
    CARDINALITY_ID=1
    Lock Info:
    Write Locks:START of [ora.cssd 1 1] on [node001] : 0x2aaab000ba10
    ReadLocks:|STATE INITED| has failed!
    2012-07-03 07:53:17.222: [   CRSPE][1357756736] Processing unplanned state change for [ora.diskmon 1 1]
    2012-07-03 07:53:17.222: [   CRSPE][1357756736] Target is not ONLINE, not recovering [ora.diskmon 1 1]
    2012-07-03 07:53:17.222: [   CRSPE][1357756736] PE Command [ Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0 ] has completed
    2012-07-03 07:53:17.222: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23501
    2012-07-03 07:53:17.222: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23501
    2012-07-03 07:53:17.223: [    AGFW][1347250496] Agfw received reply from PE for resource state change for ora.diskmon 1 1
    2012-07-03 07:53:17.223: [  CRSOCR][1351452992] Multi Write Batch done.
    2012-07-03 07:54:56.523: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:54:56.523: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
    2012-07-03 07:54:56.524: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23220
    2012-07-03 07:54:56.524: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:54:56.524: [   CRSPE][1357756736]Created alert : (:CRSPE00111:) :  Start action timed out!
    2012-07-03 07:54:56.524: [   CRSPE][1357756736] Start action failed with error code: 3
    2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
    2012-07-03 07:54:56.524: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23220
    2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
    2012-07-03 07:54:56.524: [    AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
    2012-07-03 07:54:56.524: [   CRSPE][1357756736] CRS-2674: Start of 'ora.cssd' on 'node001' failed
    2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
         MESSAGE:
         TextMessage[CRS-2674: Start of 'ora.cssd' on 'node001' failed]
         MSGTYPE:
         TextMessage[1]
         OBJID:
         TextMessage[ora.cssd]
         WAIT:
         TextMessage[0]
    2012-07-03 07:54:56.525: [   CRSPE][1357756736] Sending message to agfw: id = 23555
    2012-07-03 07:54:56.525: [   CRSPE][1357756736] CRS-2679: Attempting to clean 'ora.cssd' on 'node001'
    2012-07-03 07:54:56.525: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
    2012-07-03 07:54:56.525: [    AGFW][1347250496] Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is being stopped. Will be restarted when agent stops completly
    2012-07-03 07:54:56.525: [    AGFW][1347250496] Could not forward message [RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555] to agent. /opt/grid/product/11gR2/grid/bin/cssdagent_root is not running
    2012-07-03 07:54:56.525: [    AGFW][1347250496] Can not start Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is waiting to be restarted.
    2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
         MESSAGE:
         TextMessage[CRS-2679: Attempting to clean 'ora.cssd' on 'node001']
         MSGTYPE:
         TextMessage[3]
         OBJID:
         TextMessage[ora.cssd]
         WAIT:
         TextMessage[0]
    2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
    2012-07-03 07:55:01.627: [ CRSCOMM][1340946752] Client disconnected.
    2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 14
    2012-07-03 07:55:01.628: [ CRSCOMM][1340946752] IPC listener connection to member 14 has been removed
    2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:14|Type:3}
    2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:14|Type:3}
    2012-07-03 07:55:01.628: [   CRSPE][1357756736] Disconnected from server:
    2012-07-03 07:55:01.628: [    AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
    2012-07-03 07:55:01.628: [    AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
    2012-07-03 07:55:01.628: [    AGFW][1347250496] Restarting the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.628: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10188] stopped!
    2012-07-03 07:55:01.628: [ CRSCOMM][1347250496] removeConnection: Member 14 does not exist.
    2012-07-03 07:55:01.628: [    AGFW][1347250496] Starting the agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id: root and incarnation:7
    2012-07-03 07:55:01.632: [    AGFW][1347250496] Starting the HB [Interval =  30000, misscount = 10kill allowed=0] for agent: /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New IPC Member:{Relative|Node:0|Process:15|Type:3}:AGENT
    2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New process connected to us ID:{Relative|Node:0|Process:15|Type:3} Info:AGENT
    2012-07-03 07:55:01.837: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
    2012-07-03 07:55:01.837: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root with pid:10530 connected to server.
    2012-07-03 07:55:01.837: [    AGFW][1347250496] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.838: [    AGFW][1347250496] Agfw Proxy Server sending message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.838: [    AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.839: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
    2012-07-03 07:55:01.851: [    AGFW][1347250496] Received the reply to the message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.854: [    AGFW][1347250496] Received the reply to the message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.861: [    AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.862: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
    2012-07-03 07:55:01.862: [    AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.862: [   CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
    2012-07-03 07:55:01.862: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
    2012-07-03 07:55:01.863: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
    2012-07-03 07:55:01.863: [   CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
    2012-07-03 07:55:01.863: [    AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
    2012-07-03 07:55:01.863: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
    2012-07-03 07:55:01.863: [   CRSPE][1357756736] CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded
    2012-07-03 07:55:01.863: [   CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
    2012-07-03 07:55:01.863: [UiServer][1361959232] Container [ Name: ORDER
         MESSAGE:
         TextMessage[CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded]
         MSGTYPE:
         TextMessage[3]
         OBJID:
         TextMessage[ora.cssd]
         WAIT:
         TextMessage[0]
    2012-07-03 07:55:01.864: [   CRSPE][1357756736] PE Command [ Start Resource : 0x2aaab0009d00 ] has completed
    2012-07-03 07:55:01.864: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23606
    2012-07-03 07:55:01.864: [   CRSPE][1357756736] UI Command [Start Resource : 0x2aaab0009d00] is replying to sender.
    2012-07-03 07:55:01.864: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23606
    2012-07-03 07:55:01.864: [UiServer][1361959232] Container [ Name: UI_DATA
         ora.cssd:
         TextMessage[215]
    2012-07-03 07:55:01.864: [UiServer][1361959232] Done for ctx=0x2aaab004ece0
    2012-07-03 07:55:01.869: [UiServer][1364060480] Closed: remote end failed/disc.
    2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
    2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] Client disconnected.
    2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 15
    2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] IPC listener connection to member 15 has been removed
    2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:15|Type:3}
    2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:15|Type:3}
    2012-07-03 07:55:03.792: [   CRSPE][1357756736] Disconnected from server:
    2012-07-03 07:55:03.792: [    AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
    2012-07-03 07:55:03.792: [    AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
    2012-07-03 07:55:03.792: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10530] stopped!
    2012-07-03 07:55:03.792: [ CRSCOMM][1347250496] removeConnection: Member 15 does not exist.

  • 11gR2 : Failed to start EVM while running root.sh in Node1

    DB verision:11GR2
    Platform : Solaris 5.10 (Amd64)
    I've reached the last step in the installation in Node1
    # ./root.sh
    .............all messages say succeeded till here............
    CRS-2676: Start of 'ora.crsd' on 'hudson1' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'hudson1'
    Start action for daemon aborted
    CRS-2674: Start of 'ora.evmd' on 'hudson1' failed
    CRS-2679: Attempting to clean 'ora.evmd' on 'hudson1'
    CRS-2681: Clean of 'ora.evmd' on 'hudson1' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/grid/bin/crsctl start resource ora.evmd -init
    Start of resource "ora.evmd -init" failed
    Failed to start EVM
    Failed to start Oracle Clusterware stackA search in OTN and MOS returned nothing. Any idea how i could diagnose this issue?

    Hi,
    Check this note on MOS.
    *Troubleshooting 11.2 Grid Infastructure Installation Root.sh Issues [ID 1053970.1]*
    A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.
    Regards,
    Levi Pereira

  • CRS-0215: Could not start resource

    Hello
    This is my first time installing clusterware. However, i have not been too successful at it. This is my configuration:
    Operating System: RHES 5.3
    Oracle Database 11gR1
    OpenFiler used to configure shared disks
    After several attempts, i was able to run the root.sh script on both nodes.
    Output on first node:
    [root@vx0301 oracle]# /u01/app/oraInventory/orainstRoot.sh
    Changing permissions of /u01/app/oraInventory to 770.
    Changing groupname of /u01/app/oraInventory to oinstall.
    The execution of the script is complete
    [root@vx0301 oracle]# /u01/crs11g/root.sh
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    clscfg: EXISTING configuration version 4 detected.
    clscfg: version 4 is 11 Release 1.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: vx0301 vx0301-priv vx0301
    node 2: vx0302 vx0302-priv vx0302
    clscfg: Arguments check out successfully.
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Cluster Synchronization Services is active on these nodes.
    vx0301
    Cluster Synchronization Services is inactive on these nodes.
    vx0302
    Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
    Output on second node:
    [root@vx0301 oracle]# /u01/app/oraInventory/orainstRoot.sh
    Changing permissions of /u01/app/oraInventory to 770.
    Changing groupname of /u01/app/oraInventory to oinstall.
    The execution of the script is complete
    [root@vx0301 oracle]# /u01/crs11g/root.sh
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    clscfg: EXISTING configuration version 4 detected.
    clscfg: version 4 is 11 Release 1.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: vx0301 vx0301-priv vx0301
    node 2: vx0302 vx0302-priv vx0302
    clscfg: Arguments check out successfully.
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Cluster Synchronization Services is active on these nodes.
    vx0301
    vx0302
    I also got the errors:
    Starting GSD application resource on (2) nodes1:CRS-0215: Could not start resource 'ora.vx0301.gsd'
    Starting ONS application resource on (2) nodes1:CRS-0215: Could not start resource 'ora.vx0301.ons'
    There were no log files.
    When i clicked ok on the execute scripts page, i got the following errors in the configuration assistants page:
    Output generated from configuration assistant "Oracle Notification Server Configuration Assistant":
    Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Configuration assistant "Oracle Notification Server Configuration Assistant" failed
    The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------Output generated from configuration assistant "Oracle Notification Server Configuration Assistant":
    Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Configuration assistant "Oracle Notification Server Configuration Assistant" failed
    The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------Output generated from configuration assistant "Oracle Notification Server Configuration Assistant" (attempt 2):
    Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Configuration assistant "Oracle Notification Server Configuration Assistant" failed
    The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------
    Contents of the /u01/crs11g/cfgtoollogs/configToolFailedCommands script:
    [oracle@vx0301 ~]$ cat /u01/crs11g/cfgtoollogs/configToolFailedCommands
    # Copyright (c) 1999, 2007, Oracle. All rights reserved.
    /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    /u01/crs11g/bin/oifcfg setif -global eth0/172.30.4.0:public eth1/192.168.1.0:cluster_interconnect
    /u01/crs11g/bin/cluvfy stage -post crsinst -n vx0301,vx0302
    I tried running /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251 manually from the terminal:
    [root@vx0301 oracle]# /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Output of the ons.log file:
    [root@vx0301 oracle]# cat /u01/crs11g/log/vx0301/racg/ons.log
    Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
    2010-01-25 13:59:15.786: [    RACG][3055679168] [10113][3055679168][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    2010-01-25 13:59:33.359: [    RACG][3055584960] [10229][3055584960][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    2010-01-25 14:01:00.319: [    RACG][3086849728] [10734][3086849728][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    2010-01-25 14:02:02.723: [    RACG][3086628544] [11105][3086628544][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    [root@vx0301 oracle]#
    I am absoutely stumped as to what to do next. Any help is greatly appreciated.

    I tried running ruclufvy.sh stage -post crsinst -n vx0301,vx0302 -verbose. Here is the output:
    [oracle@vx0301 clusterware]$ ./runcluvfy.sh stage -post crsinst -n vx0301,vx0302 -verbose
    Performing post-checks for cluster services setup
    Checking node reachability...
    Check: Node reachability from node "vx0301"
    Destination Node Reachable?
    vx0301 yes
    vx0302 yes
    Result: Node reachability check passed from node "vx0301".
    Checking user equivalence...
    Check: User equivalence for user "oracle"
    Node Name Comment
    vx0302 passed
    vx0301 passed
    Result: User equivalence check passed for user "oracle".
    Checking Cluster manager integrity...
    Checking CSS daemon...
    Node Name Status
    vx0302 running
    vx0301 running
    Result: Daemon status check passed for "CSS daemon".
    Cluster manager integrity check passed.
    Checking cluster integrity...
    Node Name
    vx0301
    vx0302
    Cluster integrity check passed
    Checking OCR integrity...
    Checking the absence of a non-clustered configuration...
    All nodes free of non-clustered, local-only configurations.
    Uniqueness check for OCR device passed.
    Checking the version of OCR...
    OCR of correct Version "2" exists.
    Checking data integrity of OCR...
    ERROR:
    OCR integrity is invalid.
    OCR integrity check failed.
    Checking CRS integrity...
    Checking daemon liveness...
    Check: Liveness for "CRS daemon"
    Node Name Running
    vx0302 yes
    vx0301 yes
    Result: Liveness check passed for "CRS daemon".
    Checking daemon liveness...
    Check: Liveness for "CSS daemon"
    Node Name Running
    vx0302 yes
    vx0301 yes
    Result: Liveness check passed for "CSS daemon".
    Checking daemon liveness...
    Check: Liveness for "EVM daemon"
    Node Name Running
    vx0302 yes
    vx0301 yes
    Result: Liveness check passed for "EVM daemon".
    Liveness of all the daemons
    Node Name CRS daemon CSS daemon EVM daemon
    vx0302 yes yes yes
    vx0301 yes yes yes
    Checking CRS health...
    Check: Health of CRS
    Node Name CRS OK?
    vx0302 yes
    vx0301 unknown
    Result: CRS health check failed.
    CRS integrity check failed.
    Checking node application existence...
    Checking existence of VIP node application
    Node Name Required Status Comment
    vx0302 yes exists passed
    vx0301 yes exists passed
    Result: Check passed.
    Checking existence of ONS node application
    Node Name Required Status Comment
    vx0302 no exists passed
    vx0301 no exists passed
    Result: Check passed.
    Checking existence of GSD node application
    Node Name Required Status Comment
    vx0302 no exists passed
    vx0301 no exists passed
    Result: Check passed.
    Post-check for cluster services setup was unsuccessful on all the nodes.
    [oracle@vx0301 clusterware]$
    I also tried running ./ocrchek on both nodes. Here is the output:
    [oracle@vx0301 bin]$ ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
    Version : 2
    Total space (kbytes) : 3306636
    Used space (kbytes) : 2056
    Available space (kbytes) : 3304580
    ID : 1425438992
    Device/File Name : /dev/sdb1
    Device/File integrity check failed
    Device/File not configured
    Cluster registry integrity check failed
    [oracle@vx0301 bin]$ ssh vx0302
    Last login: Mon Jan 25 13:52:03 2010 from vx0301
    [oracle@vx0302 ~]$ cd /u01/crs11g/bin/
    [oracle@vx0302 bin]$ ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
    Version : 2
    Total space (kbytes) : 3306636
    Used space (kbytes) : 2056
    Available space (kbytes) : 3304580
    ID : 1425438992
    Device/File Name : /dev/sdb1
    Device/File integrity check failed
    Device/File not configured
    Cluster registry integrity check failed
    [oracle@vx0302 bin]$
    From the results, the ids are the same, and so is the OCR disk. But the integrity check has failed. How do i interpret this output?

  • CRS-0223: Resource 'ora.xt3xx.ons' has placement error.

    All,
    I recently downgraded my RAC 10.2.0.4 database to 10.2.0.1 . and after that I am not able to start the database using srvctl . however i am able to start the db manually.
    this is the error message
    srvctl start database -d RDBE
    PRKP-1001 : Error starting instance RDBE1 on node xt33db006
    CRS-0215: Could not start resource 'ora.RDBE.RDBE1.inst'.
    PRKP-1001 : Error starting instance RDBE2 on node xt33db007
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Fri May 16 09:32:30 2008
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:Enter user-name: Connected to an idle instance.
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:SQL> ASM instance started
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:Total System Global Area 130023424 bytes
    xt33db007:ora.xt33db007.ASM2.asm:Fixed Size 1976920 bytes
    xt33db007:ora.xt33db007.ASM2.asm:Variable Size 102880680 bytes
    xt33db007:ora.xt33db007.ASM2.asm:ASM Cache 25165824 bytes
    xt33db007:ora.xt33db007.ASM2.asm:ASM diskgroups mounted
    xt33db007:ora.xt33db007.ASM2.asm:SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
    xt33db007:ora.xt33db007.ASM2.asm:With the Partitioning, Real Application Clusters, OLAP and Data Mining options
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Fri May 16 09:32:53 2008
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:Enter user-name:
    xt33db007:ora.xt33db007.ASM2.asm:Connected to:
    xt33db007:ora.xt33db007.ASM2.asm:Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
    xt33db007:ora.xt33db007.ASM2.asm:With the Partitioning, Real Application Clusters, OLAP and Data Mining options
    xt33db007:ora.xt33db007.ASM2.asm:
    xt33db007:ora.xt33db007.ASM2.asm:SQL> ASM diskgroups dismounted
    xt33db007:ora.xt33db007.ASM2.asm:ASM instance shutdown
    xt33db007:ora.xt33db007.ASM2.asm:SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
    xt33db007:ora.xt33db007.ASM2.asm:With the Partitioning, Real Application Clusters, OLAP and Data Mining options
    xt33db007:ora.xt33db007.ASM2.asm:
    CRS-0215: Could not start resource 'ora.RDBE.RDBE2.inst'.
    please advise.
    thanks
    Pk

    What is the error message you are seeing in CRS & RACG logs related to the database startup.?
    What is the STATE & TARGET fields showing up for the database and instances?
    Please paste the srvm log for the srvctl command.
    export SRVM_TRACE=TRUE
    srvctl start database -d RDBE
    exit

Maybe you are looking for

  • Accounts payable: Reconciliation field(s)

    Dear all, I have posted the same thread in the Materials Management forum as I am not sure where it belongs. As I am not very familiar with vendor payments, I hope you can help me with this question. As per our accounts payable department, when they

  • 3rd gen ipod won't connect with firewire.

    3rd gen ipod won't connect to itunes with firewire. USB connects rarely but drains battery quickly. I've rebooted the ipod and computer and updated itunes. No effect. Any remedies besides buying a new ipod.

  • Restricting HTTP Port on J2EE Engine

    I have to disable the HTTP Port on Web AS J2EE Engine and allow connections only through https. Is it possible to do so ? If yes, how ? Thanks for the help in advance.

  • How to split string into element ?

    How do you split common string ?? String dd = "aaa"; dd.split("a");  // result ["a","a","a"]

  • Windows 7 Instalation Problem - Error Code 0xC0000005

    I have been trying to install Windows 7 on my, rather old, laptop and it barely gets to the installation screen after inserting the CD into the CD drive when the message appears: Windows installation encountered an unexpected error. Verify that the i