Crs Not Starting _ private Interconnect Down

Hello All,
I Have Installed 2 node 10g R2(10.2.0.1) RAC on Solaris 10 T2000 Machines. Yesterday my Second Node Crs gone down. I tried to start it but it didn't start. Then i checked that Private IP (interconnect) is not Pinging from both the node. But Node 1 was up and working so my Users Can Connect to It.
But Today morning I see that Crs on node 1aslo goes down .
Is this is problem of private interconnect.? My network guys are trying to up Private Interconnect.
If Private Interconnect is down, why node 1 goes down after few hours. i think private interconnect is for interconnect with node 2 but node 2 is down .
Previously My interconnect was connected with cross cables now i have asked them to connect them through switch.
Help me Out.
Regards,
Pankaj.

Previously My interconnect was connected with cross cables now i have asked them to connect them through switch
Even we are planning to do the same.Please share your experienceHope you have done this before - moving to switch
(Update for record id(s): 105681546)
QUESTION
========
1.Will the database and the Clusterware need to be shutdown etc?
2.Will our ip addresses need to be reconfigured?
3.Are there any steps that need to be carried out before unplugging the CROSS CABLE
and after the interconnect is connected to the switch...?
ANSWER
======
1. Yes, you have to stop CRS on each node.
2. No, not required.Provided you are planning to use same ip addresses.
3. Steps:
a. Stop CRS on each node. "crsctl stop crs"
b. Replace the crossover cable with switch.
c. Start the CRS on each node. "crsctl start crs"
Even we are planning to do the same.Please share your experienceFollowed by the above answers from Customer Support, It went smooth, we stopped all the services, and with both the nodes reboot.
Message was edited by:
Ravi Prakash

Similar Messages

  • CRS not starting after Oracle 11g Grid was succesuflly installed

    Hi:
    Have 2 node RAC. My Shared storage is on openfiler.
    Oracle 11g Grid was successfully installed and all the services was working fine. When i restarted the system I found CRS was not coming up. Remaining processes were up and running.
    Let me explain what I did: (O/s RHEL 5.4 & oracle 11g R2 grid)
    I have a partition in shared disk /dev/sdc1 (I have other partitions, but thought of using only one for OCR and voting disk). (did not define it as raw partition as I used to do in 10g RAC).
    While installing Oracle grid and when it prompted to enter the Diskgroup for OCR and Voting disk I gave a name DGDATA and choose only one (external redundancy) disk “/dev/sdc1″ (it does not ask for one for OCR and one for Voting as it used to do in 10g clusterware installation). As per my assumption both OCR and Voting disk were created within /dev/sdc1 (after installation when I queried I got the same info)
    The Oracle grid installation has gone through successfully. when I restart the system I find CRS not starting.
    The log says: “Error PROC:26: Error while accessing the physical storage ASM……..
    ORA-01034: oracle not available...
    Could not init OCR, code:26….
    Linux permission denied".
    For your information….ASM instance is up (check in both the nodes) and I find the diskgroup mounted.
    Can anyone help on this.
    Also let me know how to uninstall Oracle 11g Grid software and configuration. If I have to reinstall the Oracle 11g grid software and config once again is it possible without reinstalling RHEL. Let me know the steps or any link please. don't want to get into a mess again.
    Thanks in advance.
    Regards
    Dinesh

    Jorg, I even tried to nomount the DB instance using initDBRAC.ora pfile. Then tried to create spfile still got the same error.
    The second trace file which I have pasted looks like some memory dump.
    ASM_DISKSTRING= /dev/sdc*
    ***********************************************************************1st trace file
    Fatal NI connect error 12547, connecting to:
    (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/crs/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_dbrac)(ENVS='ORACLE_HOME=/crs/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@skyit6)(HOST=skyit6)(USER=oracle))))
    VERSION INFORMATION:
    TNS for Linux: Version 11.2.0.1.0 - Production
    Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
    Time: 01-JUN-2011 01:27:07
    Tracing not turned on.
    Tns error struct:
    ns main err code: 12547
    TNS for Linux: Version 11.2.0.1.0 - Production
    Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
    Time: 01-JUN-2011 01:27:07
    Tracing not turned on.
    Tns error struct:
    ns main err code: 12547
    TNS-12547: TNS:lost contact
    ns secondary err code: 12560
    nt main err code: 517
    TNS-00517: Lost contact
    nt secondary err code: 32
    nt OS err code: 0
    ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/crs/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_dbrac)(ENVS='ORACLE_HOME=/crs/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
    Wed Jun 01 01:27:07 2011
    ERROR: slave communication error with ASM; terminating process 16772
    Errors in file /oraeng/app/oracle/product/diag/rdbms/dbrac/DBRAC/trace/DBRAC_ora_16772.trc:
    ----======================2nd trace file
    Trace file /oraeng/app/oracle/product/diag/rdbms/dbrac/DBRAC/trace/DBRAC_ora_16772.trc
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
    With the Real Application Clusters option
    ORACLE_HOME = /oraeng/app/oracle/product/11.2.0
    System name: Linux
    Node name: skyit6
    Release: 2.6.18-164.el5
    Version: #1 SMP Tue Aug 18 15:51:54 EDT 2009
    Machine: i686
    Instance name: DBRAC
    Redo thread mounted by this instance: 0 <none>
    Oracle process number: 25
    Unix process pid: 16772, image: oracle@skyit6 (TNS V1-V3)
    *** 2011-06-01 01:27:07.010
    *** SESSION ID:(1.3) 2011-06-01 01:27:07.010
    *** CLIENT ID:() 2011-06-01 01:27:07.010
    *** SERVICE NAME:() 2011-06-01 01:27:07.010
    *** MODULE NAME:(sqlplus@skyit6 (TNS V1-V3)) 2011-06-01 01:27:07.010
    *** ACTION NAME:() 2011-06-01 01:27:07.010
    ERROR: slave communication error with ASM; terminating process 16772
    *** 2011-06-01 01:27:07.035
    dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=2, mask=0x0)
    ----- Error Stack Dump -----
    ----- Current SQL Statement for this session (sql_id=bff5sku8phfgx) -----
    create spfile='+dgasmspfile/spfileDBRAC.ora' from pfile
    *** 2011-06-01 01:27:07.070
    ----- Call Stack Trace -----
    calling call entry argument values in hex
    location type point (? means dubious value)
    skdstdst()+41 call kgdsdst() BF9E05FC ? 2 ?
    ksedst1()+77 call skdstdst() BF9E05FC ? 0 ? 1 ? AB80E70 ?
    852F42A ? AB80E70 ?
    ksedst()+33 call ksedst1() 0 ? 1 ?
    dbkedDefDump()+2699 call ksedst() 0 ? 0 ? BF9E0D40 ? 0 ? 0 ?
    0 ?
    ksedmp()+47 call dbkedDefDump() 2 ? 0 ?
    kfTerminateMe()+75 call ksedmp() 2 ? 3 ? F1170AC ? F149234 ?
    5 ? BF9E0A6C ?
    kfnCheckCommError() call kfTerminateMe() F149234 ? BF9E0C1C ?
    +32 B033D83 ? 29392378 ?
    F149234 ? EB00601 ?
    kfncSlaveSubmit()+5 call kfnCheckCommError() 29392378 ? F149234 ?
    83
    kfncFileDelete()+11 call kfncSlaveSubmit() BF9E0E5C ? 0 ? F147B24 ?
    81
    kfioDelete()+126 call kfncFileDelete() 29DEE0 ? 1C ?
    ksfddel1()+582 call kfioDelete() 29DEE0 ? 1C ? D ?
    ksfddel()+187 call ksfddel1() 29DEE0 ? 1C ? D ? 0 ?
    BF9E0F2C ? 9BA2 ?
    ksp_spfile_create() call ksfddel() 29DEE0 ? 1C ? D ? 0 ?
    +677
    kspcspfp()+791 call ksp_spfile_create() BF9E49D4 ? 0 ? 29DEE0 ? 1C ?
    7D02 ? 0 ?
    kspocte()+67 call kspcspfp() 29DEE0 ? 1C ? F074CC4 ? F ?
    61,1 1%
    Regards
    Dinesh

  • Ipad will not start up, keeps shutting down and trying to restart, computer will not recognize device to restore.

    ipad will not start up, keeps shutting down and trying to restart, but never gets past the apple logo.  The computer will not recognize it as a device that is connected to try and restore. 

    Frozen or unresponsive iPad
    Resolve these most common issues:
        •    Display remains black or blank
        •    Touch screen not responding
        •    Application unexpectedly closes or freezes
    http://www.apple.com/support/ipad/assistant/ipad/
    iPad Frozen, not responding, how to fix
    http://appletoolbox.com/2012/07/ipad-frozen-not-responding-how-to-fix/
    iPad Frozen? How to Force Quit an App, Reset or Restart Your iPad
    http://ipadacademy.com/2010/11/ipad-frozen-how-to-force-quit-an-app-reset-or-res tart-your-ipad
    Black or Blank Screen on iPad or iPhone
    http://appletoolbox.com/2012/10/black-or-blank-screen-on-ipad-or-iphone/
    What to Do When Your iPad Won't Turn On
    http://ipad.about.com/od/iPad_Troubleshooting/ss/What-To-Do-When-Your-Ipad-Wo-No t-Turn-On.htm
    iOS: Not responding or does not turn on
    http://support.apple.com/kb/TS3281
    Home button not working or unresponsive, fix
    http://appletoolbox.com/2013/04/home-button-not-working-or-unresponsive-fix/
    Fixing an iPad Home Button
    http://tinyurl.com/om6rd6u
    iPad: Basic troubleshooting
    http://support.apple.com/kb/TS3274
     Cheers, Tom

  • CRS not starting

    OS: OEL 5 U4 x86_64
    DB: Oracle 11.2.0.1 EE
    Grid Infrastructure: Oracle 11.2.0.1
    CRS and Voting disk Storage: ASM
    Datafile and FRA storage: ASM
    I'm not sure exactly what caused this, but anyways, I changed MTU from 1500 to 900 online. After some time, 3 out of 4 nodes in the cluster went down and CRS refuses to start on these nodes after trying the switch back from MTU 9000 to 1500, reboots, and making sure disk permissions and ownership are correct. The logs are not too helpful (and cryptic) so I'm at a loss and appreciate any ideas or help.
    The installation was successful, the RAC was up for a few days while running some tests (including restart of a node). Currently only a single node has everything up and functional, the others are not working. Below are some output that might help:
    [root@ucstst11 bin]# ./crsctl check cluster -n ucstst11
    ucstst11:
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4533: Event Manager is online
    [root@ucstst11 bin]# ./crsctl start cluster -n ucstst11
    CRS-2672: Attempting to start 'ora.cssd' on 'ucstst11'
    CRS-2672: Attempting to start 'ora.diskmon' on 'ucstst11'
    CRS-2676: Start of 'ora.diskmon' on 'ucstst11' succeeded
    CRS-4404: The following nodes did not reply within the allotted time:
    ucstst11
    [root@ucstst11 bin]# ./crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4533: Event Manager is online
    [root@ucstst11 bin]# ./crsctl start crs
    CRS-4640: Oracle High Availability Services is already active
    CRS-4000: Command Start failed, or completed with errors.
    [root@ucstst11 bin]# oracleasm querydisk -p CRSVOL01
    Disk "CRSVOL01" is a valid ASM disk
    /dev/sdz1: LABEL="CRSVOL01" TYPE="oracleasm"
    /dev/sdcj1: LABEL="CRSVOL01" TYPE="oracleasm"
    [root@ucstst11 bin]# ll /dev/sdz1 /dev/sdcj1
    brw-rw---- 1 oracle dba 69, 113 Mar 27 19:00 /dev/sdcj1
    brw-rw---- 1 oracle dba 65, 145 Mar 27 19:00 /dev/sdz1
    [root@ucstst11 bin]# oracleasm querydisk -d CRSVOL01
    Disk "CRSVOL01" is a valid ASM disk on device [65, 145]
    From the functional node:
    [root@ucstst12 bin]# ./crsctl check cluster -all
    ucstst12:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    Cluster verification now hangs when it tries to contact the other nodes.
    Please help!

    For the most part this issue has been resolved. The SA partially changed to jumbo frames (OS, but not the switch), we reverted all the jumbo frame changes and the system is back online, except for one node (the one which was working ironically) not being reported via "crsctl check cluster -all", and one instance not starting due to it not seeing an interconnect (weird).
    We did attempt to fully implement jumbo frames but that did not work hence the reversion.

  • CRS not starting after reboot

    Hi,
    I have just installed RAC in a test environment using vmware server 2. I have Oracle Enterprise Linux 5 and used oracle 11g (11.2.0.1). I have not installed database software yet and just installed oracle grid infrastructure.
    When i restart any of the node and issue "crsctl check crs", i see following output.
    [root@rac2 ~]# crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    Now, "crs_stat -t" also does not work unless i manually start the crs daemon using "crsd start" command. Why crs daemon is not starting automatic unlike 10g RAC. Is it normal behaviour and what is proper way of making this daemon start automatic.
    Salman

    I am assuming you are running crsctl check crs immediately after reboot. It takes bit of time for crs to start and you do not have to start it manually. By default crs is started automatically and if for some reason you disabled it, you can enable it by using "crsctl enable crs" .
    You can also check the status of individual services by using:
    # crsctl check crsd

  • Crs not start

    Hi
    oracle cluster version 11.2
    o/s windows 2008R2
    i was install oracle cluster success and when i was check crsctl check crs all was online and clufy stage -post crsinst was success
    i left few days
    and now the out put of
    crsctl check crs is
    crs-4638 oracle high availability services is online
    crs-4535 connot communicate with cluster ready services
    crs-4530 communications failure contactin cluster synchronization services daemon
    crs-4534 connot communiccate with event manager
    please help me!
    any idea
    Regards
    Thanks in advance.

    CRSD.LOG
    =========================
    Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:591920
    2011-07-04 11:21:56.156: [    AGFW][5420] Received state change for ora.net1.network srv03 1 [old state = ONLINE, new state = OFFLINE]
    2011-07-04 11:21:56.156: [    AGFW][5420] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:Invalid|MIDFrom:0|Type:4|Pri2|Id:31719]
    2011-07-04 11:21:56.156: [    AGFW][5420] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:591920
    2011-07-04 11:21:56.156: [   CRSPE][2228] State change received from srv03 for ora.net1.network srv03 1
    2011-07-04 11:21:56.156: [   CRSPE][2228] Processing PE command id=471. Description: [Resource State Change (ora.net1.network srv03 1) : 000000000A090B00]
    2011-07-04 11:21:56.156: [   CRSPE][2228] RI [ora.net1.network srv03 1] new external state [OFFLINE] old value: [ONLINE] on srv03 label = []
    2011-07-04 11:21:56.156: [   CRSPE][2228] Resource Resource Instance ID[ora.net1.network srv03 1]. Values:
    STATE=OFFLINE
    TARGET=ONLINE
    LAST_SERVER=srv03
    CURRENT_RCOUNT=0
    LAST_RESTART=0
    FAILURE_COUNT=0
    FAILURE_HISTORY=
    STATE_DETAILS=
    INCARNATION=1
    STATE_CHANGE_VERS=0
    LAST_FAULT=0
    DEGREE_ID=1
    ID=ora.net1.network srv03 1
    Lock Info:
    Write Locks:none
    ReadLocks:|STATE INITED| has failed!
    2011-07-04 11:21:56.156: [  CRSRPT][4916] Publishing event: Cluster Resource State Change Event for ora.net1.network:srv03 : 000000000670CBB0
    2011-07-04 11:21:56.156: [  CRSRPT][4916] Publish to eons buffered event : 000000000670CBB0
    2011-07-04 11:21:56.156: [   CRSPE][2228] Processing unplanned state change for [ora.net1.network srv03 1]
    2011-07-04 11:21:56.156: [   CRSPE][2228] Scheduled local recovery for [ora.net1.network srv03 1]
    2011-07-04 11:21:56.171: [    AGFW][5420] Agfw Proxy Server received the message: RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31723
    2011-07-04 11:21:56.171: [    AGFW][5420] Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31723 to the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
    2011-07-04 11:21:56.171: [   CRSPE][2228] Sending message to agfw: id = 31726
    2011-07-04 11:21:56.171: [   CRSPE][2228] CRS-2672: Attempting to start 'ora.net1.network' on 'srv03'
    2011-07-04 11:21:56.171: [    AGFW][5420] Agfw Proxy Server received the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726
    2011-07-04 11:21:56.171: [    AGFW][5420] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726 to the agent C:\app\11.2.0\grid\bin\orarootagent.exe_system
    2011-07-04 11:21:56.187: [    AGFW][5420] Received the reply to the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31727 from the agent C:\app\11.2.0\grid\bin\orarootagent.exe_system
    2011-07-04 11:21:56.187: [    AGFW][5420] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726
    2011-07-04 11:21:56.187: [   CRSPE][2228] Received reply to action [Start] message ID: 31726
    2011-07-04 11:21:56.296: [    AGFW][5420] Received the reply to the message: RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31724 from the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
    2011-07-04 11:21:56.296: [    AGFW][5420] ora.ons srv03 1 received state from probe request. Old state = ONLINE, New state = ONLINE
    2011-07-04 11:21:56.296: [    AGFW][5420] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31723
    2011-07-04 11:21:56.296: [    AGFW][5420] Received the reply to the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31727 from the agent C:\app\11.2.0\grid\bin\orarootagent.exe_system
    2011-07-04 11:21:56.296: [    AGFW][5420] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726
    2011-07-04 11:21:56.296: [   CRSPE][2228] Received reply to action [Start] message ID: 31726
    2011-07-04 11:21:56.296: [   CRSPE][2228] CRS-2674: Start of 'ora.net1.network' on 'srv03' failed
    2011-07-04 11:21:56.296: [   CRSPE][2228] Sequencer for [ora.net1.network srv03 1] has completed with error: CRS-0215: Could not start resource 'ora.net1.network'.
    2011-07-04 11:21:56.296: [   CRSPE][2228] Failover cannot be completed for [ora.net1.network srv03 1]. Stopping it and the resource tree
    2011-07-04 11:21:56.296: [   CRSPE][2228] Sending message to agfw: id = 31735
    2011-07-04 11:21:56.296: [    AGFW][5420] Agfw Proxy Server received the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735
    2011-07-04 11:21:56.296: [    AGFW][5420] Agfw Proxy Server forwarding the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735 to the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
    2011-07-04 11:21:56.296: [   CRSPE][2228] CRS-2673: Attempting to stop 'ora.ons' on 'srv03'
    2011-07-04 11:21:56.670: [    AGFW][5420] Received the reply to the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31736 from the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
    2011-07-04 11:21:56.670: [    AGFW][5420] Agfw Proxy Server sending the reply to PE for message:RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735
    2011-07-04 11:21:56.670: [   CRSPE][2228] Received reply to action [Stop] message ID: 31735
    2011-07-04 11:21:58.854: [    AGFW][5420] Received the reply to the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31736 from the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
    2011-07-04 11:21:58.854: [    AGFW][5420] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735
    2011-07-04 11:21:58.854: [   CRSPE][2228] Received reply to action [Stop] message ID: 31735
    2011-07-04 11:21:58.854: [   CRSPE][2228] RI [ora.ons srv03 1] new external state [OFFLINE] old value: [ONLINE] label = []
    2011-07-04 11:21:58.854: [  CRSRPT][4916] Publishing event: Cluster Resource State Change Event for ora.ons:srv03 : 000000000670CAD0
    2011-07-04 11:21:58.854: [  CRSRPT][4916] Publish to eons buffered event : 000000000670CAD0
    2011-07-04 11:21:58.854: [   CRSPE][2228] CRS-2677: Stop of 'ora.ons' on 'srv03' succeeded
    2011-07-04 11:21:58.854: [   CRSPE][2228] PE Command [ Resource State Change (ora.net1.network srv03 1) : 000000000A090B00 ] has completed
    2011-07-04 11:21:58.854: [    AGFW][5420] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:31744
    2011-07-04 11:21:58.854: [    AGFW][5420] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:31744
    2011-07-04 11:21:58.854: [    AGFW][5420] Agfw received reply from PE for resource state change for ora.net1.network srv03 1

  • Urgent: patched crs bundle patch #3. CRS not starting up

    hi,
    i had patched the 7117233 patch bundle. everything went fine on my rac . the database version is 10.2.0.3 i updated the crs to 10.2.0.3. with the patch,
    i had issues with te post patch but it was some permission issue and it worked out.
    but now the last :
    custom/scripts/postrootpatch.sh -crshome /app/oracle/product/10.2.0/crs
    Checking to se if Oracle CRS stack is already up...
    Checking to se if Oracle CRS stack is already starting
    /etc/init.d/init.cssd: line 320: /tmp/oratz.30171 : No such file or directory
    /etc/init.d/init.cssd: line 320: /tmp/oratz.30193 : No such file or directory
    /etc/init.d/init.cssd: line 320: /tmp/oratz.30215 : No such file or directory
    /etc/init.d/init.cssd: line 320: /tmp/oratz.30240 : No such file or directory
    /etc/init.d/init.cssd: line 320: /tmp/oratz.30262 : No such file or directory
    Startup will be queued to init within 30 seconds
    it has been freezed as such for more than 10 minutes .. Has not start up!
    any sorta help will be most appreciated !
    regards
    Susan J

    user10758089 wrote:
    Hi Susan,
    2 possible solutions:
    1. as root user, "chmod -R 777 /tmp" and retry start CRSdon't set it to 777 as it would lead to a security nightmare.
    the problem is indeed a permission problem. the script tries to write to the file as user oracle and to remove it afterwards. the error message is from the removal.
    check, if oracle user can create files in /tmp and if not fix it by setting the proper permissions which are 1777 (drwxrwxrwt).
    regards,
    -ap

  • TS3474 My ipod will not start even after holding down the Sleep/Wake button and the Volume Down button for at least eight seconds.  What is my next step?

    I have a sixth generation ipod 16gb nano.  It will not turn on.  I have tried hoding down the power button with the down volumn button without results.
    Any other suggestions?

    i have the same problem...my ipod is 4th generation?uggghhhhh

  • Computers not starting up and shutting down as programmed

    I have a lab of 30 computers running 10.3.9 that are set to startup in the morning at 8am and shutdown at 5:30pm. They were working just fine for the last month, but now for the last week, they've stopped shutting down when scheduled. As far as I can see, there isn't anything running that would cause the shutdown to stop, unless there is a rogue process running, but even that wouldn't make sense as sometimes the lab isn't even used all day, and so it stays at the login screen all day. I've repaired permissions, but that didn't help. I thought about dumping the preferences, and thought I knew it once before, but am drawing a blank at the moment. Any suggestions?
    Ben

    I came across the same thing yesterday morning. I gave it try, and they shut down properly yesterday afternoon, but they didn't start back up this morning. I turned one on and checked the settings on it and they're correct (scheduled to start up every weekday morning at 8am). I know there has to be something I'm missing but just can't think of what that is.
    Ben

  • MacBook Pro does not start up. Shuts down after apple logo screen with a loading bar.

    I booted my macbook pro up today and i noticed it took a while to do so. i also noticed that the fans were going fast and the macbook was getting hot during the start up process. When i went to the screen i also noticed a loading bar which i havent noticed before when starting up the mac. Once the loading bar has reached the end, the macbook just turns off and the same thing happens over and over. I am on mountain lion

    I went to apple today and they told me that the hard drive has unmounted itself. They have my macbook right now and they said they're going to replace the wire that connects the hard drive to the logic board and see if this solves the issue. If it doesn't work they will replace the existing 250GB hard drive in my macbook with a 500GB as they no longer make that size hard drives (cost of £130 for the replacement). The problem is that i dont have my macbook backed up so will there be anyway to get data off the od hard drive?

  • OAS with CRS, Private Interconnect

    Hi,
    When using CRS with OAS only, do you still need a private interconnect like you would with RAC databases?
    Thanks.

    Thats what I thought but I was told that wasn't true. Did you setup your OAS/CRS combo with a private interconnect?
    Thanks!

  • INS-20802 Oracle Private Interconnect Configuration Assistant failed

    Thought I would post what information I've gathered, after facing this error during install of RAC Grid Infrastructure 11.2.0.1 on Red Hat Enterprise Linux Server release 5.5 64-bit, as Oracle Support is once again unable to help. Maybe this will save someone else some time and the aggravation of dealing with lousy Oracle Support.
    The error occurs after root.sh has successfully completed on all nodes. Oracle Net Configuration Assistant runs successfully, then Oracle Private Interconnect Configuration Assistant launches and subsequently fails with the following.
    [INS-20802] Oracle Private Interconnect Configuration Assistant failed.
    /u01/app/oraInventory/logs/installActions2010-12-13_01-26-10PM.log
    INFO: Starting 'Oracle Private Interconnect Configuration Assistant'
    INFO: Starting 'Oracle Private Interconnect Configuration Assistant'
    INFO: PRIF-26: Error in update the profiles in the cluster
    INFO:
    WARNING:
    INFO: Completed Plugin named: Oracle Private Interconnect Configuration Assistant
    INFO: Oracle Private Interconnect Configuration Assistant failed.
    INFO: Oracle Private Interconnect Configuration Assistant failed.
    I was able to find another error that coincides with the PRIF-26 error: "CRS-2324:Error(s) occurred while trying to push GPnP profile. State may be inconsistent."
    I was also able to duplicate the PRIF-26 error by trying to add a non-existent network interface via oifcfg:
    ./oifcfg setif -global jjj1/192.167.1.0:cluster_interconnect
    PRIF-26: Error in update the profiles in the cluster
    My best guess is the Oracle Private Interconnect Configuration Assistant makes a call to oifcfg. When oifcfg makes an update or adds a public/private interface, some XML files are also updated or maybe cross-referenced. These files are located here: <grid_home>/gpnp/<host>/profiles/peer
    Any updates/changes/addtions to the private or public interfaces include changes for the Grid Plug-n-Play component, which uses the XML files. If the interface name is not contained in the XML files, my best guess is that triggers the "CRS-2324:Error(s) occurred while trying to push GPnP profile. State may be inconsistent.
    I verified everything was configured correctly; the cluster verification utility reported everything was ok. I also ran the cluster verifcation utility against the GP-nP:
    ./cluvfy comp gpnp -verbose
    I also reviewed the public and private interfaces via oifcfg and they are correct:
    [oracle@ryoradb1 bin]$ ./oifcfg getif -global
    eth0 10.10.2.0 global public
    eth1 192.167.1.0 global cluster_interconnect
    [oracle@ryoradb1 bin]$ ./oifcfg iflist -p
    eth0 10.10.2.0 PRIVATE
    eth1 192.167.1.0 UNKNOWN
    My conclusion is the environment is configured correctly, in spite of the error generated by the Oracle Private Configuration Assistant.

    I understand that you have installed 11.2.0.1 not 11.2.0.2 because multicating must be enabled if you have installed 11.2.0.2 and you may face these sort of problems because cluster nodes would not be able to communicate with each other.
    Please check ocssd.log especially on the first node because this file will give more inforamtion as to why first node is not able to push GPnP file. As you have executed cluvfy to confirm the GPnP but to confirm whether GPnP profile is accurate to just narrow down the problem, I would suggest you to try to start cluster in exclusive mode so that you can sure that GPnP profile is good.
    Shutdown CRS on all nodes, if there is any hung processes then kill them before executing the fllowing command to start cluster in exclusive mode.
    $GRID_HOME/bin/crsctl start crs -excl
    If you are able to start cluster in exclusive then it is sure that GPnP is correct and then next step would be to verify the private network.
    See how you goes.
    FYI, I was recently analyzing the same sort of problems where cluster was not able to access GPnP profile and finally I found issues on my private network. Customer had enabled IGMP snooping, which was avoiding multicast communication over the private network but it was 11.2.0.2, which is not the case here.
    Harish Kumar
    http://www.oraxperts.com

  • CRS-0215: Could not start resource

    Hello
    This is my first time installing clusterware. However, i have not been too successful at it. This is my configuration:
    Operating System: RHES 5.3
    Oracle Database 11gR1
    OpenFiler used to configure shared disks
    After several attempts, i was able to run the root.sh script on both nodes.
    Output on first node:
    [root@vx0301 oracle]# /u01/app/oraInventory/orainstRoot.sh
    Changing permissions of /u01/app/oraInventory to 770.
    Changing groupname of /u01/app/oraInventory to oinstall.
    The execution of the script is complete
    [root@vx0301 oracle]# /u01/crs11g/root.sh
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    clscfg: EXISTING configuration version 4 detected.
    clscfg: version 4 is 11 Release 1.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: vx0301 vx0301-priv vx0301
    node 2: vx0302 vx0302-priv vx0302
    clscfg: Arguments check out successfully.
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Cluster Synchronization Services is active on these nodes.
    vx0301
    Cluster Synchronization Services is inactive on these nodes.
    vx0302
    Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
    Output on second node:
    [root@vx0301 oracle]# /u01/app/oraInventory/orainstRoot.sh
    Changing permissions of /u01/app/oraInventory to 770.
    Changing groupname of /u01/app/oraInventory to oinstall.
    The execution of the script is complete
    [root@vx0301 oracle]# /u01/crs11g/root.sh
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    clscfg: EXISTING configuration version 4 detected.
    clscfg: version 4 is 11 Release 1.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: vx0301 vx0301-priv vx0301
    node 2: vx0302 vx0302-priv vx0302
    clscfg: Arguments check out successfully.
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Cluster Synchronization Services is active on these nodes.
    vx0301
    vx0302
    I also got the errors:
    Starting GSD application resource on (2) nodes1:CRS-0215: Could not start resource 'ora.vx0301.gsd'
    Starting ONS application resource on (2) nodes1:CRS-0215: Could not start resource 'ora.vx0301.ons'
    There were no log files.
    When i clicked ok on the execute scripts page, i got the following errors in the configuration assistants page:
    Output generated from configuration assistant "Oracle Notification Server Configuration Assistant":
    Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Configuration assistant "Oracle Notification Server Configuration Assistant" failed
    The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------Output generated from configuration assistant "Oracle Notification Server Configuration Assistant":
    Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Configuration assistant "Oracle Notification Server Configuration Assistant" failed
    The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------Output generated from configuration assistant "Oracle Notification Server Configuration Assistant" (attempt 2):
    Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Configuration assistant "Oracle Notification Server Configuration Assistant" failed
    The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------
    Contents of the /u01/crs11g/cfgtoollogs/configToolFailedCommands script:
    [oracle@vx0301 ~]$ cat /u01/crs11g/cfgtoollogs/configToolFailedCommands
    # Copyright (c) 1999, 2007, Oracle. All rights reserved.
    /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    /u01/crs11g/bin/oifcfg setif -global eth0/172.30.4.0:public eth1/192.168.1.0:cluster_interconnect
    /u01/crs11g/bin/cluvfy stage -post crsinst -n vx0301,vx0302
    I tried running /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251 manually from the terminal:
    [root@vx0301 oracle]# /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
    The ONS configuration failed to create
    Output of the ons.log file:
    [root@vx0301 oracle]# cat /u01/crs11g/log/vx0301/racg/ons.log
    Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
    2010-01-25 13:59:15.786: [    RACG][3055679168] [10113][3055679168][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    2010-01-25 13:59:33.359: [    RACG][3055584960] [10229][3055584960][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    2010-01-25 14:01:00.319: [    RACG][3086849728] [10734][3086849728][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    2010-01-25 14:02:02.723: [    RACG][3086628544] [11105][3086628544][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
    [root@vx0301 oracle]#
    I am absoutely stumped as to what to do next. Any help is greatly appreciated.

    I tried running ruclufvy.sh stage -post crsinst -n vx0301,vx0302 -verbose. Here is the output:
    [oracle@vx0301 clusterware]$ ./runcluvfy.sh stage -post crsinst -n vx0301,vx0302 -verbose
    Performing post-checks for cluster services setup
    Checking node reachability...
    Check: Node reachability from node "vx0301"
    Destination Node Reachable?
    vx0301 yes
    vx0302 yes
    Result: Node reachability check passed from node "vx0301".
    Checking user equivalence...
    Check: User equivalence for user "oracle"
    Node Name Comment
    vx0302 passed
    vx0301 passed
    Result: User equivalence check passed for user "oracle".
    Checking Cluster manager integrity...
    Checking CSS daemon...
    Node Name Status
    vx0302 running
    vx0301 running
    Result: Daemon status check passed for "CSS daemon".
    Cluster manager integrity check passed.
    Checking cluster integrity...
    Node Name
    vx0301
    vx0302
    Cluster integrity check passed
    Checking OCR integrity...
    Checking the absence of a non-clustered configuration...
    All nodes free of non-clustered, local-only configurations.
    Uniqueness check for OCR device passed.
    Checking the version of OCR...
    OCR of correct Version "2" exists.
    Checking data integrity of OCR...
    ERROR:
    OCR integrity is invalid.
    OCR integrity check failed.
    Checking CRS integrity...
    Checking daemon liveness...
    Check: Liveness for "CRS daemon"
    Node Name Running
    vx0302 yes
    vx0301 yes
    Result: Liveness check passed for "CRS daemon".
    Checking daemon liveness...
    Check: Liveness for "CSS daemon"
    Node Name Running
    vx0302 yes
    vx0301 yes
    Result: Liveness check passed for "CSS daemon".
    Checking daemon liveness...
    Check: Liveness for "EVM daemon"
    Node Name Running
    vx0302 yes
    vx0301 yes
    Result: Liveness check passed for "EVM daemon".
    Liveness of all the daemons
    Node Name CRS daemon CSS daemon EVM daemon
    vx0302 yes yes yes
    vx0301 yes yes yes
    Checking CRS health...
    Check: Health of CRS
    Node Name CRS OK?
    vx0302 yes
    vx0301 unknown
    Result: CRS health check failed.
    CRS integrity check failed.
    Checking node application existence...
    Checking existence of VIP node application
    Node Name Required Status Comment
    vx0302 yes exists passed
    vx0301 yes exists passed
    Result: Check passed.
    Checking existence of ONS node application
    Node Name Required Status Comment
    vx0302 no exists passed
    vx0301 no exists passed
    Result: Check passed.
    Checking existence of GSD node application
    Node Name Required Status Comment
    vx0302 no exists passed
    vx0301 no exists passed
    Result: Check passed.
    Post-check for cluster services setup was unsuccessful on all the nodes.
    [oracle@vx0301 clusterware]$
    I also tried running ./ocrchek on both nodes. Here is the output:
    [oracle@vx0301 bin]$ ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
    Version : 2
    Total space (kbytes) : 3306636
    Used space (kbytes) : 2056
    Available space (kbytes) : 3304580
    ID : 1425438992
    Device/File Name : /dev/sdb1
    Device/File integrity check failed
    Device/File not configured
    Cluster registry integrity check failed
    [oracle@vx0301 bin]$ ssh vx0302
    Last login: Mon Jan 25 13:52:03 2010 from vx0301
    [oracle@vx0302 ~]$ cd /u01/crs11g/bin/
    [oracle@vx0302 bin]$ ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
    Version : 2
    Total space (kbytes) : 3306636
    Used space (kbytes) : 2056
    Available space (kbytes) : 3304580
    ID : 1425438992
    Device/File Name : /dev/sdb1
    Device/File integrity check failed
    Device/File not configured
    Cluster registry integrity check failed
    [oracle@vx0302 bin]$
    From the results, the ids are the same, and so is the OCR disk. But the integrity check has failed. How do i interpret this output?

  • RAC CRS not getting started

    Hi,
    We had installed RAC DB namely ORCL on a two-node cluster abcoracledb01 and abcoracledb02 with the OUI launched from db01 node. After installation, the NETCA is able to identify the RAC DB ORCL on both the nodes as well as the ORCL1 instance on db01 node and ORCL2 instance on db02 node. Now when I try to launch the DBCA from db01 node, it prompts me with the two normal options Real Applications Cluster database and normal database on its welcome screen as it detects the CRS running on this node. The problem is coming when Im trying to launch DBCA from the second node as it does not extends the RAC option in its welcome screen. I can understand that the CRS is not running on this db02 instance and and i verified this by running the OLSNODES -v command that gives me an error stating that the cluster interface is not getting started.
    Will highly appreciate if anyone can help me to get some referenctial documentation or diagnostic information that may help me in troubsleshooting the exact reason behind the CRS not getting started.
    Thanks,
    Manoj ([email protected])

    Ummm... Why are you trying to run the DBCA twice? The DBCA is really only there to give you a starter database configured for either OLTP, OLAP, or multi-purpose, but since both nodes will be attached to the same database, you shouldn't run the DBCA more than once. Once you have the database created with DBCA, attach the second node to it and go. Your problem is not with DBCA, it's in the cluster interconnect. Once that starts up, the second instance will attach to the database and go.

  • Could Not start CRS or HAS in RAC 11.2.0.1

    Dear Legends,
    We have the following environment in SOLARIS 2 Node Database. Not sure why the 2nd node went down today morning but 1st Node is still Up and Running.
    My Tries so for
    1. Changed the following line as per Doc 1368382.1 from
       h1:3:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    To
       h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2. Provided a Restart of Entire Linux Box
    3. Tried issuing the following command ./crsctl start crs but Error out as follows
    CRS-4124: Oracle High Availability Services startup failed.
    CRS-4000: Command Start failed, or completed with errors.
    4. Tried checking in Database, ASM alert logs nothing related with OHASD.
    5. Also checked in ohasd.log and ohasdOUT.log not able to find any thing.
    Please help me. This is our Production Environment.
    Thanks,
    Karthik

    Karthik,
    I encountered the same error few days back and below is what I did to resolve the issue as mentioned in metalink 1368382.1. Did you see anything in the crs logs?
    did you follow all the steps in the solution section below like killiing remaining rc3 scripts?
    checks :
    1. Command '$GRID_HOME/bin/crsctl check crs' returns error:
         CRS-4639: Could not contact Oracle High Availability Services
    2. Command 'ps -ef | grep init' does not show a line similar to:
         root 4878 1 0 Sep12 ? 00:00:02 /bin/sh /etc/init.d/init.ohasd run
    3. Command 'ps -ef | grep d.bin' does not show a line similar to:
         root 21350 1 6 22:24 ? 00:00:01 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
        Or it may only show "ohasd.bin reboot" process without any other processes
    4. ohasd.log report:
           2013-11-04 09:09:15.541: [ default][2609911536] Created alert : (:OHAS00117:) :  TIMED OUT WAITING FOR OHASD MONITOR
    5. ohasOUT.log report:
           2013-11-04 08:59:14
           Changing directory to /u01/app/11.2.0/grid/log/lc1n1/ohasd
           OHASD starting
           Timed out waiting for init.ohasd script to start; posting an alert
    Solutions:
    1. Add the following line to /etc/inittab
        h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
       and then run "init q" as the root user.
    2. Run command 'ps -ef | grep rc' and kill any remaining rc3 scripts that appear to be stuck.
    3. Remove the bad entry before init.ohasd. Consult with OS vendor if "init q" does not spawn "init.ohasd run" process. As a workaround,
       start the init.ohasd manually, eg: as root user, run "/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null &"
    4. Enable CRS autostart:
       # crsctl enable crs
       # crsctl start crs
    5. Restore OLR from backup, as root user:
       # touch $GRID_HOME/cdata/<node>.olr
      # chown root:oinstall $GRID_HOME/cdata/<node>.olr
      # ocrconfig -local -restore$GRID_HOME/cdata/<node>/backup_<date>_<num>.olr
      # crsctl start crs
    If OLR backup does not exist for any reason, perform deconfig and rerun root.sh is required to recreate OLR, as root user:
       # $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
       # $GRID_HOME/root.sh
    6. If above does not help, check OS messages for ohasd.bin logger message and manually execute crswrapexece.pl command mentioned in the OS message with LD_LIBRARY_PATH set to <GRID_HOME/lib to continue debug.

Maybe you are looking for

  • Third Party Plug-in for Multiple Swatch Palettes in InDesign

    Hello, Does anyone know if there is a third-party plug in to have multiple swatch palettes in InDesign? I would like to create different palettes for different sections of the long document --- scrolling through all of the swatches each time is cumbe

  • Why is the word and holiday Christmas no longer in the Firefox dictionary, and why did I have to add it myself?

    When I typed in Christmas, it was underlined in red. Which means that it wasn't in the dictionary. When I right-clicked on it, there were no spelling suggestions and I had to add it to the dictionary myself. Christmas is a word and as far as I know,

  • Battery info very inaccurate...

    I know apple said the battery display was approximate but it will be 100% straight off the charger and I will play ~30 secs. of video and stop. the battery bar is down to like 80% and then after about 2 mins. it goes back up to 100%. It really needs

  • Possible to have link object on row level?

    Hi, it seems impossible but, would like to know if there is a workaround to link the column's cell on the basis of its content. for example, one column called base document will have the docentry / docnum of the document from which the detail row is

  • How to loop sql query

    hi , i've already define date value as below , DEFINE START_TIME = 2011-08-12 09:00:00 DEFINE END_TIME = 2011-08-12 10:00:00And Quering the table depending on the Defined date as below , select * from My_Table where ( to_date(Accounting_Start_Time,'y