Solaris 10 CRS-4535 CRS-4000 ASM

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
crsd.log的日志如下：
2013-03-19 17:07:31.185: [ CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2013-03-19 17:07:32.187: [ CSSCLNT][1]clssscConnect: gipc request failed with 29 (16)
2013-03-19 17:07:32.187: [ CSSCLNT][1]clsssInitNative: connect failed, rc 29
2013-03-19 17:07:32.188: [ CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
是不是没有连接上存储，因为之前安装集群的时候就只是改变了存储的磁盘权限和使用zpool创建磁盘，没有做过其他操作了
裸设备是不是需要做映射？
DB：11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
帖子经 994726编辑过
帖子经 994726编辑过

Can you run cluvfy
cluvfy stage -post hwos stage
http://docs.oracle.com/cd/E14072_01/rac.112/e10717/cvu.htm#CJGJHIEG

Similar Messages

CRS-4000 error very very urgent

Hi DBA's
Database : 11.2.0.1
OS :- Solaris 10
QFS File system
I am unable to start the listener and vip ip is showing fail over.
when i tried to start crs it has thronging some error like bellow.
CRS-4000 Command start Failover, complete with error
oracle@m5k2a $ crs_stat -t
Name Type Target State Host
ora....ER.lsnr ora....er.type ONLINE ONLINE m5k2b
ora....N1.lsnr ora....er.type ONLINE ONLINE m5k2b
ora.asm ora.asm.type OFFLINE OFFLINE
ora.cdrrac1.db ora....se.type ONLINE ONLINE m5k2a
ora.eons ora.eons.type ONLINE ONLINE m5k2a
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....SM1.asm application OFFLINE OFFLINE
ora....2A.lsnr application ONLINE OFFLINE
ora.m5k2a.gsd application OFFLINE OFFLINE
ora.m5k2a.ons application ONLINE OFFLINE
ora.m5k2a.vip ora....t1.type ONLINE ONLINE m5k2b
ora....SM2.asm application OFFLINE OFFLINE
ora....2B.lsnr application ONLINE ONLINE m5k2b
ora.m5k2b.gsd application OFFLINE OFFLINE
ora.m5k2b.ons application ONLINE ONLINE m5k2b
ora.m5k2b.vip ora....t1.type ONLINE ONLINE m5k2b
ora....network ora....rk.type ONLINE ONLINE m5k2a
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE m5k2b
ora.scan1.vip ora....ip.type ONLINE ONLINE m5k2b
oracle@m5k2a $ ./crsctl status resource -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.LISTENER.lsnr
ONLINE OFFLINE m5k2a
ONLINE ONLINE m5k2b
ora.asm
OFFLINE OFFLINE m5k2a
OFFLINE OFFLINE m5k2b
ora.eons
ONLINE ONLINE m5k2a
ONLINE ONLINE m5k2b
ora.gsd
OFFLINE OFFLINE m5k2a
OFFLINE OFFLINE m5k2b
ora.net1.network
ONLINE ONLINE m5k2a
ONLINE ONLINE m5k2b
ora.ons
ONLINE OFFLINE m5k2a
ONLINE INTERMEDIATE m5k2b CHECK TIMED OUT
Cluster Resources
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE m5k2b
ora.cdrrac1.db
1 ONLINE ONLINE m5k2a Open
2 ONLINE ONLINE m5k2b Open
ora.m5k2a.vip
1 ONLINE INTERMEDIATE m5k2b FAILED OVER
ora.m5k2b.vip
1 ONLINE ONLINE m5k2b
ora.oc4j
1 OFFLINE OFFLINE
ora.scan1.vip
1 ONLINE ONLINE m5k2b
oracle@m5k2a $

Hi,
Can you post the output from
$ > srvctl stop listener -l LISTENER -n m5k2a
$ > srvctl start listener -l LISTENER -n m5k2a
also post the log from cssd.log and crsd.log
Cheers

11g CRS and ASM, 10.2.0.4 database, how to set up RMAN

Hi all, I am fairly green when it comes to RAC so please bear with me. I have a Linux 2 node RAC environment. The servers are lux148 and lux149. The 2 node cluster is running 11g CRS and ASM. The database is 10.2.0.4. It had to be set up this way in order for IBM DataStage to work. The database name is fictrp0. The instances are fictrp01 (lux148) and fictrp02 (lux149). I am trying to register the database with an RMAN catalog. I am under the impression that I need to register the database (fictrp0) not the instances (fictrp01 and fictrp02) with the RMAN catalog. Is this correct? So I log into lux148 and set my environment so the ORACLE_SID=FICTRP0. From the command line I issue the following:
fictrp0:/u01/app/oracle> rman target / catalog rman102/[email protected]
The command returns the following:
Recovery Manager: Release 10.2.0.4.0 - Production on Wed Oct 29 14:05:50 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database (not started)
connected to recovery catalog database
Is this normal "connected to target database (not started)"? I was expecting to see a DBID=FICTRP0 for the target. If this is typical, how will RMAN know the ID of the database? Should I be trying to register an instance perhaps such as FICTRP01? If so do I need to register both instances (FICTRP01 and FICTRP02)?
Bottom line I am very confused on how RAC and RMAN work together. Any help would be greatly appreciated.

You no need to register the instance,You need to register only the database and database only will have DBID

Patching Strategy for CRS and ASM homes

I'm fairly new to RAC/ASM and haven't performed any patch set upgrades yet. Back in the simple days when I wanted to apply a patch set to a database, say from 10.2.0.4 to 10.2.0.5, I would create a brand new Oracle home ahead of time and apply the patch set to it. I'd name my homes like this:
/opt/oracle/product/10.2.0.4/db1
/opt/oracle/product/10.2.0.5/db1
During the maintenance window I would change /etc/oratab to point the database to the new 10.2.0.5 and complete the database upgrade scripts. The advantages of this strategy:
1 - Less risk installing software as nothing uses the new home yet. If something goes wrong in the install, no big deal. Research the problem and try again without being under the stress of a defined maintenance window.
2 - No need to backup old home for back-out purposes.
3 - Less time required for database to be down during actual patch window since Oracle Installer does not need to run.
Now with CRS and ASM, is there a way to pre-stage a new home for those, but not have them "active" to the node until later during the maintenance window?
For ASM, it seems like it would be possible to treat the same way as database and simply update ASM SID in /etc/oratab
+ASM1:/opt/oracle/product/10.2.0.5/asm1
but I'm not totally confident in that as I'm afraid the CRS home may already have references to the ASM home in the cluster registry.
For CRS, it seems like the home is pretty well hard-wired into the node startup scripts and installing a brand new CRS home will probably disrupt the running CRS home.
Any thoughts about this?

Hi,
user5448593 wrote:
I'm fairly new to RAC/ASM and haven't performed any patch set upgrades yet. Back in the simple days when I wanted to apply a patch set to a database, say from 10.2.0.4 to 10.2.0.5, I would create a brand new Oracle home ahead of time and apply the patch set to it.
Now with CRS and ASM, is there a way to pre-stage a new home for those, but not have them "active" to the node until later during the maintenance window?Although you have not mentioned the version you are actually on, it is a quite up-to-date question and dilemma.
Starting with 11.2 for Grid Infrastructure only "out-of-place" patchset upgrades are supported.
>
For ASM, it seems like it would be possible to treat the same way as database and simply update ASM SID in /etc/oratab
+ASM1:/opt/oracle/product/10.2.0.5/asm1
but I'm not totally confident in that as I'm afraid the CRS home may already have references to the ASM home in the cluster registry.
For CRS, it seems like the home is pretty well hard-wired into the node startup scripts and installing a brand new CRS home will probably disrupt the running CRS home.
Any thoughts about this?As of 11gR2 the ASM is part of the Grid Infrastructure, therefore it is running from the same home and not recommended to separate them. (although you can do that)
By the way, what is your upgrade path? It could be easier to answer your questions if we knew that as there has been a quite a few enhancements and changes in the upgrade/patching process from 10g to 11g. (even between 11gR1 and 11gR2)
Regards,
Jozsef

Ohasd failed to start: Inappropriate ioctl for device, CRS-4124:, CRS-4000:

New install on amazon, 11.2.0-64 image. Did NOT install canned 11.2.0 on the host, just downloaded the grid component and running the installer to configure Grid Infrastructure for a Stand-Alone Server.
Fails with the following error where I do Stand Alone Server or Grid Software only;
$ /u01/app/grid/1120/grid/perl/bin/perl -I/u01/app/grid/1120/grid/perl/lib -I/u01/app/grid/1120/grid/crs/install /u01/app/grid/1120/grid/crs/install/roothas.pl -verbose
2010-04-25 16:46:00: Checking for super user privileges
2010-04-25 16:46:00: User has super user privileges
2010-04-25 16:46:00: Parsing the host name
Using configuration parameter file: /u01/app/grid/1120/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
CRS-4664: Node domu-12-31-36-00-35-b2 successfully pinned.
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start: Inappropriate ioctl for device at /u01/app/grid/1120/grid/crs/install/roothas.pl line 296.
root@domU-12-31-36-00-35-B2:[u01/app/grid/1120/grid/install/utl]
Been through the related noted to no avail;
Doc ID 1063552.1 ----> rm -rf /tmp/.oracle /usr/tmp/.oracle /var/tmp/.oracle
Doc ID 969878.1 ----> service syslog restart
And ipV6 is NOT in configured in the hosts file.
Any ideas / solutions?
TIA,
Neil
and restarted

Hi,
Ran into this myself and then found your note. After much investigating it turns out that the Linux run level on my AMI's is set to 4 (even though the inittab has it set to 3) and the CRS init script init.ohasd expects 3 and 5. Because of that problem, the "init" process was not able to launch the init.ohasd script.
For now the workaound for this problem is to run the system at run levels 3 or 5 before executing the root script. However, I am still trying to figure out why my AMI's are running at 4.
I am curious, if you do read this, did you get this error during the install and ignore it?
Run Level
This is a prerequisite condition to test whether the system is running with proper run level. (more details)
Expected Value
: 3,5
Actual Value
: 4
I did and of course it didn't work. Also, what Amazon AMI did you use?
Thanks,
Larry

CRS on ASM

Quick question:
Can I install CRS on ASM? anoher word, can I create a CRSFILE on ASM by specifing the raw device path (ie ORCL:VOL1) using ASM?
Thx much,

Without the +ASM instance "/dev/oracleasmdisks/VOL1" still looks like a regular (block) raw device.
Just modify the /etc/sysconfig/rawdevices and you should be able access it as regular raw device and install CRS, looks like this should be able to install CRS on it.
Please advise...Thanks!

Install CRS so ASM can share diskgroups accross instances

I saw a note in a 10.1 document that mentions CRS can be used to allow ASM instances to share disk groups and that this configuration does not require a RAC license if single instance databases are being used.
We have a scenario in our development and test environments where clustered databases are not required but I would like to set up ASM similiar to production for consistency purposes.
Is anyone familiar with this? Is there any difference between this type of CRS install and one for RAC databases?

OK so you start off pretend that you are going to install RAC.
You install the clusterware onto your nodes
Then you install ASM onto your nodes
and create network listeners and ASM instances / diskgroup(s)
Then - you do not install the RAC database
instead you may wish to install a new database home that has RAC linked off and then use dbca to create single instance databases - you put the datafiles on asm by specifying something like +DATA (for the DATA) diskgroup

How to cleanup / remove RAC installation (CRS,RDBMS,ASM)

Hi, all
How to remove or cleanup RAC installation after ran OUI deinstall it couldn't finish and get a error... because 1 node of 3 was off (the box).
ACS

Don't forget to mention the Oracle version you're working with.

How to restore ASM based OCR after complete loss of the CRS diskgroup (Doc

Hello, I'm testing Metalink doc id 1062983.1.
I have rac 11.2.0 with asm on Hp-Ux. I have an CRS diskgroup with external redundancy. The Unix System administrator dropped crs disk:
GRID user:           oradb
GRID home:           /oracle/product/11.2.0/grid ($CRS_HOME)
ASM disk group name for OCR:      CRS
ASM/ASMLIB disk name:      CRS_0000
Linux device name for ASM disk:      /dev/rdisk/disk27
Cluster name:           crs_desa
Nodes:                rx2
I've checked backups
rx2:/oracle/product/11.2.0/grid/cdata/crs_desa $ ls -ltr
total 89384
-rw------- 1 root sys 6537216 Dec 7 20:34 week.ocr
-rw------- 1 root sys 6537216 Dec 8 00:34 day.ocr
-rw------- 1 root sys 6537216 Dec 9 00:35 backup02.ocr
-rw------- 1 root sys 6537216 Dec 9 00:35 day_.ocr
-rw------- 1 root sys 6537216 Dec 9 04:35 backup01.ocr
-rw------- 1 root sys 6537216 Dec 9 08:35 backup00.ocr
But I can't start CRS stack in exclusive mode. I've tried crsctl start crs -excl as root but I've got
rx2:/oracle/product/11.2.0/grid/bin # crsctl start crs -excl
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
can anyone help me please?
Thanks in advance!
Edited by: user13398689 on 09-dic-2010 10:02

You need to first stop the crs stack on all the nodes and then start the CRS stack on one node in exclusive mode.

Pros and Cons of installing CRS, ASM and DB in separate homes

Planning to install 11gR1 RAC a 2 nodes on HPUX IA64 and there are several architecture options:
Option 1: same owner and one single home (home1 = crs asm db)
pros - easier to patch on single home
cons - patch level may be required at crs level but not allowed at db level because of E-Business Suite certification constraints
Option 2: same owner and 2 homes (home1 = crs / home2 = asm db)
pros - one less home to upgrade
cons - ?
Option 3: same owner and 2 homes (home 1 = crs asm / home2 = db)
Is there any reason why Option 3 would be preferable or worse than Option 2?
Option 4: same owner and 3 homes (crs home / asm home / db home)
pros - each home can be on different patch levels
cons - more storage, more maintenance when patching
Any comments?

I've made my decision to use Option 2 and here's why...
Excerpt from Known issues documented in
810663.1 11.1.0.X CRS Bundle Patch Information
CRS Bundle Patch has been renamed as CRS PSU. CRS PSU and Database PSU are two separate patches, i.e. Database PSU does NOT include the CRS PSU.
There should be no conflict or overlap between a CRS PSU and an RDBMS PSU -- both should be applied to the ASM and DATABASE Homes.
Also note that CRS PSU's can be applied to all homes (CRS, ASM and RDBMS). The general recommendation is to apply the bundle patch to all homes unless the homes are on a different patch level. This is because there are clusterware binaries in the database home (e.g.: srvctl).
PSUs for the RDBMS should be applied to the ASM and RDBMS homes.

Could Not start CRS or HAS in RAC 11.2.0.1

Dear Legends,
We have the following environment in SOLARIS 2 Node Database. Not sure why the 2nd node went down today morning but 1st Node is still Up and Running.
My Tries so for
1. Changed the following line as per Doc 1368382.1 from
   h1:3:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
To
   h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
2. Provided a Restart of Entire Linux Box
3. Tried issuing the following command ./crsctl start crs but Error out as follows
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
4. Tried checking in Database, ASM alert logs nothing related with OHASD.
5. Also checked in ohasd.log and ohasdOUT.log not able to find any thing.
Please help me. This is our Production Environment.
Thanks,
Karthik

Karthik,
I encountered the same error few days back and below is what I did to resolve the issue as mentioned in metalink 1368382.1. Did you see anything in the crs logs?
did you follow all the steps in the solution section below like killiing remaining rc3 scripts?
checks :
1. Command '$GRID_HOME/bin/crsctl check crs' returns error:
     CRS-4639: Could not contact Oracle High Availability Services
2. Command 'ps -ef | grep init' does not show a line similar to:
     root 4878 1 0 Sep12 ? 00:00:02 /bin/sh /etc/init.d/init.ohasd run
3. Command 'ps -ef | grep d.bin' does not show a line similar to:
     root 21350 1 6 22:24 ? 00:00:01 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
    Or it may only show "ohasd.bin reboot" process without any other processes
4. ohasd.log report:
       2013-11-04 09:09:15.541: [ default][2609911536] Created alert : (:OHAS00117:) : TIMED OUT WAITING FOR OHASD MONITOR
5. ohasOUT.log report:
       2013-11-04 08:59:14
       Changing directory to /u01/app/11.2.0/grid/log/lc1n1/ohasd
       OHASD starting
       Timed out waiting for init.ohasd script to start; posting an alert
Solutions:
1. Add the following line to /etc/inittab
    h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
   and then run "init q" as the root user.
2. Run command 'ps -ef | grep rc' and kill any remaining rc3 scripts that appear to be stuck.
3. Remove the bad entry before init.ohasd. Consult with OS vendor if "init q" does not spawn "init.ohasd run" process. As a workaround,
   start the init.ohasd manually, eg: as root user, run "/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null &"
4. Enable CRS autostart:
   # crsctl enable crs
   # crsctl start crs
5. Restore OLR from backup, as root user:
   # touch $GRID_HOME/cdata/<node>.olr
# chown root:oinstall $GRID_HOME/cdata/<node>.olr
# ocrconfig -local -restore$GRID_HOME/cdata/<node>/backup_<date>_<num>.olr
# crsctl start crs
If OLR backup does not exist for any reason, perform deconfig and rerun root.sh is required to recreate OLR, as root user:
   # $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
   # $GRID_HOME/root.sh
6. If above does not help, check OS messages for ohasd.bin logger message and manually execute crswrapexece.pl command mentioned in the OS message with LD_LIBRARY_PATH set to <GRID_HOME/lib to continue debug.

CRS errors. Unable to bring up node 2

Hi, we have a 2 node rac Oracle on Linux 6 installed on VM. After the OS reboot node 1 will not start. Please advice as this is stopping my testing. thanks,
RACTEST1
[root@ractest1 bin]# crsctl check crs                                           CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@ractest1 bin]# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@ractest1 bin]# crsctl stat res -t -init
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE OFFLINE                               Instance Shutdown
ora.cluster_interconnect.haip
      1        ONLINE OFFLINE
ora.crf
      1        ONLINE ONLINE       ractest1
ora.crsd
      1        ONLINE OFFLINE
ora.cssd
      1        ONLINE OFFLINE
ora.cssdmonitor
      1        ONLINE ONLINE       ractest1
ora.ctssd
      1        ONLINE OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE OFFLINE
ora.gipcd
      1        ONLINE ONLINE       ractest1
ora.gpnpd
      1        ONLINE ONLINE       ractest1
ora.mdnsd
      1        ONLINE ONLINE       ractest1
RACTEST2
[root@ractest2 bin]# crsctl stat res -t -init
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE ONLINE       ractest2                 Started
ora.cluster_interconnect.haip
      1        ONLINE ONLINE       ractest2
ora.crf
      1        ONLINE ONLINE       ractest2
ora.crsd
      1        ONLINE ONLINE       ractest2
ora.cssd
      1        ONLINE ONLINE       ractest2
ora.cssdmonitor
      1        ONLINE ONLINE       ractest2
ora.ctssd
      1        ONLINE ONLINE       ractest2                 ACTIVE:0
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE ONLINE       ractest2
ora.gipcd
      1        ONLINE ONLINE       ractest2
ora.gpnpd
      1        ONLINE ONLINE       ractest2
ora.mdnsd
      1        ONLINE ONLINE       ractest2
alertlogRACTEST1.log
2014-11-27 14:24:20.517:
[/u01/app/11.2.0/grid/bin/cssdagent(22078)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:95:24} in /u01/app/11.2.0/grid/log/ractest1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2014-11-27 14:24:25.649:
[ohasd(1812)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ractest1'.
2014-11-27 14:24:25.926:
[ohasd(1812)]CRS-2878:Failed to restart resource 'ora.cssd'
2014-11-27 14:24:26.949:
[cssd(22951)]CRS-1713:CSSD daemon is started in clustered mode
2014-11-27 14:24:32.587:
[cssd(22951)]CRS-1707:Lease acquisition for node ractest1 number 1 completed
2014-11-27 14:24:33.847:
[cssd(22951)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/REDO; details in /u01/app/11.2.0/grid/log/ractest1/cssd/ocssd.log.
2014-11-27 14:34:25.937:
[cssd(22951)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/ractest1/cssd/ocssd.log
2014-11-27 14:34:25.937:
[cssd(22951)]CRS-1603:CSSD on node ractest1 shutdown by user.
2014-11-27 14:34:25.936:
[/u01/app/11.2.0/grid/bin/cssdagent(22923)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:95:24} in /u01/app/11.2.0/grid/log/ractest1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2014-11-27 14:34:31.079:
[ohasd(1812)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ractest1'.
2014-11-27 14:34:31.405:
[ohasd(1812)]CRS-2878:Failed to restart resource 'ora.cssd'

Are there any additional details in the following log files?
/u01/app/11.2.0/grid/log/ractest1/cssd/ocssd.log
/u01/app/11.2.0/grid/log/ractest1/agent/ohasd/oracssdagent_root/oracssdagent_root.log

CRS not starting

OS: OEL 5 U4 x86_64
DB: Oracle 11.2.0.1 EE
Grid Infrastructure: Oracle 11.2.0.1
CRS and Voting disk Storage: ASM
Datafile and FRA storage: ASM
I'm not sure exactly what caused this, but anyways, I changed MTU from 1500 to 900 online. After some time, 3 out of 4 nodes in the cluster went down and CRS refuses to start on these nodes after trying the switch back from MTU 9000 to 1500, reboots, and making sure disk permissions and ownership are correct. The logs are not too helpful (and cryptic) so I'm at a loss and appreciate any ideas or help.
The installation was successful, the RAC was up for a few days while running some tests (including restart of a node). Currently only a single node has everything up and functional, the others are not working. Below are some output that might help:
[root@ucstst11 bin]# ./crsctl check cluster -n ucstst11
ucstst11:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
[root@ucstst11 bin]# ./crsctl start cluster -n ucstst11
CRS-2672: Attempting to start 'ora.cssd' on 'ucstst11'
CRS-2672: Attempting to start 'ora.diskmon' on 'ucstst11'
CRS-2676: Start of 'ora.diskmon' on 'ucstst11' succeeded
CRS-4404: The following nodes did not reply within the allotted time:
ucstst11
[root@ucstst11 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
[root@ucstst11 bin]# ./crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@ucstst11 bin]# oracleasm querydisk -p CRSVOL01
Disk "CRSVOL01" is a valid ASM disk
/dev/sdz1: LABEL="CRSVOL01" TYPE="oracleasm"
/dev/sdcj1: LABEL="CRSVOL01" TYPE="oracleasm"
[root@ucstst11 bin]# ll /dev/sdz1 /dev/sdcj1
brw-rw---- 1 oracle dba 69, 113 Mar 27 19:00 /dev/sdcj1
brw-rw---- 1 oracle dba 65, 145 Mar 27 19:00 /dev/sdz1
[root@ucstst11 bin]# oracleasm querydisk -d CRSVOL01
Disk "CRSVOL01" is a valid ASM disk on device [65, 145]
From the functional node:
[root@ucstst12 bin]# ./crsctl check cluster -all
ucstst12:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
Cluster verification now hangs when it tries to contact the other nodes.
Please help!

For the most part this issue has been resolved. The SA partially changed to jumbo frames (OS, but not the switch), we reverted all the jumbo frame changes and the system is back online, except for one node (the one which was working ironically) not being reported via "crsctl check cluster -all", and one instance not starting due to it not seeing an interconnect (weird).
We did attempt to fully implement jumbo frames but that did not work hence the reversion.

Not able to stop crs

Dear Legends,
As I was trying to decommision a database as I did
- Shutdown 2 DB's
- Shutdown 2 Listeners
- Shutdown agents
- Shutdown ASM Instance
But I forget to stop the crs when I checked with
ps -ef|grep has
grid      2280 2104 0 07:18 pts/3    00:00:00 grep has
grid      3030     1 0 Apr26 ?        00:06:27 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
root      3338     1 0 Apr26 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
Error or Email Alert receiving due to this as follows
Corrective action=USER_DEFINED_ASM_USAGE_ERRORS_CORRECTIVE_ACTION
Corrective action owner=SYSMAN
Corrective action status=Succeeded
Corrective action output=
Command:Output Log
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 4 00:07:15 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
SQL> SQL> SQL> SQL> Connected.
SQL> SQL> SQL> SQL> SQL> SQL> 2 3
AUE_HOST AUE_DATAB
AUE_TIMES
AUE_ERROR_TEXT
Beldon.host.com RMCLDDMO
04-JUN-14
ORA-12541: TNS:no listener
SQL> SQL> 2 3
1 row updated.
SQL> SQL>
Commit complete.
SQL> SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
~~End Step Output/Error Log~~
Target Name=GOCPRD1.host.com
Target type=
Host=
Occurred At=Jun 4, 2014 12:07:16 AM EDT
Message=
Metric=User-Defined Numeric Metric
Metric value=1
Script=User_Defined_ASM_USAGE_ERRORS
Severity=Warning
Acknowledged=
Notification Rule Name=ASM Usage Error Notification rule (User Defined)
Notification Rule Owner=SYSMAN
When I tried to stop using below command and it errored out
./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'beldon'
CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'beldon'
CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'beldon' failed
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'beldon' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
Let me know what steps should I follow as I am new to the Environment need to stick and learn more. But I agree I didn't follow a standard procedure to decommission.
Thanks,
Karthik

Thanks Freddie,
I tried and the output as follows
[grid@host bin]$ ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'host'
CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'host'
CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'host' failed
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'host' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
[grid@host bin]$ ./crsctl check has
CRS-4638: Oracle High Availability Services is online
Also tried
[grid@host ~]$ crsctl stop res ora.LISTENER_RMCLDDG1.lsnr -f
CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'host'
CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'host' failed
CRS-5802: Unable to start the agent process
CRS-4000: Command Stop failed, or completed with errors.
/etc/init.d/ohasd stop
Stopping Oracle Clusterware stackCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'host'
CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'host'
CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'host' failed
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'host' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
Thanks,
Karthik

Metalink SR category for problem with CRS install

HI
Can You please tell me , where I find in SR category to create request on metaling about problem with install/start crs :
I wont to create RAC env where host is ORACLE VM and clusterware is OCFS2 and wit Oracle Enterprise linux 5 , but after istal ai have problem with start crs (root.sh) and I don't know to find relevant category
[root@lin1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.gipcd' on 'lin1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'lin1'
CRS-2676: Start of 'ora.mdnsd' on 'lin1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lin1'
CRS-2676: Start of 'ora.gpnpd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lin1'
CRS-2676: Start of 'ora.cssdmonitor' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lin1'
CRS-2672: Attempting to start 'ora.diskmon' on 'lin1'
CRS-2676: Start of 'ora.diskmon' on 'lin1' succeeded
CRS-2674: Start of 'ora.cssd' on 'lin1' failed
CRS-679: Attempting to clean 'ora.cssd' on 'lin1'
CRS-2681: Clean of 'ora.cssd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lin1'
CRS-2676: Start of 'ora.cssdmonitor' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lin1'
CRS-2674: Start of 'ora.cssd' on 'lin1' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'lin1'
CRS-2681: Clean of 'ora.cssd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'lin1'
CRS-2676: Start of 'ora.diskmon' on 'lin1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'lin1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'lin1' succeeded
CRS-4000: Command Start failed, or completed with errors.
[root@lin1 bin]# ./crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@lin1 bin]# ./crsctl check cluster -all
lin1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
Thank You Brano

HI
Can You please tell me , where I find in SR category to create request on metaling about problem with install/start crs :
I wont to create RAC env where host is ORACLE VM and clusterware is OCFS2 and wit Oracle Enterprise linux 5 , but after istal ai have problem with start crs (root.sh) and I don't know to find relevant category
[root@lin1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.gipcd' on 'lin1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'lin1'
CRS-2676: Start of 'ora.mdnsd' on 'lin1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lin1'
CRS-2676: Start of 'ora.gpnpd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lin1'
CRS-2676: Start of 'ora.cssdmonitor' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lin1'
CRS-2672: Attempting to start 'ora.diskmon' on 'lin1'
CRS-2676: Start of 'ora.diskmon' on 'lin1' succeeded
CRS-2674: Start of 'ora.cssd' on 'lin1' failed
CRS-679: Attempting to clean 'ora.cssd' on 'lin1'
CRS-2681: Clean of 'ora.cssd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lin1'
CRS-2676: Start of 'ora.cssdmonitor' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lin1'
CRS-2674: Start of 'ora.cssd' on 'lin1' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'lin1'
CRS-2681: Clean of 'ora.cssd' on 'lin1' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'lin1'
CRS-2676: Start of 'ora.diskmon' on 'lin1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'lin1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'lin1' succeeded
CRS-4000: Command Start failed, or completed with errors.
[root@lin1 bin]# ./crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@lin1 bin]# ./crsctl check cluster -all
lin1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
Thank You Brano

Solaris 10 CRS-4535 CRS-4000 ASM

Similar Messages

Maybe you are looking for