Crs not start
Hi
oracle cluster version 11.2
o/s windows 2008R2
i was install oracle cluster success and when i was check crsctl check crs all was online and clufy stage -post crsinst was success
i left few days
and now the out put of
crsctl check crs is
crs-4638 oracle high availability services is online
crs-4535 connot communicate with cluster ready services
crs-4530 communications failure contactin cluster synchronization services daemon
crs-4534 connot communiccate with event manager
please help me!
any idea
Regards
Thanks in advance.
CRSD.LOG
=========================
Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:591920
2011-07-04 11:21:56.156: [ AGFW][5420] Received state change for ora.net1.network srv03 1 [old state = ONLINE, new state = OFFLINE]
2011-07-04 11:21:56.156: [ AGFW][5420] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:Invalid|MIDFrom:0|Type:4|Pri2|Id:31719]
2011-07-04 11:21:56.156: [ AGFW][5420] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:591920
2011-07-04 11:21:56.156: [ CRSPE][2228] State change received from srv03 for ora.net1.network srv03 1
2011-07-04 11:21:56.156: [ CRSPE][2228] Processing PE command id=471. Description: [Resource State Change (ora.net1.network srv03 1) : 000000000A090B00]
2011-07-04 11:21:56.156: [ CRSPE][2228] RI [ora.net1.network srv03 1] new external state [OFFLINE] old value: [ONLINE] on srv03 label = []
2011-07-04 11:21:56.156: [ CRSPE][2228] Resource Resource Instance ID[ora.net1.network srv03 1]. Values:
STATE=OFFLINE
TARGET=ONLINE
LAST_SERVER=srv03
CURRENT_RCOUNT=0
LAST_RESTART=0
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=1
STATE_CHANGE_VERS=0
LAST_FAULT=0
DEGREE_ID=1
ID=ora.net1.network srv03 1
Lock Info:
Write Locks:none
ReadLocks:|STATE INITED| has failed!
2011-07-04 11:21:56.156: [ CRSRPT][4916] Publishing event: Cluster Resource State Change Event for ora.net1.network:srv03 : 000000000670CBB0
2011-07-04 11:21:56.156: [ CRSRPT][4916] Publish to eons buffered event : 000000000670CBB0
2011-07-04 11:21:56.156: [ CRSPE][2228] Processing unplanned state change for [ora.net1.network srv03 1]
2011-07-04 11:21:56.156: [ CRSPE][2228] Scheduled local recovery for [ora.net1.network srv03 1]
2011-07-04 11:21:56.171: [ AGFW][5420] Agfw Proxy Server received the message: RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31723
2011-07-04 11:21:56.171: [ AGFW][5420] Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31723 to the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
2011-07-04 11:21:56.171: [ CRSPE][2228] Sending message to agfw: id = 31726
2011-07-04 11:21:56.171: [ CRSPE][2228] CRS-2672: Attempting to start 'ora.net1.network' on 'srv03'
2011-07-04 11:21:56.171: [ AGFW][5420] Agfw Proxy Server received the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726
2011-07-04 11:21:56.171: [ AGFW][5420] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726 to the agent C:\app\11.2.0\grid\bin\orarootagent.exe_system
2011-07-04 11:21:56.187: [ AGFW][5420] Received the reply to the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31727 from the agent C:\app\11.2.0\grid\bin\orarootagent.exe_system
2011-07-04 11:21:56.187: [ AGFW][5420] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726
2011-07-04 11:21:56.187: [ CRSPE][2228] Received reply to action [Start] message ID: 31726
2011-07-04 11:21:56.296: [ AGFW][5420] Received the reply to the message: RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31724 from the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
2011-07-04 11:21:56.296: [ AGFW][5420] ora.ons srv03 1 received state from probe request. Old state = ONLINE, New state = ONLINE
2011-07-04 11:21:56.296: [ AGFW][5420] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.ons srv03 1] ID 4097:31723
2011-07-04 11:21:56.296: [ AGFW][5420] Received the reply to the message: RESOURCE_START[ora.net1.network srv03 1] ID 4098:31727 from the agent C:\app\11.2.0\grid\bin\orarootagent.exe_system
2011-07-04 11:21:56.296: [ AGFW][5420] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.net1.network srv03 1] ID 4098:31726
2011-07-04 11:21:56.296: [ CRSPE][2228] Received reply to action [Start] message ID: 31726
2011-07-04 11:21:56.296: [ CRSPE][2228] CRS-2674: Start of 'ora.net1.network' on 'srv03' failed
2011-07-04 11:21:56.296: [ CRSPE][2228] Sequencer for [ora.net1.network srv03 1] has completed with error: CRS-0215: Could not start resource 'ora.net1.network'.
2011-07-04 11:21:56.296: [ CRSPE][2228] Failover cannot be completed for [ora.net1.network srv03 1]. Stopping it and the resource tree
2011-07-04 11:21:56.296: [ CRSPE][2228] Sending message to agfw: id = 31735
2011-07-04 11:21:56.296: [ AGFW][5420] Agfw Proxy Server received the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735
2011-07-04 11:21:56.296: [ AGFW][5420] Agfw Proxy Server forwarding the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735 to the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
2011-07-04 11:21:56.296: [ CRSPE][2228] CRS-2673: Attempting to stop 'ora.ons' on 'srv03'
2011-07-04 11:21:56.670: [ AGFW][5420] Received the reply to the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31736 from the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
2011-07-04 11:21:56.670: [ AGFW][5420] Agfw Proxy Server sending the reply to PE for message:RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735
2011-07-04 11:21:56.670: [ CRSPE][2228] Received reply to action [Stop] message ID: 31735
2011-07-04 11:21:58.854: [ AGFW][5420] Received the reply to the message: RESOURCE_STOP[ora.ons srv03 1] ID 4099:31736 from the agent C:\app\11.2.0\grid\bin\oraagent.exe_system
2011-07-04 11:21:58.854: [ AGFW][5420] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_STOP[ora.ons srv03 1] ID 4099:31735
2011-07-04 11:21:58.854: [ CRSPE][2228] Received reply to action [Stop] message ID: 31735
2011-07-04 11:21:58.854: [ CRSPE][2228] RI [ora.ons srv03 1] new external state [OFFLINE] old value: [ONLINE] label = []
2011-07-04 11:21:58.854: [ CRSRPT][4916] Publishing event: Cluster Resource State Change Event for ora.ons:srv03 : 000000000670CAD0
2011-07-04 11:21:58.854: [ CRSRPT][4916] Publish to eons buffered event : 000000000670CAD0
2011-07-04 11:21:58.854: [ CRSPE][2228] CRS-2677: Stop of 'ora.ons' on 'srv03' succeeded
2011-07-04 11:21:58.854: [ CRSPE][2228] PE Command [ Resource State Change (ora.net1.network srv03 1) : 000000000A090B00 ] has completed
2011-07-04 11:21:58.854: [ AGFW][5420] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:31744
2011-07-04 11:21:58.854: [ AGFW][5420] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:31744
2011-07-04 11:21:58.854: [ AGFW][5420] Agfw received reply from PE for resource state change for ora.net1.network srv03 1
Similar Messages
-
CRS not starting after Oracle 11g Grid was succesuflly installed
Hi:
Have 2 node RAC. My Shared storage is on openfiler.
Oracle 11g Grid was successfully installed and all the services was working fine. When i restarted the system I found CRS was not coming up. Remaining processes were up and running.
Let me explain what I did: (O/s RHEL 5.4 & oracle 11g R2 grid)
I have a partition in shared disk /dev/sdc1 (I have other partitions, but thought of using only one for OCR and voting disk). (did not define it as raw partition as I used to do in 10g RAC).
While installing Oracle grid and when it prompted to enter the Diskgroup for OCR and Voting disk I gave a name DGDATA and choose only one (external redundancy) disk “/dev/sdc1″ (it does not ask for one for OCR and one for Voting as it used to do in 10g clusterware installation). As per my assumption both OCR and Voting disk were created within /dev/sdc1 (after installation when I queried I got the same info)
The Oracle grid installation has gone through successfully. when I restart the system I find CRS not starting.
The log says: “Error PROC:26: Error while accessing the physical storage ASM……..
ORA-01034: oracle not available...
Could not init OCR, code:26….
Linux permission denied".
For your information….ASM instance is up (check in both the nodes) and I find the diskgroup mounted.
Can anyone help on this.
Also let me know how to uninstall Oracle 11g Grid software and configuration. If I have to reinstall the Oracle 11g grid software and config once again is it possible without reinstalling RHEL. Let me know the steps or any link please. don't want to get into a mess again.
Thanks in advance.
Regards
DineshJorg, I even tried to nomount the DB instance using initDBRAC.ora pfile. Then tried to create spfile still got the same error.
The second trace file which I have pasted looks like some memory dump.
ASM_DISKSTRING= /dev/sdc*
***********************************************************************1st trace file
Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/crs/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_dbrac)(ENVS='ORACLE_HOME=/crs/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@skyit6)(HOST=skyit6)(USER=oracle))))
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 01-JUN-2011 01:27:07
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 01-JUN-2011 01:27:07
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
TNS-00517: Lost contact
nt secondary err code: 32
nt OS err code: 0
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/crs/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_dbrac)(ENVS='ORACLE_HOME=/crs/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
Wed Jun 01 01:27:07 2011
ERROR: slave communication error with ASM; terminating process 16772
Errors in file /oraeng/app/oracle/product/diag/rdbms/dbrac/DBRAC/trace/DBRAC_ora_16772.trc:
----======================2nd trace file
Trace file /oraeng/app/oracle/product/diag/rdbms/dbrac/DBRAC/trace/DBRAC_ora_16772.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters option
ORACLE_HOME = /oraeng/app/oracle/product/11.2.0
System name: Linux
Node name: skyit6
Release: 2.6.18-164.el5
Version: #1 SMP Tue Aug 18 15:51:54 EDT 2009
Machine: i686
Instance name: DBRAC
Redo thread mounted by this instance: 0 <none>
Oracle process number: 25
Unix process pid: 16772, image: oracle@skyit6 (TNS V1-V3)
*** 2011-06-01 01:27:07.010
*** SESSION ID:(1.3) 2011-06-01 01:27:07.010
*** CLIENT ID:() 2011-06-01 01:27:07.010
*** SERVICE NAME:() 2011-06-01 01:27:07.010
*** MODULE NAME:(sqlplus@skyit6 (TNS V1-V3)) 2011-06-01 01:27:07.010
*** ACTION NAME:() 2011-06-01 01:27:07.010
ERROR: slave communication error with ASM; terminating process 16772
*** 2011-06-01 01:27:07.035
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=2, mask=0x0)
----- Error Stack Dump -----
----- Current SQL Statement for this session (sql_id=bff5sku8phfgx) -----
create spfile='+dgasmspfile/spfileDBRAC.ora' from pfile
*** 2011-06-01 01:27:07.070
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
skdstdst()+41 call kgdsdst() BF9E05FC ? 2 ?
ksedst1()+77 call skdstdst() BF9E05FC ? 0 ? 1 ? AB80E70 ?
852F42A ? AB80E70 ?
ksedst()+33 call ksedst1() 0 ? 1 ?
dbkedDefDump()+2699 call ksedst() 0 ? 0 ? BF9E0D40 ? 0 ? 0 ?
0 ?
ksedmp()+47 call dbkedDefDump() 2 ? 0 ?
kfTerminateMe()+75 call ksedmp() 2 ? 3 ? F1170AC ? F149234 ?
5 ? BF9E0A6C ?
kfnCheckCommError() call kfTerminateMe() F149234 ? BF9E0C1C ?
+32 B033D83 ? 29392378 ?
F149234 ? EB00601 ?
kfncSlaveSubmit()+5 call kfnCheckCommError() 29392378 ? F149234 ?
83
kfncFileDelete()+11 call kfncSlaveSubmit() BF9E0E5C ? 0 ? F147B24 ?
81
kfioDelete()+126 call kfncFileDelete() 29DEE0 ? 1C ?
ksfddel1()+582 call kfioDelete() 29DEE0 ? 1C ? D ?
ksfddel()+187 call ksfddel1() 29DEE0 ? 1C ? D ? 0 ?
BF9E0F2C ? 9BA2 ?
ksp_spfile_create() call ksfddel() 29DEE0 ? 1C ? D ? 0 ?
+677
kspcspfp()+791 call ksp_spfile_create() BF9E49D4 ? 0 ? 29DEE0 ? 1C ?
7D02 ? 0 ?
kspocte()+67 call kspcspfp() 29DEE0 ? 1C ? F074CC4 ? F ?
61,1 1%
Regards
Dinesh -
Hi,
I have just installed RAC in a test environment using vmware server 2. I have Oracle Enterprise Linux 5 and used oracle 11g (11.2.0.1). I have not installed database software yet and just installed oracle grid infrastructure.
When i restart any of the node and issue "crsctl check crs", i see following output.
[root@rac2 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
Now, "crs_stat -t" also does not work unless i manually start the crs daemon using "crsd start" command. Why crs daemon is not starting automatic unlike 10g RAC. Is it normal behaviour and what is proper way of making this daemon start automatic.
SalmanI am assuming you are running crsctl check crs immediately after reboot. It takes bit of time for crs to start and you do not have to start it manually. By default crs is started automatically and if for some reason you disabled it, you can enable it by using "crsctl enable crs" .
You can also check the status of individual services by using:
# crsctl check crsd -
Urgent: patched crs bundle patch #3. CRS not starting up
hi,
i had patched the 7117233 patch bundle. everything went fine on my rac . the database version is 10.2.0.3 i updated the crs to 10.2.0.3. with the patch,
i had issues with te post patch but it was some permission issue and it worked out.
but now the last :
custom/scripts/postrootpatch.sh -crshome /app/oracle/product/10.2.0/crs
Checking to se if Oracle CRS stack is already up...
Checking to se if Oracle CRS stack is already starting
/etc/init.d/init.cssd: line 320: /tmp/oratz.30171 : No such file or directory
/etc/init.d/init.cssd: line 320: /tmp/oratz.30193 : No such file or directory
/etc/init.d/init.cssd: line 320: /tmp/oratz.30215 : No such file or directory
/etc/init.d/init.cssd: line 320: /tmp/oratz.30240 : No such file or directory
/etc/init.d/init.cssd: line 320: /tmp/oratz.30262 : No such file or directory
Startup will be queued to init within 30 seconds
it has been freezed as such for more than 10 minutes .. Has not start up!
any sorta help will be most appreciated !
regards
Susan Juser10758089 wrote:
Hi Susan,
2 possible solutions:
1. as root user, "chmod -R 777 /tmp" and retry start CRSdon't set it to 777 as it would lead to a security nightmare.
the problem is indeed a permission problem. the script tries to write to the file as user oracle and to remove it afterwards. the error message is from the removal.
check, if oracle user can create files in /tmp and if not fix it by setting the proper permissions which are 1777 (drwxrwxrwt).
regards,
-ap -
OS: OEL 5 U4 x86_64
DB: Oracle 11.2.0.1 EE
Grid Infrastructure: Oracle 11.2.0.1
CRS and Voting disk Storage: ASM
Datafile and FRA storage: ASM
I'm not sure exactly what caused this, but anyways, I changed MTU from 1500 to 900 online. After some time, 3 out of 4 nodes in the cluster went down and CRS refuses to start on these nodes after trying the switch back from MTU 9000 to 1500, reboots, and making sure disk permissions and ownership are correct. The logs are not too helpful (and cryptic) so I'm at a loss and appreciate any ideas or help.
The installation was successful, the RAC was up for a few days while running some tests (including restart of a node). Currently only a single node has everything up and functional, the others are not working. Below are some output that might help:
[root@ucstst11 bin]# ./crsctl check cluster -n ucstst11
ucstst11:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
[root@ucstst11 bin]# ./crsctl start cluster -n ucstst11
CRS-2672: Attempting to start 'ora.cssd' on 'ucstst11'
CRS-2672: Attempting to start 'ora.diskmon' on 'ucstst11'
CRS-2676: Start of 'ora.diskmon' on 'ucstst11' succeeded
CRS-4404: The following nodes did not reply within the allotted time:
ucstst11
[root@ucstst11 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
[root@ucstst11 bin]# ./crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
[root@ucstst11 bin]# oracleasm querydisk -p CRSVOL01
Disk "CRSVOL01" is a valid ASM disk
/dev/sdz1: LABEL="CRSVOL01" TYPE="oracleasm"
/dev/sdcj1: LABEL="CRSVOL01" TYPE="oracleasm"
[root@ucstst11 bin]# ll /dev/sdz1 /dev/sdcj1
brw-rw---- 1 oracle dba 69, 113 Mar 27 19:00 /dev/sdcj1
brw-rw---- 1 oracle dba 65, 145 Mar 27 19:00 /dev/sdz1
[root@ucstst11 bin]# oracleasm querydisk -d CRSVOL01
Disk "CRSVOL01" is a valid ASM disk on device [65, 145]
From the functional node:
[root@ucstst12 bin]# ./crsctl check cluster -all
ucstst12:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
Cluster verification now hangs when it tries to contact the other nodes.
Please help!For the most part this issue has been resolved. The SA partially changed to jumbo frames (OS, but not the switch), we reverted all the jumbo frame changes and the system is back online, except for one node (the one which was working ironically) not being reported via "crsctl check cluster -all", and one instance not starting due to it not seeing an interconnect (weird).
We did attempt to fully implement jumbo frames but that did not work hence the reversion. -
Crs Not Starting _ private Interconnect Down
Hello All,
I Have Installed 2 node 10g R2(10.2.0.1) RAC on Solaris 10 T2000 Machines. Yesterday my Second Node Crs gone down. I tried to start it but it didn't start. Then i checked that Private IP (interconnect) is not Pinging from both the node. But Node 1 was up and working so my Users Can Connect to It.
But Today morning I see that Crs on node 1aslo goes down .
Is this is problem of private interconnect.? My network guys are trying to up Private Interconnect.
If Private Interconnect is down, why node 1 goes down after few hours. i think private interconnect is for interconnect with node 2 but node 2 is down .
Previously My interconnect was connected with cross cables now i have asked them to connect them through switch.
Help me Out.
Regards,
Pankaj.Previously My interconnect was connected with cross cables now i have asked them to connect them through switch
Even we are planning to do the same.Please share your experienceHope you have done this before - moving to switch
(Update for record id(s): 105681546)
QUESTION
========
1.Will the database and the Clusterware need to be shutdown etc?
2.Will our ip addresses need to be reconfigured?
3.Are there any steps that need to be carried out before unplugging the CROSS CABLE
and after the interconnect is connected to the switch...?
ANSWER
======
1. Yes, you have to stop CRS on each node.
2. No, not required.Provided you are planning to use same ip addresses.
3. Steps:
a. Stop CRS on each node. "crsctl stop crs"
b. Replace the crossover cable with switch.
c. Start the CRS on each node. "crsctl start crs"
Even we are planning to do the same.Please share your experienceFollowed by the above answers from Customer Support, It went smooth, we stopped all the services, and with both the nodes reboot.
Message was edited by:
Ravi Prakash -
Hi,
We had installed RAC DB namely ORCL on a two-node cluster abcoracledb01 and abcoracledb02 with the OUI launched from db01 node. After installation, the NETCA is able to identify the RAC DB ORCL on both the nodes as well as the ORCL1 instance on db01 node and ORCL2 instance on db02 node. Now when I try to launch the DBCA from db01 node, it prompts me with the two normal options Real Applications Cluster database and normal database on its welcome screen as it detects the CRS running on this node. The problem is coming when Im trying to launch DBCA from the second node as it does not extends the RAC option in its welcome screen. I can understand that the CRS is not running on this db02 instance and and i verified this by running the OLSNODES -v command that gives me an error stating that the cluster interface is not getting started.
Will highly appreciate if anyone can help me to get some referenctial documentation or diagnostic information that may help me in troubsleshooting the exact reason behind the CRS not getting started.
Thanks,
Manoj ([email protected])Ummm... Why are you trying to run the DBCA twice? The DBCA is really only there to give you a starter database configured for either OLTP, OLAP, or multi-purpose, but since both nodes will be attached to the same database, you shouldn't run the DBCA more than once. Once you have the database created with DBCA, attach the second node to it and go. Your problem is not with DBCA, it's in the cluster interconnect. Once that starts up, the second instance will attach to the database and go.
-
Crs will not start on 10.2.0.3
Hi All !
RHEL 4 AS U2, 2.6.9-42.0.3.EL
After installation of a patch 10.2.0.3 it has ceased to be started crs.
[root@racnode1 init.d] ./init.crs start
Startup will be queued to init within 30 seconds
Thus crs will not start
cd /u01/app/oracle/product/10.2.0/crs/install
[root@racnode1 install] ./root102.sh
All fades on string "Starting will be queued to init withhin 30 seconds"
In crsd.log:
2007-02-01 12:11:53:10.143 [ COMMCRS][2808851376]clsc_connect (0x8b5b930) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
Help me please !!!We experienced the same failure with CRS a none start after the patch (from 10.2.0.1 to 10.2.0.3)-- running RH 4.3 with Linux 2.6.9-42.ELsmp on 32 bit HP boxen. SELINUX is very disabled! Logging is no help. Could there be a connection with the following?:
11.8 VIP May Relocate to the Last Node While Upgrading Oracle
Clusterware
Rolling upgrade or upgrade of Oracle Clusterware to 10.2.0.3 patch set
may cause VIP to move to the last node.
Workaround:
Enter the following command to relocate the VIP to the preferred node:
crs_relocate VIP Resource
Then try to restart the CRS daemons
This issue is tracked with Oracle bug 5673067.
However, I cannot find a bug with that id...
Hope an answer is forthcoming, as this is holding up the proof-of-concept test for a mid-sized project...
Thanks -
CRS could not start after reboot
I installed oracle 10g R2 RAC on RHEL4 U4 with 2 nodes. It runs fine on 2 nodes (without reboot after installation). But when I rebooted one node, I found that the CRS daemon could not start automatically. The boot menu show "Starting CRS" [OK]. The "init.crs start" command won't init the CRS daemon.
$ ./crs_stat
CRS-0184: Cannot communicate with the CRS daemon.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA1/RAC/spfileRAC.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA1/RAC/spfileRAC.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Manager
$ /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
$ cat /etc/inittab
id:5:initdefault:
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
$ ls /etc/rc5.d
K01tog-pegasus K35smb K85mdmpd S05openibd S18rpcidmapd S55sshd S96readahead
K02NetworkManager K35vncserver K87auditd S06cpuspeed S19rpcgssd S56rawdevices S97messagebus
K05saslauthd K35winbind K87ipmi S08arptables_jf S19vmware-tools S56xinetd S97rhnsd
K10dc_server K40smartd K89netplugd S08iptables S25netfs S58ntpd S98cups-config-daemon
K10psacct K50ibmasm K90bluetooth S09isdn S26apmd S80sendmail S98haldaemon
K12dc_client K50netdump K94diskdump S10network S26lm_sensors S85gpm S99local
K15httpd K50snmpd K96pcmcia S12syslog S27vasd S90crond
K20nfs K50snmptrapd S00microcode_ctl S13irqbalance S28autofs S90xfs
K24irda K50tux S01sysstat S13portmap S29oracleasm S95anacron
K25squid K73ypbind S04readahead_early S14nfslock S44acpid S95atd
K30spamassassin K74nscd S05kudzu S15mdmonitor S55cups S96init.crs
Do I miss any configuration to initialize the CRSD at boot?
thanks, ezleedid you run cluster verification utility for node connectivity?
http://docs.oracle.com/cd/E11882_01/rac.112/e16794/cvu.htm#autoId19
also check the logs ad $GRID_HOME/log/<hostname>/...
starting with alert.log, then the ones its referencing -
CRS-0215: Could not start resource
Hello
This is my first time installing clusterware. However, i have not been too successful at it. This is my configuration:
Operating System: RHES 5.3
Oracle Database 11gR1
OpenFiler used to configure shared disks
After several attempts, i was able to run the root.sh script on both nodes.
Output on first node:
[root@vx0301 oracle]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory to 770.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete
[root@vx0301 oracle]# /u01/crs11g/root.sh
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: vx0301 vx0301-priv vx0301
node 2: vx0302 vx0302-priv vx0302
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
vx0301
Cluster Synchronization Services is inactive on these nodes.
vx0302
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
Output on second node:
[root@vx0301 oracle]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory to 770.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete
[root@vx0301 oracle]# /u01/crs11g/root.sh
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: vx0301 vx0301-priv vx0301
node 2: vx0302 vx0302-priv vx0302
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
vx0301
vx0302
I also got the errors:
Starting GSD application resource on (2) nodes1:CRS-0215: Could not start resource 'ora.vx0301.gsd'
Starting ONS application resource on (2) nodes1:CRS-0215: Could not start resource 'ora.vx0301.ons'
There were no log files.
When i clicked ok on the execute scripts page, i got the following errors in the configuration assistants page:
Output generated from configuration assistant "Oracle Notification Server Configuration Assistant":
Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
The ONS configuration failed to create
Configuration assistant "Oracle Notification Server Configuration Assistant" failed
The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------Output generated from configuration assistant "Oracle Notification Server Configuration Assistant":
Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
The ONS configuration failed to create
Configuration assistant "Oracle Notification Server Configuration Assistant" failed
The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------Output generated from configuration assistant "Oracle Notification Server Configuration Assistant" (attempt 2):
Command = /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
The ONS configuration failed to create
Configuration assistant "Oracle Notification Server Configuration Assistant" failed
The "/u01/crs11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.-----------------------------------------------------------------------------
Contents of the /u01/crs11g/cfgtoollogs/configToolFailedCommands script:
[oracle@vx0301 ~]$ cat /u01/crs11g/cfgtoollogs/configToolFailedCommands
# Copyright (c) 1999, 2007, Oracle. All rights reserved.
/u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
/u01/crs11g/bin/oifcfg setif -global eth0/172.30.4.0:public eth1/192.168.1.0:cluster_interconnect
/u01/crs11g/bin/cluvfy stage -post crsinst -n vx0301,vx0302
I tried running /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251 manually from the terminal:
[root@vx0301 oracle]# /u01/crs11g/install/onsconfig add_config vx0301:6251 vx0302:6251
The ONS configuration failed to create
Output of the ons.log file:
[root@vx0301 oracle]# cat /u01/crs11g/log/vx0301/racg/ons.log
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2010-01-25 13:59:15.786: [ RACG][3055679168] [10113][3055679168][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
2010-01-25 13:59:33.359: [ RACG][3055584960] [10229][3055584960][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
2010-01-25 14:01:00.319: [ RACG][3086849728] [10734][3086849728][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
2010-01-25 14:02:02.723: [ RACG][3086628544] [11105][3086628544][default]: clsrons: procr_init:PROC-32: Cluster Ready Services on the local node is not running Messaging error [9] status = 32
[root@vx0301 oracle]#
I am absoutely stumped as to what to do next. Any help is greatly appreciated.I tried running ruclufvy.sh stage -post crsinst -n vx0301,vx0302 -verbose. Here is the output:
[oracle@vx0301 clusterware]$ ./runcluvfy.sh stage -post crsinst -n vx0301,vx0302 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "vx0301"
Destination Node Reachable?
vx0301 yes
vx0302 yes
Result: Node reachability check passed from node "vx0301".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
vx0302 passed
vx0301 passed
Result: User equivalence check passed for user "oracle".
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
vx0302 running
vx0301 running
Result: Daemon status check passed for "CSS daemon".
Cluster manager integrity check passed.
Checking cluster integrity...
Node Name
vx0301
vx0302
Cluster integrity check passed
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
ERROR:
OCR integrity is invalid.
OCR integrity check failed.
Checking CRS integrity...
Checking daemon liveness...
Check: Liveness for "CRS daemon"
Node Name Running
vx0302 yes
vx0301 yes
Result: Liveness check passed for "CRS daemon".
Checking daemon liveness...
Check: Liveness for "CSS daemon"
Node Name Running
vx0302 yes
vx0301 yes
Result: Liveness check passed for "CSS daemon".
Checking daemon liveness...
Check: Liveness for "EVM daemon"
Node Name Running
vx0302 yes
vx0301 yes
Result: Liveness check passed for "EVM daemon".
Liveness of all the daemons
Node Name CRS daemon CSS daemon EVM daemon
vx0302 yes yes yes
vx0301 yes yes yes
Checking CRS health...
Check: Health of CRS
Node Name CRS OK?
vx0302 yes
vx0301 unknown
Result: CRS health check failed.
CRS integrity check failed.
Checking node application existence...
Checking existence of VIP node application
Node Name Required Status Comment
vx0302 yes exists passed
vx0301 yes exists passed
Result: Check passed.
Checking existence of ONS node application
Node Name Required Status Comment
vx0302 no exists passed
vx0301 no exists passed
Result: Check passed.
Checking existence of GSD node application
Node Name Required Status Comment
vx0302 no exists passed
vx0301 no exists passed
Result: Check passed.
Post-check for cluster services setup was unsuccessful on all the nodes.
[oracle@vx0301 clusterware]$
I also tried running ./ocrchek on both nodes. Here is the output:
[oracle@vx0301 bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 3306636
Used space (kbytes) : 2056
Available space (kbytes) : 3304580
ID : 1425438992
Device/File Name : /dev/sdb1
Device/File integrity check failed
Device/File not configured
Cluster registry integrity check failed
[oracle@vx0301 bin]$ ssh vx0302
Last login: Mon Jan 25 13:52:03 2010 from vx0301
[oracle@vx0302 ~]$ cd /u01/crs11g/bin/
[oracle@vx0302 bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 3306636
Used space (kbytes) : 2056
Available space (kbytes) : 3304580
ID : 1425438992
Device/File Name : /dev/sdb1
Device/File integrity check failed
Device/File not configured
Cluster registry integrity check failed
[oracle@vx0302 bin]$
From the results, the ids are the same, and so is the OCR disk. But the integrity check has failed. How do i interpret this output? -
Could Not start CRS or HAS in RAC 11.2.0.1
Dear Legends,
We have the following environment in SOLARIS 2 Node Database. Not sure why the 2nd node went down today morning but 1st Node is still Up and Running.
My Tries so for
1. Changed the following line as per Doc 1368382.1 from
h1:3:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
To
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
2. Provided a Restart of Entire Linux Box
3. Tried issuing the following command ./crsctl start crs but Error out as follows
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
4. Tried checking in Database, ASM alert logs nothing related with OHASD.
5. Also checked in ohasd.log and ohasdOUT.log not able to find any thing.
Please help me. This is our Production Environment.
Thanks,
KarthikKarthik,
I encountered the same error few days back and below is what I did to resolve the issue as mentioned in metalink 1368382.1. Did you see anything in the crs logs?
did you follow all the steps in the solution section below like killiing remaining rc3 scripts?
checks :
1. Command '$GRID_HOME/bin/crsctl check crs' returns error:
CRS-4639: Could not contact Oracle High Availability Services
2. Command 'ps -ef | grep init' does not show a line similar to:
root 4878 1 0 Sep12 ? 00:00:02 /bin/sh /etc/init.d/init.ohasd run
3. Command 'ps -ef | grep d.bin' does not show a line similar to:
root 21350 1 6 22:24 ? 00:00:01 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
Or it may only show "ohasd.bin reboot" process without any other processes
4. ohasd.log report:
2013-11-04 09:09:15.541: [ default][2609911536] Created alert : (:OHAS00117:) : TIMED OUT WAITING FOR OHASD MONITOR
5. ohasOUT.log report:
2013-11-04 08:59:14
Changing directory to /u01/app/11.2.0/grid/log/lc1n1/ohasd
OHASD starting
Timed out waiting for init.ohasd script to start; posting an alert
Solutions:
1. Add the following line to /etc/inittab
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
and then run "init q" as the root user.
2. Run command 'ps -ef | grep rc' and kill any remaining rc3 scripts that appear to be stuck.
3. Remove the bad entry before init.ohasd. Consult with OS vendor if "init q" does not spawn "init.ohasd run" process. As a workaround,
start the init.ohasd manually, eg: as root user, run "/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null &"
4. Enable CRS autostart:
# crsctl enable crs
# crsctl start crs
5. Restore OLR from backup, as root user:
# touch $GRID_HOME/cdata/<node>.olr
# chown root:oinstall $GRID_HOME/cdata/<node>.olr
# ocrconfig -local -restore$GRID_HOME/cdata/<node>/backup_<date>_<num>.olr
# crsctl start crs
If OLR backup does not exist for any reason, perform deconfig and rerun root.sh is required to recreate OLR, as root user:
# $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
# $GRID_HOME/root.sh
6. If above does not help, check OS messages for ohasd.bin logger message and manually execute crswrapexece.pl command mentioned in the OS message with LD_LIBRARY_PATH set to <GRID_HOME/lib to continue debug. -
CRS Can not start instance PRKP-1001
Hello
I have a 2 node database and on the first node I can not start the instance using srvctl start instance it gives the following error: PRKP-1001 : Error starting instance htdb1 on node telepin1. When I start it using sqlplus it is ok.
Can someone please explain?
ThanksDo you also see CRS-0215? There's a known problem , when parameter sqlnet.inbound_connect_timeout is set in sqlnet.ora, it should be unset. Unfortunately the error is a little bit generic, if this parameter is currently not set, you have another problem.
Werner -
Cssd does not start in non-RAC environment Thus we can not bring up ASM
Non-RAC environment
ASM version = 11.1.0.7
HP-UX Itanium 11.23
After power outage, CSSD does not start on non-RAC environment
Running as root "/sbin/init.d/init.cssd start" does not start cssd
Oracle support tried "$ASM_HOME/bin/localconfig delete" and "$ASM_HOME/bin/localconfig add"
but it did not start CSS
Oracle support tried "$ASM_HOME/bin/localconfig reset $ASM_HOME"
It started the CSSD and the "crsctl check css" came back with CSS is healthy
But around 1 minute later it rebooted the server and when it came up again CSS does not start.
They checked /etc/inittab and it looked fine.
Before the reboot we saw this message in the /var/adm/syslog/OLDsyslog.log:
Cluster Ready Services completed waiting on dependencies
Again it is a NON-RAC environment. We only need CSSD for ASM. We do not have CRS installed on this server.
Our test system has been down for a week and we did not get the resolution from Oracle support yet !
Any pointers are greately appriciated.
Thanks,
DzungHere is the message in $ASM_HOME/log/<hostname>/alert<hostname>.log :
2010-07-16 09:42:02.956
[client(11930)]CRS-1006:The OCR location /db/app/oracle/product/11.1/cdata/localhost/local.ocr is inacce
ssible. Details in /db/app/oracle/product/11.1/log/rmodbd01/client/clscfg10.log.
2010-07-16 09:42:02.971
[client(11930)]CRS-1006:The OCR location /db/app/oracle/product/11.1/cdata/localhost/local.ocr is inacce
ssible. Details in /db/app/oracle/product/11.1/log/rmodbd01/client/clscfg10.log.
2010-07-16 09:42:03.054
[client(11930)]CRS-1013:The OCR at /db/app/oracle/product/11.1/cdata/localhost/local.ocr was successfull
y formatted using version 2. Ignore earlier CRS-1006 messages if any.
2010-07-16 09:42:46.379
[cssd(12297)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rmodbd01 .
Here is the message in $ASM_HOME/log/<hostname>/cssd/cssdOUT.log:
setsid: failed with -1/1
s0clssscGetEnvOracleUser: calling getpwnam_r for user oracle
s0clssscGetEnvOracleUser: info for user oracle complete
07/16/10 09:42:36: CSSD starting
Here is the message in $ASM_HOME/log/<hostname>/cssd/ocssd.log:
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 7, from 6, changes 6
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmSetVersions: properties common to all peers: 1,2,3,4,5,6,7
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmEstablishMasterNode: MASTER for 174732166 is node(0) birth(174732166)
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 8, from 7, changes 7
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 9, from 8, changes 8
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssgmCompareSwapEventValue: changed CmInfo State val 10, from 9, changes 9
[ CSSD]CLSS-3000: reconfiguration successful, incarnation 174732166 with 1 nodes
[ CSSD]CLSS-3001: local node number 0, master node number 0
[ CSSD]2010-07-16 09:42:46.378 [18] >TRACE: clssscSAGEInitFenceCompl: Completing kgzf fence initialization
[ CSSD]2010-07-16 09:42:46.394 [12] >TRACE: clssgmUpdateEventValue: Client listener incarn val 174732166, changes 1
[ CSSD]2010-07-16 09:42:46.395 [12] >TRACE: clssgmAllocProc: (60000000003c7120) allocated
[ CSSD]2010-07-16 09:42:46.395 [12] >TRACE: clssgmAllocProc: (60000000003c73a0) allocated
[ CSSD]2010-07-16 09:42:46.396 [14] >TRACE: Connect request from user oracle
[ CSSD]2010-07-16 09:42:46.396 [14] >TRACE: Connect request from user root
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c7120 - 1,2,3
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2810) proc(60000000003c7120) pid(12350) version 11:1:1:4
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c73a0 - 1,2,3
[ CSSD]2010-07-16 09:42:46.397 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2990) proc(60000000003c73a0) pid(12131) version 11:1:1:4
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(1/600000000096e7c0)
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 1 (600000000096e7c0)
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmJoinGrock: local grock CSS_INTERNAL_NODE_GROUP new client 600000000096e7c0 with con 60000000003b2b10, requested num 0
[ CSSD]2010-07-16 09:42:46.402 [12] >TRACE: clssgmAddNodeGrpMember: member (60000000009e0030) added
[ CSSD]2010-07-16 09:42:46.403 [12] >TRACE: clssgmGroupState: requested group state of group localhost_NG, member count 0
[ CSSD]2010-07-16 09:42:46.403 [12] >TRACE: clssgmGroupState: requested group state of group localhost_NG, member count 0
[ CSSD]2010-07-16 09:42:46.404 [12] >TRACE: clssgmDeadProc: proc 60000000003c73a0
[ CSSD]2010-07-16 09:42:46.404 [12] >TRACE: clssgmDestroyProc: cleaning up proc(60000000003c73a0) con(60000000003b2990) skgpid ospid 12131 with 0 clients, refcount 0
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmGroupState: requested group state of unknown group MASTER#DISKMON#GROUP
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmGroupState: requested group state of group MASTER#DISKMON#GROUP, member count 0
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: KGZF: context successfully initialized, API version 1.4, using pipe default
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssscSAGEInitFenceCompl: kgzf fence initialization successfully completed
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmReconfigThread: CSS/GM open for global group registrations
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmReconfigThread: completed for reconfig(174732166), with status(1)
[ CSSD]2010-07-16 09:42:46.425 [18] >TRACE: clssgmUpdateEventValue: Reconfig Event val 2, changes 2
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmWaitOnEventValue: after Reconfig Event val 2, eval 2 waited 47
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(2/600000000096e870)
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmUpdateEventValue: Reconfig Event val 0, changes 3
[ CSSD]2010-07-16 09:42:46.425 [1] >TRACE: clssgmStartNMMon: previous reconfig complete, incarnation(174732166)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmJoinGrock: global grock MASTER#DISKMON#GROUP#MX new client 600000000096e870 with con 60000000003b2990, requested num -1
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddGrockMember: adding member to grock MASTER#DISKMON#GROUP#MX
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddMember: granted member(0) flags(0x2) node(0) grock (6000000000989e50/MASTER#DISKMON#GROUP#MX)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmQueueGrockEvent: lockName(MASTER#DISKMON#GROUP#MX) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmCommonAddMember: global lock grock MASTER#DISKMON#GROUP#MX member(0/Local) node(0) flags 0x2 0x2
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmRegisterClient: proc(1/60000000003c7120), client(3/600000000096e920)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 3 (600000000096e920)
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmJoinGrock: global grock MASTER#DISKMON#GROUP new client 600000000096e920 with con 60000000003b2bd0, requested num 0
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddGrockMember: adding member to grock MASTER#DISKMON#GROUP
[ CSSD]2010-07-16 09:42:46.425 [12] >TRACE: clssgmAddMember: new master 0 for group(MASTER#DISKMON#GROUP)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmAddMember: Adding fencing for member 0, group MASTER#DISKMON#GROUP, death 1, SAGE 0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmAddMember: member (0/60000000009e0230) added. pbsz(72) prsz(0) flags 0x0 to grock (600000000098a170/MASTER#DISKMON#GROUP)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmQueueGrockEvent: groupName(MASTER#DISKMON#GROUP) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x78, state 0x0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmCommonAddMember: global group grock MASTER#DISKMON#GROUP member(0/Local) node(0) flags 0x0 0x1e00
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmExitGrock: client 2 (600000000096e870), grock MASTER#DISKMON#GROUP#MX, member 0
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock MASTER#DISKMON#GROUP#MX
[ CSSD]2010-07-16 09:42:46.426 [12] >TRACE: clssgmRemoveMember: grock MASTER#DISKMON#GROUP#MX, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 3
[ CSSD]2010-07-16 09:42:48.405 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:48.405 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:42:52.444 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:52.444 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:42:56.484 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:42:56.484 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:00.516 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:00.516 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:04.563 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:04.563 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:08.603 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:08.603 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:12.643 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:12.643 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:16.676 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:16.676 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:20.723 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:20.723 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:24.762 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:24.762 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:28.802 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:28.802 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:32.842 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:32.842 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:36.882 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:36.882 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:40.922 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:40.922 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:44.964 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:44.964 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:49.002 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:49.002 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:53.043 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:53.043 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:57.085 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:43:57.085 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:43:58.354 [12] >TRACE: clssgmAllocProc: (60000000003c7ee0) allocated
[ CSSD]2010-07-16 09:43:58.355 [14] >TRACE: Connect request from user oracle
[ CSSD]2010-07-16 09:43:58.356 [12] >TRACE: clssgmClientConnectMsg: properties of cmProc 60000000003c7ee0 - 1,2,3
[ CSSD]2010-07-16 09:43:58.356 [12] >TRACE: clssgmClientConnectMsg: Connect from con(60000000003b2990) proc(60000000003c7ee0) pid(13157) version 11:1:1:4
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmRegisterClient: proc(2/60000000003c7ee0), client(1/600000000096e870)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 1 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmJoinGrock: global grock CLSSSCHECK_GROUP new client 600000000096e870 with con 60000000003b2c90, requested num -1
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddGrockMember: adding member to grock CLSSSCHECK_GROUP
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: new master 0 for group(CLSSSCHECK_GROUP)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: Adding fencing for member 0, group CLSSSCHECK_GROUP, death 1, SAGE 0
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmAddMember: member (0/60000000009e0130) added. pbsz(8) prsz(8) flags 0x0 to grock (6000000000989e50/CLSSSCHECK_GROUP)
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmQueueGrockEvent: groupName(CLSSSCHECK_GROUP) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x0, state 0x0
[ CSSD]2010-07-16 09:43:58.360 [12] >TRACE: clssgmCommonAddMember: global group grock CLSSSCHECK_GROUP member(0/Local) node(0) flags 0x0 0x0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 1 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExitGrock: client 1 (600000000096e870), grock CLSSSCHECK_GROUP, member 0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock CLSSSCHECK_GROUP
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRemoveMember: grock CLSSSCHECK_GROUP, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 2
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRegisterClient: proc(2/60000000003c7ee0), client(2/600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmJoinGrock: global grock CLSSSCHECK_LOCK new client 600000000096e870 with con 60000000003b2c90, requested num -1
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmAddGrockMember: adding member to grock CLSSSCHECK_LOCK
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmAddMember: granted member(0) flags(0x2) node(0) grock (6000000000989e50/CLSSSCHECK_LOCK)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmQueueGrockEvent: lockName(CLSSSCHECK_LOCK) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmCommonAddMember: global lock grock CLSSSCHECK_LOCK member(0/Local) node(0) flags 0x2 0x2
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExecuteClientRequest: GRKEXIT recvd from client 2 (600000000096e870)
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmExitGrock: client 2 (600000000096e870), grock CLSSSCHECK_LOCK, member 0
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmUnregisterPrimary: Unregistering member 0 (60000000009e0130) in global grock CLSSSCHECK_LOCK
[ CSSD]2010-07-16 09:43:58.361 [12] >TRACE: clssgmRemoveMember: grock CLSSSCHECK_LOCK, member number 0 (60000000009e0130) node number 0 state 0x14 member refcnt 0 grock type 3
[ CSSD]2010-07-16 09:43:58.362 [12] >TRACE: clssgmDeadProc: proc 60000000003c7ee0
[ CSSD]2010-07-16 09:43:58.362 [12] >TRACE: clssgmDestroyProc: cleaning up proc(60000000003c7ee0) con(60000000003b2990) skgpid ospid 13157 with 0 clients, refcount 0
[ CSSD]2010-07-16 09:44:01.125 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:01.125 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:05.164 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:05.164 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:09.196 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:09.196 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:13.236 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:13.236 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:17.284 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:17.284 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:21.322 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:21.322 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:25.316 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:25.316 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[ CSSD]2010-07-16 09:44:30.333 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:30.333 [8] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[ CSSD]2010-07-16 09:44:34.324 [8] >TRACE: clssnmSendingThread: sending status msg to all nodes
[ CSSD]2010-07-16 09:44:34.324 [8] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
Here is the message in $ASM_HOME/log/<hostname>/diskmon/client.log:
[ DISKMON] 07/16/2010 18:33:32.050 dskm_send_command: process 23246 sending command 8 to master diskmon listening on default pipe
[ DISKMON] 07/16/2010 18:33:32.080 dskm_send_command3: skgznp_connect failed with error 56815
[ DISKMON] 07/16/2010 18:33:32.080 dskm_send_command3: error 56815 at location skgznpcon6 - connect() - Connection refused
Here is the message in $ASM_HOME/log/<hostname>/diskmon/diskmonOUT.log:
setsid: failed with -1/1
dskm_getenv_oracle_user: calling getpwnam_r for user oracle
dskm_getenv_oracle_user: info for user oracle complete
07/16/10 18:33:31: Master Diskmon starting
Here is the message in $ASM_HOME/log/<hostname>/diskmon/diskmon.log:
[ DISKMON] 07/16/2010 09:42:35.573 dskm main: starting up
[ DISKMON] 07/16/2010 09:42:35.588 [12350:3] dskm_rac_thrd_main: running
[ DISKMON] 07/16/2010 09:42:35.588 [12350:1] dskm_rac_thrd_creat2: got the post from the css event handling thread
[ DISKMON] 07/16/2010 09:42:35.589 [12350:1] dskm main: startup complete
[ DISKMON] 07/16/2010 09:42:35.589 [12350:1] listening on -> default pipe
[ DISKMON] 07/16/2010 09:42:35.792 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:36.385 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:36.906 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:37.426 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:37.945 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:38.465 clsc_connect: (6000000000700420) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rmodbd01_))
[ DISKMON] 07/16/2010 09:42:46.376 [12350:1] dskm_slave_thrd_creat: thread created
[ DISKMON] 07/16/2010 09:42:46.376 [12350:11] dskm_slave_thrd_main1: slave 0 running
[ DISKMON] 07/16/2010 09:42:46.376 [12350:11] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_proc_identify8: client kgzf/12297, version 0x01020000, slave 0, reid cid=3e0391f05e06cfafbf7419d7cf085a44,icin=174732166,nmn=0,lnid=174732166,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_send_version1:
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_send_version4: done
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_process_msg5: received msg type KGZM_KGZF_HANDSHAKE (0x0010)
[ DISKMON] 07/16/2010 09:42:46.389 [12350:11] dskm_proc_kgzf_handshake3: client kgzf/12297, kgzf version 0x00010004, slave 0
[ DISKMON] 07/16/2010 09:42:46.401 [12350:3] dskm_clss_ini2: successful clsssinit(), clssvers 2.1
[ DISKMON] 07/16/2010 09:42:46.402 [12350:3] dskm_clss_ini12: node rmodbd01 (0) registered in cluster
[ DISKMON] 07/16/2010 09:42:46.403 [12350:3] dskm_reid_ini12: diskmon reid cid=3e0391f05e06cfafbf7419d7cf085a44,icin=174732166,nmn=0,lnid=174732166,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=12350,opsn=1279291355,lvl=process
[ DISKMON] 07/16/2010 09:42:46.424 [12350:3] dskm_sage_config: CELL storage configuration file cellinit.ora not found
[ DISKMON] 07/16/2010 09:42:46.425 [12350:3] dskm_nfy_kgzf1: notified thread kgzf enabled
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_proc_kgzf_handshake5: got the post from the hb thread
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_proc_kgzf_handshake9: done, kgzf enabled
[ DISKMON] 07/16/2010 09:42:46.425 [12350:11] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
[ DISKMON] 07/16/2010 09:42:46.426 [12350:3] dskm_rac_ini22: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 0 (GSDGRPSZ 512) -
Oracle ODB Console will not start - Oracle 11.1g
PC info: Windows XP
Oracle new install: 11.1 g
Listener is running. I can log-in to SQLPLUS from DOS as sysdba and run commands. But I can not start the Oracle Enterprise Manager session. I get the error " Can't establish connection to xx.xxx.xx.xxx:1158 "
When I try to start the service in windows ORACLE DB CONSOLE ORCL, I get the following error:
================================================================================
Windows could not start the ORACLE DB CONSOLE ORCL on local computer. For more review the system even log. If this is a non-Microsoft service, contact the service vendor, and refer to the service-specific error code 2.
======================================================================================
Please advice.
I am seeing this error in log client folder:
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2012-04-11 08:03:28.968: [ OCROSD][1884]utgdv:1:could not open registry key SOFTWARE\Oracle\ocr os error The system could not find the environment option that was entered.
2012-04-11 08:03:29.015: [ OCRRAW][1884]proprinit: Could not open raw device
2012-04-11 08:03:29.015: [ default][1884]a_init:7!: Backend init unsuccessful : [33]
2012-04-11 08:03:31.437: [ CSSCLNT][1884]clsssinit: error(32 PROC-32: Cluster Ready Services on the local node is not running Messaging error [9]) in OCR initialization
Thanks.
Edited by: 789308 on Apr 11, 2012 6:32 AM
emomos log last entries ( note they are old from January, I am not seeing new error logs):
==========================================================================================
2012-01-18 16:54:32,765 [shutdownHookThread] WARN jdbc.ConnectionCache _getConnection.353 - Got a fatal exeption when getting a connection; Error code = 17002; Cleaning up cache and retrying
2012-01-18 16:54:53,984 [shutdownHookThread] ERROR em.notification unregisterOMS.1417 - Error unregistering: Io exception: The Network Adapter could not establish the connection
java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:390)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:519)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:167)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:816)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:325)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:235)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:157)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:94)
at oracle.jdbc.pool.OracleImplicitConnectionCache.makeCacheConnection(OracleImplicitConnectionCache.java:1702)
at oracle.jdbc.pool.OracleImplicitConnectionCache.getCacheConnection(OracleImplicitConnectionCache.java:575)
at oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(OracleImplicitConnectionCache.java:435)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:432)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:203)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:179)
at oracle.sysman.util.jdbc.ConnectionCache._getConnection(ConnectionCache.java:359)
at oracle.sysman.util.jdbc.ConnectionCache._getConnection(ConnectionCache.java:322)
at oracle.sysman.util.jdbc.ConnectionCache.getUnwrappedConnection(ConnectionCache.java:575)
at oracle.sysman.emSDK.svc.conn.FGAConnectionCache.getFGAConnection(FGAConnectionCache.java:207)
at oracle.sysman.emSDK.svc.conn.ConnectionService.getSystemConnection(ConnectionService.java:1304)
at oracle.sysman.emdrep.notification.NotificationMgr.unregisterOMS(NotificationMgr.java:1408)
at oracle.sysman.emdrep.notification.NotificationMgr.destroy(NotificationMgr.java:1867)
at oracle.sysman.emSDK.svc.ServiceUtil.cleanupServices(ServiceUtil.java:212)
at oracle.sysman.eml.app.ContextInitializer.contextDestroyed(ContextInitializer.java:878)
at com.evermind.server.http.HttpApplication.destroyContextListeners(HttpApplication.java:5651)
at com.evermind.server.http.HttpApplication.destroy(HttpApplication.java:5618)
at com.evermind.server.http.HttpSite.destroy(HttpSite.java:865)
at com.evermind.server.http.HttpServer.destroy(HttpServer.java:549)
at com.evermind.server.ApplicationServer.destroy(ApplicationServer.java:1937)
at com.evermind.server.ApplicationServerShutdownHandler.run(ApplicationServerShutdownHandler.java:94)
at java.lang.Thread.run(Thread.java:595)
=================================================================================================
Edited by: 789308 on Apr 11, 2012 7:13 AM
When I run this command: C:\>emctl status dbconsole
I get : ENVIRONMENT VARIABLE ORACLE_SID NOT DEFINED. PLEASE DEFINE IT.
Edited by: 789308 on Apr 11, 2012 7:22 AM
Edited by: 789308 on Apr 11, 2012 7:41 AM
This is what I did:
SQL> emctl status dbconsole
SP2-0734: unknown command beginning "emctl stat..." - rest of line ignored.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Pr
oduction
With the Partitioning, OLAP, Data Mining and Real Application Testing options
C:\Documents and Settings\AIM>emctl status dbconsole
Environment variable ORACLE_SID not defined. Please define it.
C:\Documents and Settings\AIM>set oracle_sid=orcl
C:\Documents and Settings\AIM>emctl stop dbconsole
OC4J Configuration issue. C:\app\AIM\product\11.1.0\db_1/oc4j/j2ee/OC4J_DBConsol
e_10.101.15.141_orcl not found.
C:\Documents and Settings\AIM>emctl start dbconsole
OC4J Configuration issue. C:\app\AIM\product\11.1.0\db_1/oc4j/j2ee/OC4J_DBConsol
e_10.101.15.141_orcl not found.
Edited by: 789308 on Apr 11, 2012 7:42 AM
Edited by: 789308 on Apr 11, 2012 10:31 AMI get : ENVIRONMENT VARIABLE ORACLE_SID NOT DEFINED. PLEASE DEFINE IT.so then
SET ORACLE_SID=FOOBAR -
CSS does not start using 11g RH4 SCSI drives
Hi All...
Followed all necessary procedures for setting up oracle. The problem is when I try and get ASM working, I get to the part that says CSS is not running and then do a $ORACLE_HOME/bin/localconfig add as root with all environmental variables set up properly. Furthermore, oracleasm was previously installed, properly configured because I can run the scan, query and list and it comes back correctly.
[root@db raw]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db raw]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
At the end of this process CSS fails to start. There is nothing placed into the /var/tmp/.oracle directory. I then ran the $ORACLE_HOME/bin/localconfig reset $ORACLE_HOME and get the same result.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started
While it is running I see:
oracle:/u01/app/oracle> ps -ef | grep crs
root 18399 18142 22 19:31 pts/2 00:00:12 /u01/app/oracle/product/11.1.0/db_1/bin/crsctl.bin check install -wait 600
oracle 18521 18432 0 19:32 pts/4 00:00:00 grep crs
ps -ef | grep css
oracle 18653 18432 0 19:33 pts/4 00:00:00 grep css
[root@db oracle]# tail ocr.loc
ocrconfig_loc=/u01/app/oracle/product/11.1.0/db_1/cdata/localhost/local.ocr
local_only=TRUE
Contents of css log file in $ORACLE_HOME/log/db/client
2009-02-05 15:30:26.331: [ CSSCLNT][2556107008]clsssInitNative: failed to connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db_)), rc 9
Contents of cslcfg log is as follows:
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:3: Problem reading buffer 53b000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:3: Problem reading buffer 53e000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.403: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:3: Problem reading buffer 53e000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.403: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]ibctx:1:ERROR: INVALID FORMAT
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]proprinit:problem reading the bootblock or superbloc 22
2009-02-05 15:20:14.404: [ default][2557164416]a_init:7!: Backend init unsuccessful : [22]
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:3: Problem reading buffer 531000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:3: Problem reading buffer 531000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:3: Problem reading buffer 531000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.404: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]ibctx:1:ERROR: INVALID FORMAT
2009-02-05 15:20:14.404: [ OCRRAW][2557164416]proprinit:problem reading the bootblock or superbloc 22
2009-02-05 15:20:14.404: [ default][2557164416]a_init:7!: Backend init unsuccessful : [22]
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 534000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 533000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 533000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]ibctx:1:ERROR: INVALID FORMAT
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]proprinit:problem reading the bootblock or superbloc 22
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 534000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:3: Problem reading buffer 533000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-02-05 15:20:14.405: [ OCROSD][2557164416]utread:4: Problem reading the buffer errno 2 errstring No such file or directory
2009-02-05 15:20:14.405: [ OCRRAW][2557164416]propriogid:1: INVALID FORMAT
2009-02-05 15:20:14.485: [ OCRRAW][2557164416]propriowv: Vote information on disk 0 [u01/app/oracle/product/11.1.0/db_1/cdata/localhost/local.ocr] is adjusted from [0/0] to [2/2]
2009-02-05 15:20:14.815: [ OCRRAW][2557164416]iniconfig:No 92 configuration
2009-02-05 15:20:14.815: [ OCRAPI][2557164416]a_init:6a: Backend init successful
There are no logs in the $ORACLE_HOME/bin/log/db/cssd directory.
Here is our disk information:
brw-rw---- 1 root disk 8, 0 Feb 5 09:56 sda
brw-rw---- 1 root disk 8, 1 Feb 5 09:56 sda1
brw-rw---- 1 root disk 8, 2 Feb 5 09:56 sda2
brw-rw---- 1 root disk 8, 16 Feb 5 09:56 sdb
brw-rw---- 1 root disk 8, 17 Feb 5 14:57 sdb1
brw-rw---- 1 root disk 8, 18 Feb 5 14:57 sdb2
brw-rw---- 1 root disk 8, 32 Feb 5 09:56 sdc
brw-r----- 1 oracle oinstall 8, 33 Feb 5 14:57 sdc1
brw-rw---- 1 root disk 8, 48 Feb 5 09:56 sdd
brw-r----- 1 oracle oinstall 8, 49 Feb 5 14:57 sdd1
brw-rw---- 1 root disk 8, 64 Feb 5 09:56 sde
brw-r----- 1 oracle oinstall 8, 65 Feb 5 14:57 sde1
brw-rw---- 1 root disk 8, 80 Feb 5 09:56 sdf
brw-r----- 1 oracle oinstall 8, 81 Feb 5 14:57 sdf1
Raw devices:
crw------- 1 oracle oinstall 162, 1 Feb 5 14:57 raw1
crw------- 1 oracle oinstall 162, 2 Feb 5 14:57 raw2
crw------- 1 oracle oinstall 162, 3 Feb 5 14:57 raw3
crw------- 1 oracle oinstall 162, 4 Feb 5 14:57 raw4
Tried
[root@db init.d]# ./init.cssd start
Startup will be queued to init within 30 seconds.
[root@db init.d]# ps -ef | grep css
root 19962 17418 0 19:50 pts/2 00:00:00 grep css
Okay, so what is wrong here? Why won;t CSS start? Checked every piece of literature and search engine results and tried almost everything...
So, dear forum people... Help.
Regards...I got same issue with Oracle 10.2.3 on Solaris. This is a single node setting.
Tried the localconfig reset with root, same issue.
Need help.
cssd/cssdOUT.log
setpriority: unable to escalate to priority -20 (13)
client/css.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2009-03-22 11:44:46.916: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
alertos631std.log
2009-03-22 11:44:31.921
[client(10951)]CRS-1006:The OCR location /opt/oracle/current/cdata/localhost/local.ocr is inaccessible. Details in /opt/oracle/current/log/os631std/client/clscfg.log.
2009-03-22 11:44:31.924
[client(10951)]CRS-1006:The OCR location /opt/oracle/current/cdata/localhost/local.ocr is inaccessible. Details in /opt/oracle/current/log/os631std/client/clscfg.log.
2009-03-22 11:44:32.486
[client(10951)]CRS-1001:The OCR was formatted using version 2.
clscfg.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2009-03-22 11:44:31.919: [ OCROSD][1]utread:3: problem reading buffer 2140000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.919: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.919: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.919: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.919: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2009-03-22 11:44:31.920: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.920: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.921: [ OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2009-03-22 11:44:31.921: [ OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
2009-03-22 11:44:31.921: [ default][1]a_init:7!: Backend init unsuccessful : [22]
2009-03-22 11:44:31.923: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.923: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.923: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.923: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2009-03-22 11:44:31.924: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.924: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2009-03-22 11:44:31.924: [ OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
2009-03-22 11:44:31.924: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 512 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.924: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCROSD][1]utread:3: problem reading buffer 2146000 buflen 4096 retval 0 phy_offset 102400 retry 0
2009-03-22 11:44:31.924: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2009-03-22 11:44:31.924: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2009-03-22 11:44:32.018: [ OCRRAW][1]propriowv: Vote information on disk 0 [opt/oracle/current/cdata/localhost/local.ocr] is a
djusted from [0/0] to [2/2]
2009-03-22 11:44:32.486: [ OCRRAW][1]propriniconfig:No 92 configuration
2009-03-22 11:44:32.486: [ OCRAPI][1]a_init:6a: Backend init successful
Edited by: user462878 on Mar 22, 2009 9:03 AM
Edited by: user462878 on Mar 22, 2009 9:03 AM
Maybe you are looking for
-
Format for certain columns in list view control
I have a list view control and I would like to format the alignment for certain columns. I heard that the only way you can do this is by formatting the cells in excel. I tried this with no success. Can someone please give me step by step instructions
-
How can I add axis to an XY-graph programmatically (in LV 6i)?
I am currently using LabVIEW 6i to make a flexible graph window, which can be called and controlled from another VI. It's easy to send commands to the graph window over a queue interface (enabling things like adding and removing plots, changing the n
-
HP Scanner problems since Mavericks Upgrade.
Has anyone else had a similar problem? I have a HP ScanJet G4050. Since upgrading to Mavericks on my i-Mac, I can scan documents and they show up in the scanning software, but if I try to print or save the document I get a striped page, no usable i
-
How to get list of modified repository objects in SPAU
Hi, While applying Support pack for HR (SAP ERP 6.0) I get SPAU prompt with message: "The system detected that 10 of the repository objects in the Support Packages have been modified in your system . Check whether you want to retain or restore these
-
"Test Movie" Window Size Too Small
I have a very small Flash document (80x10 pixels). When I select "Test Movie" the window that pops-up to test my application is so small that I have to constantly resize the window to see the content. This didn't happen in Flash MX prior and if I cha