RAC node is not starting

I have 2 node RAC instance in 2 virtual machines. My node 2 ASM instance is not starting after the reboot but node 1 is working fine.
I got below output from Node 2.
srvctl status asm
PRCR-1070 : Failed to check if resource ora.asm is registered
Cannot communicate with crsd
alertlog
[/u01/app/11.2.0/grid/bin/cssdagent(3782)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/11.2.0/grid/log/ol5-112-rac2/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2015-03-13 16:38:23.671
[ohasd(3315)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/11.2.0/grid/log/ol5-112-rac2/ohasd/ohasd.log.
2015-03-13 16:38:24.264
[ohasd(3315)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:35.915
[cssd(4326)]CRS-1713:CSSD daemon is started in clustered mode
2015-03-13 16:38:41.195
[cssd(4326)]CRS-1707:Lease acquisition for node ol5-112-rac2 number 2 completed
2015-03-13 16:38:41.259
[cssd(4326)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DISK1; details in /u01/app/11.2.0/grid/log/ol5-112-rac2/cssd/ocssd.log.
2015-03-13 16:38:41.562
[crsd(4285)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/ol5-112-rac2/crsd/crsd.log.
2015-03-13 16:38:42.217
[ohasd(3315)]CRS-2765:Resource 'ora.crsd' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:43.313
[crsd(4357)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/ol5-112-rac2/crsd/crsd.log.
2015-03-13 16:38:44.262
[ohasd(3315)]CRS-2765:Resource 'ora.crsd' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:44.716
[ohasd(3315)]CRS-2765:Resource 'ora.diskmon' has failed on server 'ol5-112-rac2'.
crsd log
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmo: kgfoCheckMount returned [7]
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmo: The ASM instance is down
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprioo: No OCR/OLR devices are usable
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmcl: asmhandle is NULL
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprinit: Could not open raw device
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmcl: asmhandle is NULL
2015-03-13 16:39:02.140: [ OCRAPI][1266976496]a_init:16!: Backend init unsuccessful : [26]
2015-03-13 16:39:02.140: [ CRSOCR][1266976496] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2015-03-13 16:39:02.140: [ CRSD][1266976496][PANIC] CRSD exiting: Could not init OCR, code: 26
2015-03-13 16:39:02.140: [ CRSD][1266976496] Done
Pls help

Hi Levi,
I got below output from cssd.log
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos op : sgipcnTcpConnect
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos dep : No route to host (113)
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos loc : connect
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos info: addr '192.168.1.101:42729'
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssscSelect: conn complete ctx 0x24d7aa0 endp 0x10d4
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmeventhndlr: node(1), endp(0x10d4) failed, probe((nil)) ninf->endp (0x1000010d4) CONNCOMPLETE
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmDiscHelper: ol5-112-rac1, node(1) connection failed, endp (0x10d4), probe(0x100000000), ninf->endp 0x10d4
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmDiscHelper: node 1 clean up, endp (0x10d4), init state 0, cur state 0
2015-03-15 10:38:48.248: [GIPCXCPT][1214789952]gipcInternalDissociate: obj 0x27c8050 [00000000000010d4] { gipcEndpoint : localAddr 'gipc://ol5-112-rac2:de93-de83-5e0c-a373#192.168.1.102#50439', remoteAddr 'gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x8061a, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2015-03-15 10:38:48.248: [GIPCXCPT][1214789952]gipcDissociateF [clssnmDiscHelper : clssnm.c : 3215]: EXCEPTION[ ret gipcretFail (1) ] failed to dissociate obj 0x27c8050 [00000000000010d4] { gipcEndpoint : localAddr 'gipc://ol5-112-rac2:de93-de83-5e0c-a373#192.168.1.102#50439', remoteAddr 'gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x8061a, usrFlags 0x0 }, flags 0x0
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmDiscEndp: gipcDestroy 0x10d4
2015-03-15 10:38:48.414: [    CSSD][1147648320]clssnmvDHBValidateNCopy: node 1, ol5-112-rac1, has a disk HB, but no network HB, DHB has rcfg 320087493, wrtcnt, 207342, LATS 355744, lastSeqNo 207342, uniqueness 1426244111, timestamp 1426396128/31164744
2015-03-15 10:38:48.414: [    CSSD][1214789952]clssnmconnect: connecting to addr gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729
2015-03-15 10:38:48.414: [    CSSD][1214789952]clssscConnect: endp 0x10e0 - cookie 0x24d7aa0 - addr gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729
2015-03-15 10:38:48.414: [    CSSD][1214789952]clssnmconnect: connecting to node(1), endp(0x10e0), flags 0x10002
2015-03-15 10:38:48.710: [    CSSD][1181219136]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmRcfgMgrThread: Local Join
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: set curtime (356544) for my node
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: scanning 32 nodes
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: Node ol5-112-rac1, number 1, is in an existing cluster with disk state 3
2015-03-15 10:38:49.210: [    CSSD][1206397248]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
Thanks,
Suranga

Similar Messages

Listener on RAC node 2 not accepting user connections

Hey guys i am experiencing a disturbing scenario, i have a 2 RAC node cluster . The problem is that the second listener is not registering any connections . I have verified the services of listener using lsnrctl status (the default name is LISTENER), i also have verified the local and remote listener parameters they are fine but running the fol query shows count =0 against inst_id=2;
SQL > select count() from gv$session where username='XYZ' and inst_id=2;*
Count
0
Any help is appreciated in this regard please.

Yeah i know very well the purpose of SCAN . But web admins are sticking to the vip . Well here is the output of lsnrctl status:-
LSNRCTL for Solaris: Version 11.2.0.1.0 - Production on 11-APR-2013 16:41:55
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Solaris: Version 11.2.0.1.0 - Production
Start Date 10-MAR-2013 13:00:14
Uptime 17 days 2 hr. 42 min. 18 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/fedb6/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=5.2.21.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=5.2.21.7)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "grid1.xyz" has 1 instance(s).
Instance "grid12", status READY, has 1 handler(s) for this service...
Service "grid1XDB.xyz" has 1 instance(s).
Instance "grid12", status READY, has 1 handler(s) for this service...
Service "xyz" has 1 instance(s).
Instance "grid12", status READY, has 1 handler(s) for this service...
The command completed successfully
and here is tnsnames.ora output :-
GRID1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = grid1-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = grid1.xyz)
PRIMARY1_SERV =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 5.2.21.1)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = grid1.xyz)
(INSTANCE_NAME= grid11)
PRIMARY2_SERV =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 5.2.21.6)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = grid1.xyz)
(INSTANCE_NAME = grid12)
)

Node Manager not starting UCM Server automatically

Hi All,
I have configured Node Manager to start and stop UCM 11g server.
But whenever, I am restarting the host OS - Windows server 2003 64 bit, it does not start the UCM managed server, automatically.
I am running two managed servers on the same host OS, the first UCM manages server, starts automatically on every restart.
The second managed UCM server, does not start. I need to then manually start the managed server. There are no errors logged, and the manual start of the managed server happens without any issues.
As a workaround, I tried starting the managed server from Windows scheduler, but it does not start the server as well. I wrote a script like -
Myscript.cmd
call E:\..\user_projects\domains\prod_domain\bin\setDomainEnv.cmd
java weblogic.WLST c:\start....\ServerStart.py
scheduled the script (myscript.cmd) to run from the Windows Scheduler.
Any pointers would be very helpful. Are there any known issues, if there are two managed servers, and one starts automatically, which affects the starting of the second managed server?
I see the following error in the log file -
<14/10/2011 6:35:22 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<14/10/2011 6:35:33 PM EST> <Info> <Management> <BEA-141223> <The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.>
<14/10/2011 6:35:33 PM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.
at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
at weblogic.Server.main(Server.java:71)
Regards
Avi

Hi Avi,
Give a cross check to your ServerStart.py
As the error states The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist
In your ServerStart.py you might have specified as follows
sh start MAnagedWeblogic.sh server=PRD_UCM_Server1
While these are the list of Server you have in your weblogic
PRDAdminServer,
PRD_UCM_server1,(You have Specified in your command PRD_UCM_Server1
PRD_IBR_server1
The Command should be as follows
sh start ManagedWeblogic.sh server=PRD_UCM_server1
This should help
Regards
Fabian

11gR2 Cluster node does not start, start of ora.gipcd fails

Hi
I have a clusert node which just does not start, and i do know understand what the problem is.
Does anyone have an idea what the problem cound be?
I am using Oracle 11.2.0.1.0 on OEL.
I appreciate any help, thank you.
Regards Bernhard
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE OFFLINE
ora.crsd
      1        ONLINE INTERMEDIATE node001
ora.cssd
      1        ONLINE OFFLINE
ora.cssdmonitor
      1        ONLINE OFFLINE
ora.ctssd
      1        ONLINE OFFLINE
ora.diskmon
      1        ONLINE OFFLINE
ora.drivers.acfs
      1        ONLINE ONLINE       node001
ora.evmd
      1        ONLINE OFFLINE
ora.gipcd
      1        ONLINE OFFLINE
ora.gpnpd
      1        ONLINE OFFLINE
ora.mdnsd
      1        ONLINE ONLINE       node001                              alertnode001.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:43:52.511
[/opt/grid/product/11gR2/grid/bin/cssdmonitor(5144)]CRS-5822:Agent '/opt/grid/product/11gR2/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-06-29 08:48:49.713
[ohasd(4346)]CRS-2112:The OLR service started on node node001.
2012-06-29 08:48:50.010
[ohasd(4346)]CRS-8017:location: /etc/oracle/lastgasp has 50 reboot advisory log files, 0 were announced and 0 errors occurred
2012-06-29 08:48:55.134
[ohasd(4346)]CRS-2772:Server 'node001' has been assigned to pool 'Free'.
[client(4634)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4652)]CRS-10001:ACFS-9322: done.
2012-06-29 08:50:59.362
[ohasd(4346)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.oracssdmonitor_root.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:48:51.003: [    AGFW][1810312288] Starting the agent: /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/
2012-06-29 08:48:51.003: [   AGENT][1810312288] Agent framework initialized, Process Id = 4414
2012-06-29 08:48:51.003: [ USRTHRD][1810312288] to enter agent main
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: setting priority to 4
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] *** Agent Framework Started ***
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_vmondisv: Compatible vendor clusterware not in use
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_thrdspawn: spawn OMON succ
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: spawn omon succ
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: default values for omon, initrate 1000, pollrate 500.
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: starting omon
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:55.041: [    AGFW][1810312288] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Inited lsf context 0x1ae1cdd0
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Initing CLS Framework messaging
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] New Framework state: 2
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] M2M is starting...
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] m_pClscCtx=0x1ae21d70m_pUgblm=0x1ae754b0
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] Starting send thread
2012-06-29 08:48:55.052: [ CRSCOMM][1119893824] clsIpc: sendWork thread started.
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] IPC Client thread started listening
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] init data sent from server
2012-06-29 08:48:55.057: [CLSFRAME][1810312288] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:node001
2012-06-29 08:48:55.058: [CLSFRAME][1810312288] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:node001
2012-06-29 08:48:55.066: [CLSFRAME][1810312288] Starting thread model named: MultiThread
2012-06-29 08:48:55.073: [CLSFRAME][1810312288] Starting thread model named: SingleThread
2012-06-29 08:48:55.076: [CLSFRAME][1810312288] Starting thread model named: SingleThreadT
2012-06-29 08:48:55.080: [CLSFRAME][1810312288] New Framework state: 3
2012-06-29 08:48:55.080: [    AGFW][1810312288] Agent Framework started successfully
2012-06-29 08:48:55.080: [    AGFW][1182832960] Agfw engine module has enabled...
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] Module Enabling is complete
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] New Framework state: 6
2012-06-29 08:48:55.080: [    AGFW][1182832960] Agent is started with userid: root , expected user: root
2012-06-29 08:48:55.081: [    AGFW][1182832960] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-06-29 08:48:55.088: [    AGFW][1182832960] Agent received the message: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.089: [    AGFW][1182832960] Added new restype: ora.cssdmonitor.type
2012-06-29 08:48:55.089: [    AGFW][1182832960] Agent sending last reply for: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.090: [    AGFW][1182832960] Agent received the message: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.090: [    AGFW][1182832960] Added new resource: ora.cssdmonitor 1 1 to the agfw
2012-06-29 08:48:55.091: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.091: [    AGFW][1182832960] Agent received the message: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.091: [    AGFW][1182832960] Preparing CHECK command for: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [    AGFW][1172343104] Executing command: check for resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_INITRATE, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_POLLRATE, value 500
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr ORA_OPROCD_MODE, value
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr PROCD_TIMEOUT, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr LOGGING_LEVEL, value 1
2012-06-29 08:48:55.096: [    AGFW][1172343104] check for resource: ora.cssdmonitor 1 1 completed with status: OFFLINE
2012-06-29 08:48:55.096: [CRSTIMER][1203812672] Timer Thread Starting.
2012-06-29 08:48:55.096: [    AGFW][1182832960] ora.cssdmonitor 1 1 state changed from: UNKNOWN to: OFFLINE
2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:28
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent received the message: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:34
2012-06-29 08:48:55.101: [    AGFW][1182832960] ora.cssdmonitor 1 1 marked as deleted.
2012-06-29 08:48:55.101: [    AGFW][1182832960] Deleting the resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.101: [    AGFW][1182832960] Agent is commiting suicide.
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: CSSD Agent was asked to exit with exit code 1
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_becomeactive: Signaling active state
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_term: still threads up (1)
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: Agent is done.
2012-06-29 08:48:55.101: [    AGFW][1182832960] Agent is exiting with exit code: 1
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29Edited by: Bernhard W on 03-Jul-2012 02:14

I am really getting desperate about this.
The ASM persmissions are on both nodes the same, so this should not be the problem and i can not find any problem on the network.
The log files of the healthy node do not say anything about the node which does not startup now, the last log entry was from 6 months ago, when i shut that node down. There seems to be no communication between the two nodes now.
I tried to manually start the ora.cssd but somehow ora.gipcd just does not start and i can not find the problem.
[oracle@dmrs11srvcla001 ~]$ crsctl stat res -t -init
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE OFFLINE
ora.crsd
      1        ONLINE INTERMEDIATE dmrs11srvcla001
ora.cssd
      1        ONLINE OFFLINE
ora.cssdmonitor
      1        ONLINE ONLINE       dmrs11srvcla001
ora.ctssd
      1        ONLINE OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE ONLINE       dmrs11srvcla001
ora.evmd
      1        ONLINE OFFLINE
ora.gipcd
      1        ONLINE OFFLINE
ora.gpnpd
      1        ONLINE OFFLINE
ora.mdnsd
      1        ONLINE ONLINE       dmrs11srvcla001
[oracle@dmrs11srvcla001 ~]$ crsctl start res ora.cssd -init
CRS-2672: Attempting to start 'ora.gipcd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gipcd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gipcd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gipcd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gpnpd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gpnpd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dmrs11srvcla001'
CRS-2672: Attempting to start 'ora.diskmon' on 'dmrs11srvcla001'
CRS-2676: Start of 'ora.diskmon' on 'dmrs11srvcla001' succeededalertnode001.log
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:40.789
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:41.654
[ohasd(4351)]CRS-2765:Resource 'ora.mdnsd' has failed on server 'node001'.
2012-07-02 17:11:43.404
[cssd(4997)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:11:46.634
[cssd(4997)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:11:46.745
[ohasd(4351)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'node001'.
2012-07-02 17:12:42.384
[/opt/grid/product/11gR2/grid/bin/orarootagent.bin(4488)]CRS-5818:Aborted command 'start for resource: ora.diskmon 1 1' for resource 'ora.diskmon'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-07-02 17:12:46.390
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.diskmon'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:14:50.768
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:55.819
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:58.354
[cssd(5137)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:16:58.425
[cssd(5137)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:25:23.194
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-02 17:25:23.200
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-02 17:26:57.384
[/opt/grid/product/11gR2/grid/bin/cssdagent(5116)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-02 17:27:02.514
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:37:11.971
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.evmd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 02:13:57.821
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:14:37.915
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:44:44.630
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:14:51.303
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:44:57.981
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:15:04.655
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:42:45.015
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:50.068
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:52.369
[cssd(10209)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-03 07:44:52.434
[cssd(10209)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-03 07:45:11.331
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-03 07:54:51.401
[/opt/grid/product/11gR2/grid/bin/cssdagent(10188)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-03 07:54:56.524
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.ohasd.log
2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [    AGFW][1347250496] Received state change for ora.crsd 1 1 [old state = PARTIAL, new state = OFFLINE]
2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23258]
2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [   CRSPE][1357756736] State change received from node001 for ora.crsd 1 1
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Processing PE command id=139. Description: [Resource State Change (ora.crsd 1 1) : 0x1c284d40]
2012-07-03 07:45:11.331: [   CRSPE][1357756736] RI [ora.crsd 1 1] new external state [OFFLINE] old value: [INTERMEDIATE] on node001 label = []
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Resource Resource Instance ID[ora.crsd 1 1]. Values:
STATE=OFFLINE
TARGET=ONLINE
LAST_SERVER=node001
CURRENT_RCOUNT=1
LAST_RESTART=1341292505
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=29
STATE_CHANGE_VERS=29
LAST_FAULT=1341292504
DEGREE_ID=1
ID=ora.crsd 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:none
ReadLocks:|STATE INITED||INITIAL CHECK DONE| has failed!
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Processing unplanned state change for [ora.crsd 1 1]
2012-07-03 07:45:11.331: [   CRSPE][1357756736] ora.crsd 1 1: uptime exceeds uptime threshold , resetting restart count
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Local Recovery picked for [ora.crsd 1 1]
2012-07-03 07:45:11.333: [   CRSPE][1357756736] Sending message to agfw: id = 23260
2012-07-03 07:45:11.333: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:11.333: [   CRSPE][1357756736] CRS-2672: Attempting to start 'ora.crsd' on 'node001'
2012-07-03 07:45:11.333: [    AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260 to the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.349: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.350: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.351: [   CRSPE][1357756736] RI [ora.crsd 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on node001 label = []
2012-07-03 07:45:12.351: [   CRSPE][1357756736] CRS-2676: Start of 'ora.crsd' on 'node001' succeeded
2012-07-03 07:45:12.351: [   CRSPE][1357756736] PE Command [ Resource State Change (ora.crsd 1 1) : 0x1c284d40 ] has completed
2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw received reply from PE for resource state change for ora.crsd 1 1
2012-07-03 07:53:17.220: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [    AGFW][1347250496] Received state change for ora.diskmon 1 1 [old state = ONLINE, new state = PLANNED_OFFLINE]
2012-07-03 07:53:17.221: [    AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23497]
2012-07-03 07:53:17.221: [    AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [   CRSPE][1357756736] State change received from node001 for ora.diskmon 1 1
2012-07-03 07:53:17.221: [   CRSPE][1357756736] Processing PE command id=140. Description: [Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0]
2012-07-03 07:53:17.221: [   CRSPE][1357756736] RI [ora.diskmon 1 1] new external state [OFFLINE] old value: [ONLINE] on node001 label = []
2012-07-03 07:53:17.221: [   CRSPE][1357756736] RI [ora.diskmon 1 1] new target state: [OFFLINE] old value: [ONLINE]
2012-07-03 07:53:17.222: [ CRSOCR][1351452992] Multi Write Batch processing...
2012-07-03 07:53:17.222: [   CRSPE][1357756736] Resource Resource Instance ID[ora.diskmon 1 1]. Values:
STATE=OFFLINE
TARGET=OFFLINE
LAST_SERVER=node001
CURRENT_RCOUNT=0
LAST_RESTART=1341294297
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=2
STATE_CHANGE_VERS=5
LAST_FAULT=1341242723
DEGREE_ID=1
ID=ora.diskmon 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:START of [ora.cssd 1 1] on [node001] : 0x2aaab000ba10
ReadLocks:|STATE INITED| has failed!
2012-07-03 07:53:17.222: [   CRSPE][1357756736] Processing unplanned state change for [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [   CRSPE][1357756736] Target is not ONLINE, not recovering [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [   CRSPE][1357756736] PE Command [ Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0 ] has completed
2012-07-03 07:53:17.222: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.222: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.223: [    AGFW][1347250496] Agfw received reply from PE for resource state change for ora.diskmon 1 1
2012-07-03 07:53:17.223: [ CRSOCR][1351452992] Multi Write Batch done.
2012-07-03 07:54:56.523: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.523: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [   CRSPE][1357756736]Created alert : (:CRSPE00111:) : Start action timed out!
2012-07-03 07:54:56.524: [   CRSPE][1357756736] Start action failed with error code: 3
2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [    AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [   CRSPE][1357756736] CRS-2674: Start of 'ora.cssd' on 'node001' failed
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
     MESSAGE:
     TextMessage[CRS-2674: Start of 'ora.cssd' on 'node001' failed]
     MSGTYPE:
     TextMessage[1]
     OBJID:
     TextMessage[ora.cssd]
     WAIT:
     TextMessage[0]
2012-07-03 07:54:56.525: [   CRSPE][1357756736] Sending message to agfw: id = 23555
2012-07-03 07:54:56.525: [   CRSPE][1357756736] CRS-2679: Attempting to clean 'ora.cssd' on 'node001'
2012-07-03 07:54:56.525: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:54:56.525: [    AGFW][1347250496] Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is being stopped. Will be restarted when agent stops completly
2012-07-03 07:54:56.525: [    AGFW][1347250496] Could not forward message [RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555] to agent. /opt/grid/product/11gR2/grid/bin/cssdagent_root is not running
2012-07-03 07:54:56.525: [    AGFW][1347250496] Can not start Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is waiting to be restarted.
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
     MESSAGE:
     TextMessage[CRS-2679: Attempting to clean 'ora.cssd' on 'node001']
     MSGTYPE:
     TextMessage[3]
     OBJID:
     TextMessage[ora.cssd]
     WAIT:
     TextMessage[0]
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 14
2012-07-03 07:55:01.628: [ CRSCOMM][1340946752] IPC listener connection to member 14 has been removed
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [   CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:01.628: [    AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:01.628: [    AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:01.628: [    AGFW][1347250496] Restarting the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.628: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10188] stopped!
2012-07-03 07:55:01.628: [ CRSCOMM][1347250496] removeConnection: Member 14 does not exist.
2012-07-03 07:55:01.628: [    AGFW][1347250496] Starting the agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id: root and incarnation:7
2012-07-03 07:55:01.632: [    AGFW][1347250496] Starting the HB [Interval = 30000, misscount = 10kill allowed=0] for agent: /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New IPC Member:{Relative|Node:0|Process:15|Type:3}:AGENT
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New process connected to us ID:{Relative|Node:0|Process:15|Type:3} Info:AGENT
2012-07-03 07:55:01.837: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.837: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root with pid:10530 connected to server.
2012-07-03 07:55:01.837: [    AGFW][1347250496] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [    AGFW][1347250496] Agfw Proxy Server sending message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [    AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.839: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.851: [    AGFW][1347250496] Received the reply to the message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.854: [    AGFW][1347250496] Received the reply to the message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.861: [    AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.862: [    AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [   CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.862: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.863: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [   CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.863: [    AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.863: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [   CRSPE][1357756736] CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded
2012-07-03 07:55:01.863: [   CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
2012-07-03 07:55:01.863: [UiServer][1361959232] Container [ Name: ORDER
     MESSAGE:
     TextMessage[CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded]
     MSGTYPE:
     TextMessage[3]
     OBJID:
     TextMessage[ora.cssd]
     WAIT:
     TextMessage[0]
2012-07-03 07:55:01.864: [   CRSPE][1357756736] PE Command [ Start Resource : 0x2aaab0009d00 ] has completed
2012-07-03 07:55:01.864: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [   CRSPE][1357756736] UI Command [Start Resource : 0x2aaab0009d00] is replying to sender.
2012-07-03 07:55:01.864: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [UiServer][1361959232] Container [ Name: UI_DATA
     ora.cssd:
     TextMessage[215]
2012-07-03 07:55:01.864: [UiServer][1361959232] Done for ctx=0x2aaab004ece0
2012-07-03 07:55:01.869: [UiServer][1364060480] Closed: remote end failed/disc.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 15
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] IPC listener connection to member 15 has been removed
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.792: [   CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:03.792: [    AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:03.792: [    AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:03.792: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10530] stopped!
2012-07-03 07:55:03.792: [ CRSCOMM][1347250496] removeConnection: Member 15 does not exist.

Node manager not starting servers as a OS service.

Node manager is starting upon simulating OS crash but it is not bringing up the managed servers running on that machine.
The following is the nodemanager.properties file -
[orawl@qa-wl2 nodemanager]$ more nodemanager.properties
#Mon Apr 11 14:55:43 EDT 2011
#Mon Apr 11 11:57:43 EDT 2011
DomainsFile=/home/orawl/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.domains
LogLimit=0
PropertiesVersion=10.3
DomainsDirRemoteSharingEnabled=false
javaHome=/usr/lib/jdk1.6.0_24
AuthenticationEnabled=true
NodeManagerHome=/home/orawl/oracle/middleware/wlserver_10.3/common/nodemanager
JavaHome=/usr/lib/jdk1.6.0_24/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=true
LogCount=1
DomainRegistrationEnabled=false
StopScriptEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=true
StartScriptEnabled=true
LogFile=/home/orawl/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50
#NodeManager SSL Properties
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreType=JKS
CustomIdentityAlias=xxxxxxxxxxxx
CustomIdentityKeyStoreFileName=/home/orawl/oracle/middleware/keystores/xxxxxxxxxxxx.jks
CustomIdentityKeyStorePassPhrase={3DES}XXXXXXXXXXXXXXXXXXXXXXX
CustomIdentityPrivateKeyPassPhrase={3DES}XXXXXXXXXXXXXXXXXXXXXXX
CustomTrustKeyStoreType=JKS
CustomTrustKeyStoreFileName=/home/orawl/oracle/middleware/keystores/trustcacerts.jks
CustomTrustKeyPassPhrase=xxxxxxxxxxx
Please advice.

Hi All,
Add to my initial post , the error which is getting logged is -
<14/10/2011 6:35:22 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<14/10/2011 6:35:33 PM EST> <Info> <Management> <BEA-141223> <The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.>
<14/10/2011 6:35:33 PM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.
     at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
     at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
     at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
     at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
     at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
     at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
     at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
     at weblogic.Server.main(Server.java:71)
Regards,
Avi

Node Manager not starting servicess after server reboot

We have WLS 10.1.3.5 with IDM/OAM(two domains in same server as its TEST environment) on Oracle Linux 5(64bit),when ever server reboots node manger is not starting managed servers.
I have enabled CrashRecoveryEnabled=true
Here is nodemanager.properties.
[oracle@oam nodemanager]$ cat nodemanager.properties
#Changed NM Listen Port
#Tue Mar 05 03:12:20 IST 2013
DomainsFile=/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager/nodemanager.domains
LogLimit=0
DomainsDirRemoteSharingEnabled=false
PropertiesVersion=10.3
AuthenticationEnabled=true
NodeManagerHome=/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager
javaHome=/u01/Oracle/Middleware/jdk160_24
JavaHome=/u01/Oracle/Middleware/jdk160_24/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=false
LogCount=1
StopScriptEnabled=false
DomainRegistrationEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=true
StartScriptEnabled=true
LogFile=/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50
here is my domain information in server.
=========================
[oracle@oam nodemanager]$ cat nodemanager.domains
#Domains and directories created by Configuration Wizard
#Tue Mar 05 05:24:37 IST 2013
OAMDomain=/u01/Oracle/Middleware/user_projects/domains/OAMDomain
IDMDomain=/u01/Oracle/Middleware/user_projects/domains/IDMDomain

Hi,
Entries in nodemanager.properties looks fine, but how are you testing this functionality?
If you will stop servers monitored by nodemanager, stop nodemanager and then reboot physical server, on startup if nodemanager is configured as daemon it will start nm process but will do nothing to managed server process.
To test "CrashRecoveryEnabled=true" kill the managed server pid and then stop/kill nodemanager process. Then perform physical server reboot, now if nm is configured as daemon process server reboot will start nodemanager. Once NM is up first thing it check is the server it was monitoring earlier and if under directory $WL_HOME/servers/<server>/data/nodemanager it finds "pid" and "lck" file it checks if that pid exists and if not then go for a restart of weblogic server instance.
This has worked for me...!
Thanks,
Ranjan
http://www.middlewaresupport.wordpress.com

Node Manager not starting UCM Server

Hi All,
I have configured Node Manager to start and stop UCM 11g server.
But whenever, I am restarting the host OS - Windows server 2003 64 bit, it does not start the UCM managed server, automatically.
I am running two managed servers on the same host OS, the first UCM manages server, starts automatically on every restart.
The second managed UCM server, does not start. I need to then manually start the managed server. There are no errors logged, and the manual start of the managed server happens without any issues.
As a workaround, I tried starting the managed server from Windows scheduler, but it does not start the server as well. I wrote a script like -
Myscript.cmd
call E:\..\user_projects\domains\prod_domain\bin\setDomainEnv.cmd
java weblogic.WLST c:\start....\ServerStart.py
scheduled the script (myscript.cmd) to run from the Windows Scheduler.
Any pointers would be very helpful. Are there any known issues, if there are two managed servers, and one starts automatically, which affects the starting of the second managed server?
Regards
Avi

Hi All,
Add to my initial post , the error which is getting logged is -
<14/10/2011 6:35:22 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<14/10/2011 6:35:33 PM EST> <Info> <Management> <BEA-141223> <The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.>
<14/10/2011 6:35:33 PM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.
     at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
     at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
     at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
     at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
     at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
     at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
     at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
     at weblogic.Server.main(Server.java:71)
Regards,
Avi

RHEL 3 - RAC - Firewire - Listener Not Starting

Hi,
I almost got thru the howto install guide on the Firewire RAC server install, but hit a snag. During the TNSListener configuration, I see both server - run the config just like the instructions say, but it barks back that permissions are denied and cannot start the listener which maybe already started. I then try manually starting the listener using the lsnrctl start command - but the bloody thing still won't start.
Any ideas - I'm about to scrap the install and start over since I was planning on doing that anyway......

I also try a startup of the nodeapps using srvctl - here's the results I get.
[oracle@cberac2 admin]$ srvctl start nodeapps -n cberac2
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:LSNRCTL for Linux: Version 10.1.0.3.0 - Production on 20-SEP-2005 17:33:14
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Copyright (c) 1991, 2004, Oracle. All rights reserved.
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Starting /u01/app/oracle/product/10.1.0/db_1/bin/tnslsnr: please wait...
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:TNSLSNR for Linux: Version 10.1.0.3.0 - Production
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:System parameter file is /u01/app/oracle/product/10.1.0/db_1/network/admin/listener.ora
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Log messages written to /u01/app/oracle/product/10.1.0/db_1/network/log/listener_cberac2.log
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.68.61.192)(PORT=1521)))
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.68.61.190)(PORT=1521)))
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip-cberac2)(PORT=1521)(IP=FIRST)))
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:TNS-12547: TNS:lost contact
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-12560: TNS:protocol adapter error
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-00517: Lost contact
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: Linux Error: 104: Connection reset by peer
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.68.61.190)(PORT=1521)(IP=FIRST)))
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:TNS-12541: TNS:no listener
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-12560: TNS:protocol adapter error
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-00511: No listener
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: Linux Error: 111: Connection refused
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:LSNRCTL for Linux: Version 10.1.0.3.0 - Production on 20-SEP-2005 17:33:14
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Copyright (c) 1991, 2004, Oracle. All rights reserved.
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip-cberac2)(PORT=1521)(IP=FIRST)))
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:TNS-12541: TNS:no listener
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-12560: TNS:protocol adapter error
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-00511: No listener
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: Linux Error: 111: Connection refused
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.68.61.190)(PORT=1521)(IP=FIRST)))
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr:TNS-12541: TNS:no listener
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-12560: TNS:protocol adapter error
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: TNS-00511: No listener
cberac2:ora.cberac2.LISTENER_CBERAC2.lsnr: Linux Error: 111: Connection refused
CRS-0215: Could not start resource ora.cberac2.LISTENER_CBERAC2.lsnr.

(RAC) node 2 dont start automatically

hi gurus,
we have intalled a RAC (with 2 nodes) on database version 11.2.0.3.
When we reboot node 1, it restart automatically but when we reboot node 2 we have to restart manually using:
srvctl start instance -d DB_name -i NODE_2
how to check if we have autostart configured in node 2?
i did
[oracle@sifnode2 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
if HAS is online it means that is autostart on?

Hi,
You sould know there is important change in 11.2 CRS...
With Oracle 11.2 database auto start policy in the clusterware is restore, which means that clusterware will remember the last state of the database. If the database was stopped normally then on the next restart of clusterware it won’t be started. Otherwise if the server crashes or by some reason the OS is rebooted then clusterware will start the database because last state was ONLINE (running).
If you are running on Linux there is no need to write your own scripts for automatically startup/shutdown anymore. But you need to configure something.
Here is what it is:
By default with Oracle 11.2 several important resources come in the profile with attribute AUTO_START=restore. Such resources are Oracle Database resource, Oracle ASM resource, CRS resource type for the Listener. This means that if Oracle database server is restarted for some reason , it will keep and restore the last state.
It is a good practice to change this default behaviour and it is the first thing I usually do after a new installation. I change AUTO_START=always for the resources listed above.
Coming back to your issue....
Please check following to know the auto_start policy
crsctl config crs
crs_stat -p <resource name>
crsctl status resource <resource name> -p | grep AUTO_START

Node Manager Not Starting Managed Servers

I have nodemanager configured to restart managed servers automatically upon reboot but when I physically restart the 2nd node none of the managed servers start and there are no errors. Steps completed and verified....
1) Nodemanager.properties has domain path and 'crashrecoveryenabled' set to 'true'
2) managed servers have autorestart set to 'true' and -Xnohup parameter added to startup arguments.
3) Boot.properties files ave the username and password (encrypted) to start up the server.
Here is the nodemanager.log file and you can clearly see it is grabbing the startup files. It almost looks like the server files are trying to get loaded before the nodemanager port is available but I see no errors. Any ideas?
<Dec 28, 2009 9:16:59 AM> <Info> <QA10311> <WebProxyServer2> <Working directory is "C:\bea\user_projects\domains\QA10311">
<Dec 28, 2009 9:16:59 AM> <Info> <QA10311> <WebProxyServer2> <Server output log file is "C:\bea\user_projects\domains\QA10311\servers\WebProxyServer2\logs\WebProxyServer2.out">
<Dec 28, 2009 9:47:09 AM> <INFO> <Loading domains file: C:\bea\WLSERV~1.3\common\nodemanager\nodemanager.domains>
<Dec 28, 2009 9:47:16 AM> <INFO> <Loading identity key store: FileName=C:/bea/WLSERV~1.3/server\lib\DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Dec 28, 2009 9:47:18 AM> <INFO> <Loaded node manager configuration properties from 'C:\bea\WLSERV~1.3\common\nodemanager\nodemanager.properties'>
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <ACR2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\ACR2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <CommandNet2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\CommandNet2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <FPCE2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\FPCE2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <MQServer2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\MQServer2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <PortalServer2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\PortalServer2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <ShoppingCart2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\ShoppingCart2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <webapp2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\webapp2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <WebProxyServer2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\WebProxyServer2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <WUA2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\WUA2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:27 AM> <INFO> <Secure socket listener started on port 5556, host qaportal2.FCQA.COM>

Hi,
Entries in nodemanager.properties looks fine, but how are you testing this functionality?
If you will stop servers monitored by nodemanager, stop nodemanager and then reboot physical server, on startup if nodemanager is configured as daemon it will start nm process but will do nothing to managed server process.
To test "CrashRecoveryEnabled=true" kill the managed server pid and then stop/kill nodemanager process. Then perform physical server reboot, now if nm is configured as daemon process server reboot will start nodemanager. Once NM is up first thing it check is the server it was monitoring earlier and if under directory $WL_HOME/servers/<server>/data/nodemanager it finds "pid" and "lck" file it checks if that pid exists and if not then go for a restart of weblogic server instance.
This has worked for me...!
Thanks,
Ranjan
http://www.middlewaresupport.wordpress.com

Node manager not starting

I have just installed a second CRS 5.0(2) server which I intend to add to the cluster, but I'm having a problem in that the Node Manager Service hangs on startup. The error I get is:
Event ID:7022
The CRS Node Manager Service Hung on starting.
I can see from the Windows services that the service remains in a 'starting state'.
I wanted to attach the MIVR logs to this post but I'm finding that there is no MIVR folder in c:/Program Files/wfavvid/log. Has the install completed successfully? I had no errors when installing the package.
I've checked the binding order of the NIC cards and they are correct. I have also run the 'postinstall.exe' but still the same problem.
On another note, how can you tell if the license provided (uploaded by someone else) includes HA software?
Thanks

Try checking out the MCVD logs and the application and system logs for any error with NIC or network connectivity.
As for the second question the key you are looking for in the Lic is the entry.
INCREMENT CRS_EXP_PRE_WARM_STBY cisco 5.0
In case its already uploaded in CRS the Control Center >Display Licenses will show HA enabled.

RACs nodes can not communicate

I am working with the installation of Oracle 11g RAC on virtual servers.
so far, I create rac1 and rac2 two nodes on virtual server.
after run two virtual mahines, I try to test the network setup, ping the nodes from each other using public address.
I have the following problems;
*rac1:internet connection ;0.k
ip_address:10.45.110.247
subnet:255.255.252.0
gateway:10.45.109.1
*rac2:internet connection:o.k
ip_address:10.45.110.178
subnet:255.255.252.0
gateway:10.45.109.1
if i ping to each other
run ping forever....
if I type,
ssh rac2 cat ~/.ssh/id_rsa/bub >> ~/.ssh/authorized_keys
error;ssh: rac2:Name or service not known.
would you help me!

I have a question with following your instruction;
In my virtual network setting ;
virtual network     summary                  subnet
vmnet0[bridged]     bridged to auto
vmnet1[host-only]    a private network    192.168.65.0
vmnet8[nat]   used to share the host   192.168.183.0
DHCP TAB
vmnet1       192.168.65.0 255.255.255.0 VMNET1
vmnet8    192.168.183.0 255.255.255.0 VMNET8
question;
Choose host only networking -> it means - go to ethernet hardware config. and change" Host-only" in network connection in vmnet8. am I right?
Also ethernet config for the cluster interconnect should not have a broadcast ?????? i do not understand, where do i go to change it?
thank you so much for your help.
by the way, if you have some reference for my problem, please share with me.

Node is not processing

Hallo friends,
after years of using Logic I am trying to set up a Node. Without success
What works: a LAN connection between my two Macs. I used automatic configuration, under TCP/IP Configured IPv4 Using DHCP.
I can connect to the file server from / to each of my Macs. I can mount hard drives, share internet connections...
I installed Logic Node app on my MacBook and I used the Logic Node installer located in utilities folder on my dualG5 (I am trying to use MacBook as a node and Dual G5 as a Master computer).
I started the Node app on MacBook first, than launched Logic app on dualG5. Without opening new project I opened Preferences / Audio / Nodes. Now: I can see my MacBook available (not with grey letters). No box is checked (near "Enable Logic Nodes" or near "MacBook" under Node). So I check "Enable Logic Nodes". I can see *Initializing Core Audio* is working. Than I try to check the box near my MacBook node, but I can' t. The whole line turns into darker grey as soon as I select the checkbox,but nothing happens. when I try to select it another time, the popup window occurs and says:
*Logic Pro lost the connection to the Node "MacBook"*
Please relaunch Node application or check network connection
My network connection seems to work fine, I can log in, mount drives, share internet... The Node app seems to be or it' s icon looks like connected in MacBook dock (and it says "Connected to G5"). What I found out is, that if I try to Quit the Node app on MacBook either when Logic is running on G5 or after I quit Logic on G5, I am never able to quit Node app on my MacBook normally as it stops responding, when I try to quit it. I always have to Force Quit this Node app...
Now the second way I tried: started Node app on MacBook, launched Logic on dualG5. Opened Preferences / Audio / Nodes. Nothing is checked AND NOW I CHECKED the checkbox near the MacBook node FIRST. It can be done (remember nothing happened when I tried to check it after I checked "Enable Logic Nodes" checkbox). Than I checked "Enable Logic Nodes" , seems to be OK. The Node on MacBook says and looks like connected...
I opened a project and created approx. 10 audio tracks. I inserted Linear phase EQs, Logic compressors, Deessers, EQs to them. I also inserted whole lot of Waves SSL bundle to some tracks, but kept Logic native and Waves plugins separate. The system was pretty maxed out. Now the maximum I could get whenassign a track to a Node processing, was that the Track Node button showed green arrow, but did not light green. Whing according to manual is: Sync Pending and not Enabled/Active. I even could see some tracks Enabled/Inactive.
After spending last 24+ hours how to figure it out, I am asking for help.
Here are some answers I expect you might ask:
-yes, I have same versions of Logic and Node
-yes, I have Waves SSL bundle installed on my MacBook, same version as on DualG5
-yes, I tried it with native Logic plugins only
-yes, the Firewall is OFF on both machines (Preferences / Sharing)
- I unchecked all Ports except "Built in Ethernet" on dualG5 and "Airport" and "Built in Ethernet" on MacBook in Preferences / Network Port Configurations
-yes, I tried it with "Airport" unchecked also
-yes, I tried to reboot many times - both machines. I put Node app into login items on MacBook. I tried to shut both machines of, start MacBook first, make sure Node is "on", started dualG5, launched Logic...
-yes, I inserted so many plugins to so many tracks that may dualG5 gave me "System Overload" message everytime I tried to playback the project, but Node did not start to process...
-no, I am not networking genius
-yes, I tried different LAN cable (CAT5 always, one "crossed" or "cross-patched")
-I've got Parallels and WinXP installed on my MacBook, but they never run, when I tried Node setup (actually, I am almost never using them )
Any ideas appreciated, thank you.
Message was edited by: Diamond Dog

I'd start with something simple - one audio track, with one Logic plugin.
Can you get a solid green node light on that? if not, no point trying anything more complicated until you can get this fixed.
Note - I briefly tried it with my MBP and my old Powerbook as a node, and it worked fine without any particular headaches setting it up...

Ocmstart.sh is not starting in one RAC node

I am having a trouble starting "oracm" on one node. I can see it is working on second node but not on first node. Is there any one haveing the same issue.

Hi Chandra Sorry for the delay, I got the output from cm.out file during restart of ocmstart.sh script
=========================================================
TRACE: HandleJoin(): JOIN from node(1)->(1), tid = ClusterListener:3076 file = nmlisten.c, line = 362 {Thu Apr 10 21:39:10 2003 }
TRACE: HandleStatus(): node(0) UNKNOWN, tid = ClusterListener:3076 file = nmlisten.c, line = 404 {Thu Apr 10 21:39:11 2003 }
TRACE: HandleStatus(): src[0] dest[1] dom[0] seq[33] sync[10], tid = ClusterListener:3076 file = nmlisten.c, line = 415 {Thu Apr 10 21:39:11 2003 }
TRACE: HandleSync(): src[0] dest[1] dom[0] seq[34] sync[10], tid = ClusterListener:3076 file = nmlisten.c, line = 506 {Thu Apr 10 21:39:11 2003 }
TRACE: SendAck(): node(0) domain(0) syncSeqNo(10) type(11), tid = ClusterListener:3076 file = nmmember.c, line = 1913 {Thu Apr 10 21:39:11 2003 }
TRACE: HandleVote(): src[0] dest[1] dom[0] seq[35] sync[10], tid = ClusterListener:3076 file = nmlisten.c, line = 643 {Thu Apr 10 21:39:12 2003 }
TRACE: SendVoteInfo(): node(0) domain(0) syncSeqNo(10), tid = ClusterListener:3076 file = nmmember.c, line = 1727 {Thu Apr 10 21:39:12 2003 }
TRACE: HandleShutdown(): src[0] dest[1] dom[0] seq[0] sync[10] type[4], tid = ClusterListener:3076 file = nmlisten.c, line = 1087 {Thu Apr 10 21:39:13 2003 }
TRACE: IncrementEventValue: *(80f2920) = (1, 1), tid = ClusterListener:3076 file = unixinc.c, line = 253 {Thu Apr 10 21:39:13 2003 }--- End Dump ---
oracm, version[ 9.2.0.2.0.47 ] started {Thu Apr 10 21:40:07 2003 }
KernelModuleName is hangcheck-timer {Thu Apr 10 21:40:07 2003 }
OemNodeConfig(): Network Address of node0: 192.168.0.50 (port 9998)
{Thu Apr 10 21:40:07 2003 }
OemNodeConfig(): Network Address of node1: 192.168.0.51 (port 9998)
{Thu Apr 10 21:40:07 2003 }
WARNING: OemInit2: Opened file(/var/opt/oracle/oradata/orcl/CMQuorumFile 8), tid = main:1024 file = oem.c, line = 491 {Thu Apr 10 21:40:07 2003 }Debug Hang : ClusterListener (PID=2541) Registered withwatchdog daemon. {Thu Apr 10 21:40:09 2003 }
InitializeCM: ModuleName = hangcheck-timer {Thu Apr 10 21:40:09 2003 }
InitializeCM: Kernel module hangcheck-timer is already loaded {Thu Apr 10 21:40:09 2003 }
Debug Hang : CmConnectListener (PID=2542):Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
Debug Hang :StartNMMon (PID=2536) Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
CreateLocalEndpoint(): Network Address: 192.168.0.51
{Thu Apr 10 21:40:09 2003 }
Debug Hang :PollingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
Debug Hang : DiskPingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
Debug Hang :SendingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
--- DUMP GROUP STATE DB ---
--- END OF GROUP STATE DUMP ---
--- Begin Dump ---
TRACE: LogListener: Spawned with tid 0x803., tid = 2051 file = logging.c, line = 116 {Thu Apr 10 21:40:07 2003 }oracm, version[ 9.2.0.2.0.47 ] started {Thu Apr 10 21:40:07 2003 }
TRACE: Can't read registry value for WatchdogTimerMargin, tid = main:1024 file = unixinc.c, line = 1011 {Thu Apr 10 21:40:07 2003 }
TRACE: Can't read registry value for WatchdogSafetyMargin, tid = main:1024 file = unixinc.c, line = 1011 {Thu Apr 10 21:40:07 2003 }KernelModuleName is hangcheck-timer {Thu Apr 10 21:40:07 2003 }
TRACE: Can't read registry value for ClientTimeout, tid = main:1024 file = unixinc.c, line = 1011 {Thu Apr 10 21:40:07 2003 }
TRACE: InitNMInfo: setting clientTimeout to 215s based on MissCount 215 and PollInterval 1000ms, tid = main:1024 file = nmconfig.c, line = 137 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): getservbyname on CMSrvr failed - 22 : assigning 9998, tid = main:1024 file = nmconfig.c, line = 212 {Thu Apr 10 21:40:07 2003 }OemNodeConfig(): Network Address of node0: 192.168.0.50 (port 9998)
{Thu Apr 10 21:40:07 2003 }
OemNodeConfig(): Network Address of node1: 192.168.0.51 (port 9998)
{Thu Apr 10 21:40:07 2003 }
TRACE: OemCreateListenPort: bound at 9998, tid = main:1024 file = oem.c, line = 858 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): found my node info at 1 name rac2, priv rac2priv, port 3623, tid = main:1024 file = nmconfig.c, line = 265 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): Local Node(1) NodeName[rac2], tid = main:1024 file = nmconfig.c, line = 283 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): Cluster(Oracle) with (2) Defined Nodes, tid = main:1024 file = nmconfig.c, line = 286 {Thu Apr 10 21:40:07 2003 }
TRACE: OEMInits(): CM Disk File (/var/opt/oracle/oradata/orcl/CMQuorumFile), tid = main:1024 file = oem.c, line = 243 {Thu Apr 10 21:40:07 2003 }
WARNING: OemInit2: Opened file(/var/opt/oracle/oradata/orcl/CMQuorumFile 8), tid = main:1024 file = oem.c, line = 491 {Thu Apr 10 21:40:07 2003 }
TRACE: ReadOthersDskInfo(): node(0) rcfg(11) wrtcnt(1052) lastcnt(0) alive(1052), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(1) rcfg(10) wrtcnt(2) lastcnt(0) alive(1), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(2) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(3) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(4) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(5) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(6) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(7) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(8) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(9) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(10) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(11) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(12) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(13) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(14) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(15) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(16) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(17) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(18) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(19) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(20) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(21) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(22) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(23) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(24) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(25) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(26) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(27) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(28) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(29) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(30) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(31) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(32) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(33) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(34) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(35) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(36) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(37) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(38) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(39) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(40) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(41) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(42) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(43) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(44) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(45) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(46) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(47) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(48) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(49) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(50) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(51) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(52) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(53) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(54) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(55) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(56) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(57) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(58) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(59) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(60) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(61) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(62) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(63) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ClusterListener: Spawned with tid 0xc04., tid = 3076 file = nmlistener.c, line = 53 {Thu Apr 10 21:40:09 2003 }Debug Hang : ClusterListener (PID=2541) Registered withwatchdog daemon. {Thu Apr 10 21:40:09 2003 }
TRACE: ClusterListener (pid=2541, tid=3076): Registered with watchdog daemon., tid = 3076 file = nmlistener.c, line = 75 {Thu Apr 10 21:40:09 2003 }InitializeCM: ModuleName = hangcheck-timer {Thu Apr 10 21:40:09 2003 }
InitializeCM: Kernel module hangcheck-timer is already loaded {Thu Apr 10 21:40:09 2003 }
TRACE: CmConnectListener: Spawned with tid 0x1005., tid = 4101 file = cmclient.c, line = 215 {Thu Apr 10 21:40:09 2003 }Debug Hang : CmConnectListener (PID=2542):Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
TRACE: CmConnectListener (pid=2542, tid=4101): Registered with watchdog daemon., tid = 4101 file = cmclient.c, line = 246 {Thu Apr 10 21:40:09 2003 }Debug Hang :StartNMMon (PID=2536) Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
TRACE: StartNMMon (pid=2536, tid=1024): Registered with watchdog daemon., tid = main:1024 file = cmnodemon.c, line = 251 {Thu Apr 10 21:40:09 2003 }CreateLocalEndpoint(): Network Address: 192.168.0.51
{Thu Apr 10 21:40:09 2003 }
TRACE: StartClusterJoin(): clusterState(0) nodeState(0), tid = main:1024 file = nmmember.c, line = 277 {Thu Apr 10 21:40:09 2003 }
TRACE: PollingThread: Spawned with tid 0x1406., tid = 5126 file = nmmember.c, line = 742 {Thu Apr 10 21:40:09 2003 }Debug Hang :PollingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
TRACE: PollingThread (pid=2543, tid=5126): Registered with watchdog daemon., tid = 5126 file = nmmember.c, line = 760 {Thu Apr 10 21:40:09 2003 }
TRACE: DiskPingThread: Spawned with tid 0x1807., tid = 6151 file = nmmember.c, line = 1050 {Thu Apr 10 21:40:09 2003 }Debug Hang : DiskPingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
TRACE: DiskPingThread (pid=2544, tid=6151): Registered with watchdog daemon., tid = 6151 file = nmmember.c, line = 1074 {Thu Apr 10 21:40:09 2003 }
TRACE: SendingThread: Spawned with tid 0x1c08, 0x0., tid = 7176 file = nmmember.c, line = 511 {Thu Apr 10 21:40:09 2003 }Debug Hang :SendingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
TRACE: SendingThread (pid=2545, tid=7176): Registered with watchdog daemon., tid = 7176 file = nmmember.c, line = 576 {Thu Apr 10 21:40:09 2003 }
TRACE: HandleJoin(): src[1] dest[1] dom[0] seq[1] sync[0], tid = ClusterListener:3076 file = nmlisten.c, line = 346 {Thu Apr 10 21:40:09 2003 }
TRACE: HandleJoin(): JOIN from node(1)->(1), tid = ClusterListener:3076 file = nmlisten.c, line = 362 {Thu Apr 10 21:40:09 2003 }
TRACE: HandleStatus(): node(0) UNKNOWN, tid = ClusterListener:3076 file = nmlisten.c, line = 404 {Thu Apr 10 21:40:10 2003 }
TRACE: HandleStatus(): src[0] dest[1] dom[0] seq[36] sync[11], tid = ClusterListener:3076 file = nmlisten.c, line = 415 {Thu Apr 10 21:40:10 2003 }
TRACE: HandleSync(): src[0] dest[1] dom[0] seq[37] sync[11], tid = ClusterListener:3076 file = nmlisten.c, line = 506 {Thu Apr 10 21:40:10 2003 }
TRACE: SendAck(): node(0) domain(0) syncSeqNo(11) type(11), tid = ClusterListener:3076 file = nmmember.c, line = 1913 {Thu Apr 10 21:40:10 2003 }
TRACE: HandleVote(): src[0] dest[1] dom[0] seq[38] sync[11], tid = ClusterListener:3076 file = nmlisten.c, line = 643 {Thu Apr 10 21:40:11 2003 }
TRACE: SendVoteInfo(): node(0) domain(0) syncSeqNo(11), tid = ClusterListener:3076 file = nmmember.c, line = 1727 {Thu Apr 10 21:40:11 2003 }
TRACE: HandleShutdown(): src[0] dest[1] dom[0] seq[0] sync[11] type[4], tid = ClusterListener:3076 file = nmlisten.c, line = 1087 {Thu Apr 10 21:40:12 2003 }
TRACE: IncrementEventValue: *(80f2920) = (1, 1), tid = ClusterListener:3076 file = unixinc.c, line = 253 {Thu Apr 10 21:40:12 2003 }--- End Dump ---
=========================================================

Database is not starting after RAC installation

Hi All,
I have installed 10g R2 2-node RAC on RHEL4 without any problem. But after the installation the database is not starting and i'm not able to connect. It is showing the below error..
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory
Can anybody help what might be the problem?
Thanks,
Praveen

Hi,
I'm giving the contents of alter log file from the first node...
Mon Feb 19 16:45:59 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
sga_target = 1073741824
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database_instances= 1
db_create_file_dest = +ORCL_DATA1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 16:46:00 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=31282
DIAG started with pid=3, OS id=31284
PSP0 started with pid=4, OS id=31286
LMON started with pid=5, OS id=31300
LMD0 started with pid=6, OS id=31315
MMAN started with pid=7, OS id=31317
DBW0 started with pid=8, OS id=31319
LGWR started with pid=9, OS id=31321
CKPT started with pid=10, OS id=31323
SMON started with pid=11, OS id=31325
RECO started with pid=12, OS id=31327
CJQ0 started with pid=13, OS id=31329
MMON started with pid=14, OS id=31331
Mon Feb 19 16:46:00 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=31333
Mon Feb 19 16:46:00 2007
starting up 1 shared server(s) ...
Mon Feb 19 16:46:00 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 16:46:01 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Reconfiguration complete
Mon Feb 19 16:46:01 2007
CREATE DATABASE "bsaprod"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET WE8ISO8859P1
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
Mon Feb 19 16:46:01 2007
Starting background process ASMB
ASMB started with pid=19, OS id=31365
Starting background process RBAL
RBAL started with pid=20, OS id=31369
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 16:46:05 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 16:46:06 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 16:46:06 2007
Database mounted in Exclusive Mode
Mon Feb 19 16:46:09 2007
Successful mount of redo thread 1, with mount id 3309011657
Assigning activation ID 3309011657 (0xc53b82c9)
Thread 1 opened at log sequence 1
Current log# 1 seq# 1 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 16:46:09 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 16:46:09 2007
SMON: enabling cache recovery
Mon Feb 19 16:46:09 2007
create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL online
Mon Feb 19 16:46:20 2007
Completed: create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL online
Mon Feb 19 16:46:20 2007
create rollback segment SYSTEM tablespace SYSTEM
storage (initial 50K next 50K)
Completed: create rollback segment SYSTEM tablespace SYSTEM
storage (initial 50K next 50K)
Mon Feb 19 16:46:24 2007
CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:31 2007
Successfully onlined Undo Tablespace 1.
Completed: CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:31 2007
create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
Completed: create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
Mon Feb 19 16:46:36 2007
CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Completed: CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Mon Feb 19 16:46:36 2007
ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
Completed: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
Mon Feb 19 16:46:36 2007
ALTER DATABASE DEFAULT TABLESPACE SYSTEM
Completed: ALTER DATABASE DEFAULT TABLESPACE SYSTEM
Mon Feb 19 16:46:38 2007
SMON: enabling tx recovery
Mon Feb 19 16:46:39 2007
Threshold validation cannot be done before catproc is loaded.
Threshold validation cannot be done before catproc is loaded.
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=22, OS id=32303
Mon Feb 19 16:46:39 2007
Completed: CREATE DATABASE "bsaprod"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET WE8ISO8859P1
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
Mon Feb 19 16:46:39 2007
CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:46 2007
CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Completed: CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Mon Feb 19 16:46:46 2007
ALTER DATABASE DEFAULT TABLESPACE "USERS"
Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"
Mon Feb 19 16:47:05 2007
Thread 1 advanced to log sequence 2
Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:47:34 2007
Thread 1 cannot allocate new log, sequence 3
Checkpoint not complete
Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 3
Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:48:10 2007
Thread 1 cannot allocate new log, sequence 4
Checkpoint not complete
Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 4
Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:49:05 2007
Thread 1 cannot allocate new log, sequence 5
Checkpoint not complete
Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 5
Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:50:19 2007
Thread 1 cannot allocate new log, sequence 6
Checkpoint not complete
Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 6
Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:51:00 2007
Thread 1 cannot allocate new log, sequence 7
Checkpoint not complete
Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 7
Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:51:54 2007
Thread 1 cannot allocate new log, sequence 8
Checkpoint not complete
Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 8
Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:55:32 2007
Thread 1 cannot allocate new log, sequence 9
Checkpoint not complete
Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 9
Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:56:35 2007
Thread 1 cannot allocate new log, sequence 10
Checkpoint not complete
Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 10
Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:56:52 2007
Thread 1 cannot allocate new log, sequence 11
Checkpoint not complete
Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 11
Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:57:04 2007
Thread 1 cannot allocate new log, sequence 12
Checkpoint not complete
Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 12
Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:57:14 2007
Thread 1 cannot allocate new log, sequence 13
Checkpoint not complete
Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 13
Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:57:31 2007
Thread 1 cannot allocate new log, sequence 14
Checkpoint not complete
Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 14
Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:58:13 2007
Thread 1 cannot allocate new log, sequence 15
Checkpoint not complete
Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 15
Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:59:42 2007
Thread 1 cannot allocate new log, sequence 16
Checkpoint not complete
Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 16
Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:00:33 2007
Thread 1 cannot allocate new log, sequence 17
Checkpoint not complete
Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 17
Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:01:49 2007
Thread 1 cannot allocate new log, sequence 18
Checkpoint not complete
Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 18
Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:02:23 2007
Thread 1 cannot allocate new log, sequence 19
Checkpoint not complete
Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 19
Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:04:06 2007
Thread 1 cannot allocate new log, sequence 20
Checkpoint not complete
Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 20
Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:05:22 2007
Thread 1 cannot allocate new log, sequence 21
Checkpoint not complete
Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 21
Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:06:22 2007
Thread 1 cannot allocate new log, sequence 22
Checkpoint not complete
Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 22
Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:08:42 2007
Thread 1 cannot allocate new log, sequence 23
Checkpoint not complete
Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 23
Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:09:43 2007
Thread 1 cannot allocate new log, sequence 24
Checkpoint not complete
Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 24
Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:10:42 2007
Thread 1 cannot allocate new log, sequence 25
Checkpoint not complete
Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 25
Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:11:31 2007
Thread 1 cannot allocate new log, sequence 26
Checkpoint not complete
Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 26
Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:12:01 2007
Thread 1 cannot allocate new log, sequence 27
Checkpoint not complete
Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 27
Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:13:12 2007
Thread 1 cannot allocate new log, sequence 28
Checkpoint not complete
Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 28
Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:14:05 2007
Thread 1 cannot allocate new log, sequence 29
Checkpoint not complete
Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 29
Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:14:57 2007
Thread 1 cannot allocate new log, sequence 30
Checkpoint not complete
Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 30
Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:16:29 2007
Thread 1 cannot allocate new log, sequence 31
Checkpoint not complete
Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:17:54 2007
Starting background process EMN0
EMN0 started with pid=25, OS id=15651
Mon Feb 19 17:17:54 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:17:55 2007
Stopping background process QMNC
Mon Feb 19 17:17:55 2007
Stopping background process CJQ0
Mon Feb 19 17:17:57 2007
Stopping background process MMNL
Mon Feb 19 17:17:58 2007
Stopping background process MMON
Mon Feb 19 17:17:59 2007
Shutting down instance (immediate)
License high water mark = 1
Mon Feb 19 17:17:59 2007
Stopping Job queue slave processes
Mon Feb 19 17:17:59 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:18:01 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:01 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:18:03 2007
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:18:04 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:04 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:18:04 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:18:04 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:18:09 2007
freeing rdom 0
Mon Feb 19 17:18:12 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database_instances= 1
db_create_file_dest = +ORCL_DATA1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:18:13 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=16124
DIAG started with pid=3, OS id=16126
PSP0 started with pid=4, OS id=16128
LMON started with pid=5, OS id=16130
LMD0 started with pid=6, OS id=16132
MMAN started with pid=7, OS id=16134
DBW0 started with pid=8, OS id=16136
LGWR started with pid=9, OS id=16138
CKPT started with pid=10, OS id=16140
SMON started with pid=11, OS id=16142
RECO started with pid=12, OS id=16144
CJQ0 started with pid=13, OS id=16146
MMON started with pid=14, OS id=16148
Mon Feb 19 17:18:13 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=16150
Mon Feb 19 17:18:13 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:18:13 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:18:14 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Reconfiguration complete
Mon Feb 19 17:18:14 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:18:14 2007
Starting background process ASMB
ASMB started with pid=19, OS id=16182
Starting background process RBAL
RBAL started with pid=20, OS id=16186
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:18:18 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:18:23 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:18:23 2007
Successful mount of redo thread 1, with mount id 3309012054
Mon Feb 19 17:18:23 2007
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:18:23 2007
alter database archivelog
Completed: alter database archivelog
Mon Feb 19 17:18:23 2007
alter database open
Mon Feb 19 17:18:23 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=22, OS id=16393
Mon Feb 19 17:18:23 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=23, OS id=16395
Mon Feb 19 17:18:23 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:18:23 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:18:23 2007
SMON: enabling cache recovery
Mon Feb 19 17:18:23 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:18:23 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:18:23 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=24, OS id=16422
Mon Feb 19 17:18:23 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:18:23 2007
SMON: enabling tx recovery
Mon Feb 19 17:18:23 2007
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=25, OS id=16449
Mon Feb 19 17:18:25 2007
Completed: alter database open
Mon Feb 19 17:18:25 2007
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
GROUP 4 SIZE 51200K
Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
GROUP 4 SIZE 51200K
Mon Feb 19 17:18:28 2007
ALTER DATABASE ENABLE PUBLIC THREAD 2
Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:18:29 2007
Starting background process EMN0
EMN0 started with pid=26, OS id=16564
Mon Feb 19 17:18:29 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:18:29 2007
Stopping background process QMNC
Mon Feb 19 17:18:29 2007
Stopping background process CJQ0
Mon Feb 19 17:18:30 2007
Stopping background process MMNL
Mon Feb 19 17:18:31 2007
Stopping background process MMON
Mon Feb 19 17:18:32 2007
Shutting down instance (immediate)
License high water mark = 3
Mon Feb 19 17:18:32 2007
Stopping Job queue slave processes
Mon Feb 19 17:18:32 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:18:34 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:34 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:18:34 2007
Shutting down archive processes
Archiving is disabled
Mon Feb 19 17:18:39 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:18:44 2007
ARCH shutting down
ARC1: Archival stopped
Mon Feb 19 17:18:49 2007
ARCH shutting down
ARC0: Archival stopped
Mon Feb 19 17:18:50 2007
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:18:51 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:51 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:18:51 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:18:51 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:18:56 2007
freeing rdom 0
Mon Feb 19 17:18:59 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +ORCL_DATA1
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:19:00 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=17373
DIAG started with pid=3, OS id=17375
PSP0 started with pid=4, OS id=17377
LMON started with pid=5, OS id=17379
LMD0 started with pid=6, OS id=17381
LMS0 started with pid=7, OS id=17383
LMS1 started with pid=8, OS id=17387
MMAN started with pid=9, OS id=17391
DBW0 started with pid=10, OS id=17393
LGWR started with pid=11, OS id=17395
CKPT started with pid=12, OS id=17397
SMON started with pid=13, OS id=17399
RECO started with pid=14, OS id=17412
CJQ0 started with pid=15, OS id=17428
MMON started with pid=16, OS id=17430
Mon Feb 19 17:19:00 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=17432
Mon Feb 19 17:19:00 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:19:01 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:19:01 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Feb 19 17:19:01 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Feb 19 17:19:01 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Mon Feb 19 17:19:01 2007
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:19:01 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:19:01 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=17460
Mon Feb 19 17:19:02 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:19:02 2007
This instance was first to mount
Mon Feb 19 17:19:02 2007
Starting background process ASMB
ASMB started with pid=22, OS id=17466
Starting background process RBAL
RBAL started with pid=23, OS id=17470
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:19:06 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:19:09 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:19:10 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:19:10 2007
Successful mount of redo thread 1, with mount id 3308997510
Mon Feb 19 17:19:10 2007
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:19:11 2007
ALTER DATABASE OPEN
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Mon Feb 19 17:19:11 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=26, OS id=17679
Mon Feb 19 17:19:11 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=27, OS id=17706
Mon Feb 19 17:19:11 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:19:11 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:19:11 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:19:11 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:19:11 2007
SMON: enabling cache recovery
Mon Feb 19 17:19:11 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=28, OS id=17708
Mon Feb 19 17:19:11 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:19:11 2007
SMON: enabling tx recovery
Mon Feb 19 17:19:11 2007
Database Characterset is WE8ISO8859P1
Mon Feb 19 17:19:11 2007
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Mon Feb 19 17:19:12 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:19:12 2007
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=17735
Mon Feb 19 17:19:13 2007
Completed: ALTER DATABASE OPEN
Mon Feb 19 17:20:11 2007
Shutting down archive processes
Mon Feb 19 17:20:16 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:23:35 2007
Starting background process EMN0
EMN0 started with pid=74, OS id=28350
Mon Feb 19 17:23:35 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:23:36 2007
Stopping background process QMNC
Mon Feb 19 17:23:36 2007
Stopping background process CJQ0
Mon Feb 19 17:23:38 2007
Stopping background process MMNL
Mon Feb 19 17:23:39 2007
Stopping background process MMON
Mon Feb 19 17:23:40 2007
Shutting down instance (immediate)
License high water mark = 34
Mon Feb 19 17:23:40 2007
Stopping Job queue slave processes
Mon Feb 19 17:23:40 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:24:07 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:24:07 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:24:08 2007
Shutting down archive processes
Archiving is disabled
Mon Feb 19 17:24:18 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC1: Archiving disabled
ARCH shutting down
Mon Feb 19 17:24:18 2007
ARC1: Archival stopped
Mon Feb 19 17:24:18 2007
ARC0: Becoming the heartbeat ARCH
Mon Feb 19 17:24:18 2007
ARC0: Archiving disabled
ARCH shutting down
ARC0: Archival stopped
Mon Feb 19 17:24:19 2007
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:24:19 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:24:19 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:24:19 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:24:19 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:24:25 2007
freeing rdom 0
Mon Feb 19 17:24:35 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
__shared_pool_size = 255852544
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 0
spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
__db_cache_size = 801112064
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +ORCL_DATA1
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:24:35 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=29950
DIAG started with pid=3, OS id=29952
PSP0 started with pid=4, OS id=29954
LMON started with pid=5, OS id=29956
LMD0 started with pid=6, OS id=29958
LMS0 started with pid=7, OS id=29960
LMS1 started with pid=8, OS id=29964
MMAN started with pid=9, OS id=29968
DBW0 started with pid=10, OS id=29970
LGWR started with pid=11, OS id=29972
CKPT started with pid=12, OS id=29974
SMON started with pid=13, OS id=29976
RECO started with pid=14, OS id=29978
CJQ0 started with pid=15, OS id=29980
MMON started with pid=16, OS id=29982
Mon Feb 19 17:24:36 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=29984
Mon Feb 19 17:24:36 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:24:36 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:24:39 2007
Reconfiguration started (old inc 0, new inc 4)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 1
* domain 0 valid = 0 according to instance 1
Mon Feb 19 17:24:39 2007
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Feb 19 17:24:39 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Feb 19 17:24:39 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Mon Feb 19 17:24:39 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:24:39 2007
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:24:39 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=30069
Mon Feb 19 17:24:40 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:24:41 2007
Starting background process ASMB
ASMB started with pid=22, OS id=30121
Starting background process RBAL
RBAL started with pid=23, OS id=30125
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:24:45 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:24:49 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:24:49 2007
Successful mount of redo thread 1, with mount id 3309017062
Mon Feb 19 17:24:49 2007
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Mon Feb 19 17:24:51 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:24:52 2007
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:24:52 2007
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Mon Feb 19 17:24:52 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=25, OS id=30406
Mon Feb 19 17:24:53 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=26, OS id=30422
Mon Feb 19 17:24:53 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:24:53 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:24:53 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:24:53 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:24:53 2007
SMON: enabling cache recovery
Mon Feb 19 17:24:53 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=27, OS id=30436
Mon Feb 19 17:25:07 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:25:07 2007
SMON: enabling tx recovery
Mon Feb 19 17:25:07 2007
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=30882
Mon Feb 19 17:26:31 2007
Completed: ALTER DATABASE OPEN
Mon Feb 19 17:26:38 2007
ALTER SYSTEM SET service_names='bsaprod','bsatest' SCOPE=MEMORY SID='bsaprod1';
Mon Feb 19 17:26:53 2007
Shutting down archive processes
Mon Feb 19 17:26:58 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:32:16 2007
Error: KGXGN aborts the instance (6)
Mon Feb 19 17:32:16 2007
Errors in file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_lmon_29956.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Mon Feb 19 17:32:17 2007
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_diag_29952.trc
Mon Feb 19 17:32:18 2007
Trace dumping is performing id=[cdmp_20070219173217]
Mon Feb 19 17:32:21 2007
Instance terminated by LMON, pid = 29956
and also the output of crs_stat -t in first node is....
Name Type Target State Host
ora....d1.inst application ONLINE OFFLINE
ora....d2.inst application ONLINE OFFLINE
ora....od1.srv application ONLINE OFFLINE
ora....od2.srv application ONLINE OFFLINE
ora....test.cs application ONLINE OFFLINE
ora.bsaprod.db application ONLINE OFFLINE
ora....SM1.asm application ONLINE UNKNOWN dsrvbd003
ora....ER.lsnr application ONLINE UNKNOWN dsrvbd003
ora....03.lsnr application ONLINE UNKNOWN dsrvbd003
ora....003.gsd application ONLINE UNKNOWN dsrvbd003
ora....003.ons application ONLINE UNKNOWN dsrvbd003
ora....003.vip application ONLINE ONLINE dsrvbd003
ora....SM2.asm application ONLINE OFFLINE
ora....04.lsnr application ONLINE OFFLINE
ora....004.gsd application ONLINE OFFLINE
ora....004.ons application ONLINE OFFLINE
ora....004.vip application ONLINE OFFLINE

RAC node is not starting

Similar Messages

Maybe you are looking for