Node manager not starting

I have just installed a second CRS 5.0(2) server which I intend to add to the cluster, but I'm having a problem in that the Node Manager Service hangs on startup. The error I get is:
Event ID:7022
The CRS Node Manager Service Hung on starting.
I can see from the Windows services that the service remains in a 'starting state'.
I wanted to attach the MIVR logs to this post but I'm finding that there is no MIVR folder in c:/Program Files/wfavvid/log. Has the install completed successfully? I had no errors when installing the package.
I've checked the binding order of the NIC cards and they are correct. I have also run the 'postinstall.exe' but still the same problem.
On another note, how can you tell if the license provided (uploaded by someone else) includes HA software?
Thanks

Try checking out the MCVD logs and the application and system logs for any error with NIC or network connectivity.
As for the second question the key you are looking for in the Lic is the entry.
INCREMENT CRS_EXP_PRE_WARM_STBY cisco 5.0
In case its already uploaded in CRS the Control Center >Display Licenses will show HA enabled.

Similar Messages

Node Manager not starting UCM Server automatically

Hi All,
I have configured Node Manager to start and stop UCM 11g server.
But whenever, I am restarting the host OS - Windows server 2003 64 bit, it does not start the UCM managed server, automatically.
I am running two managed servers on the same host OS, the first UCM manages server, starts automatically on every restart.
The second managed UCM server, does not start. I need to then manually start the managed server. There are no errors logged, and the manual start of the managed server happens without any issues.
As a workaround, I tried starting the managed server from Windows scheduler, but it does not start the server as well. I wrote a script like -
Myscript.cmd
call E:\..\user_projects\domains\prod_domain\bin\setDomainEnv.cmd
java weblogic.WLST c:\start....\ServerStart.py
scheduled the script (myscript.cmd) to run from the Windows Scheduler.
Any pointers would be very helpful. Are there any known issues, if there are two managed servers, and one starts automatically, which affects the starting of the second managed server?
I see the following error in the log file -
<14/10/2011 6:35:22 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<14/10/2011 6:35:33 PM EST> <Info> <Management> <BEA-141223> <The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.>
<14/10/2011 6:35:33 PM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.
at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
at weblogic.Server.main(Server.java:71)
Regards
Avi

Hi Avi,
Give a cross check to your ServerStart.py
As the error states The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist
In your ServerStart.py you might have specified as follows
sh start MAnagedWeblogic.sh server=PRD_UCM_Server1
While these are the list of Server you have in your weblogic
PRDAdminServer,
PRD_UCM_server1,(You have Specified in your command PRD_UCM_Server1
PRD_IBR_server1
The Command should be as follows
sh start ManagedWeblogic.sh server=PRD_UCM_server1
This should help
Regards
Fabian

Node manager not starting servers as a OS service.

Node manager is starting upon simulating OS crash but it is not bringing up the managed servers running on that machine.
The following is the nodemanager.properties file -
[orawl@qa-wl2 nodemanager]$ more nodemanager.properties
#Mon Apr 11 14:55:43 EDT 2011
#Mon Apr 11 11:57:43 EDT 2011
DomainsFile=/home/orawl/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.domains
LogLimit=0
PropertiesVersion=10.3
DomainsDirRemoteSharingEnabled=false
javaHome=/usr/lib/jdk1.6.0_24
AuthenticationEnabled=true
NodeManagerHome=/home/orawl/oracle/middleware/wlserver_10.3/common/nodemanager
JavaHome=/usr/lib/jdk1.6.0_24/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=true
LogCount=1
DomainRegistrationEnabled=false
StopScriptEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=true
StartScriptEnabled=true
LogFile=/home/orawl/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50
#NodeManager SSL Properties
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreType=JKS
CustomIdentityAlias=xxxxxxxxxxxx
CustomIdentityKeyStoreFileName=/home/orawl/oracle/middleware/keystores/xxxxxxxxxxxx.jks
CustomIdentityKeyStorePassPhrase={3DES}XXXXXXXXXXXXXXXXXXXXXXX
CustomIdentityPrivateKeyPassPhrase={3DES}XXXXXXXXXXXXXXXXXXXXXXX
CustomTrustKeyStoreType=JKS
CustomTrustKeyStoreFileName=/home/orawl/oracle/middleware/keystores/trustcacerts.jks
CustomTrustKeyPassPhrase=xxxxxxxxxxx
Please advice.

Hi All,
Add to my initial post , the error which is getting logged is -
<14/10/2011 6:35:22 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<14/10/2011 6:35:33 PM EST> <Info> <Management> <BEA-141223> <The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.>
<14/10/2011 6:35:33 PM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.
     at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
     at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
     at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
     at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
     at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
     at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
     at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
     at weblogic.Server.main(Server.java:71)
Regards,
Avi

Node Manager not starting UCM Server

Hi All,
I have configured Node Manager to start and stop UCM 11g server.
But whenever, I am restarting the host OS - Windows server 2003 64 bit, it does not start the UCM managed server, automatically.
I am running two managed servers on the same host OS, the first UCM manages server, starts automatically on every restart.
The second managed UCM server, does not start. I need to then manually start the managed server. There are no errors logged, and the manual start of the managed server happens without any issues.
As a workaround, I tried starting the managed server from Windows scheduler, but it does not start the server as well. I wrote a script like -
Myscript.cmd
call E:\..\user_projects\domains\prod_domain\bin\setDomainEnv.cmd
java weblogic.WLST c:\start....\ServerStart.py
scheduled the script (myscript.cmd) to run from the Windows Scheduler.
Any pointers would be very helpful. Are there any known issues, if there are two managed servers, and one starts automatically, which affects the starting of the second managed server?
Regards
Avi

Hi All,
Add to my initial post , the error which is getting logged is -
<14/10/2011 6:35:22 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<14/10/2011 6:35:33 PM EST> <Info> <Management> <BEA-141223> <The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.>
<14/10/2011 6:35:33 PM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name PRD_UCM_Server1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {PRDAdminServer,PRD_UCM_server1,PRD_IBR_server1}.
     at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
     at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
     at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
     at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
     at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
     at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
     at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
     at weblogic.Server.main(Server.java:71)
Regards,
Avi

Node Manager Not Starting Managed Servers

I have nodemanager configured to restart managed servers automatically upon reboot but when I physically restart the 2nd node none of the managed servers start and there are no errors. Steps completed and verified....
1) Nodemanager.properties has domain path and 'crashrecoveryenabled' set to 'true'
2) managed servers have autorestart set to 'true' and -Xnohup parameter added to startup arguments.
3) Boot.properties files ave the username and password (encrypted) to start up the server.
Here is the nodemanager.log file and you can clearly see it is grabbing the startup files. It almost looks like the server files are trying to get loaded before the nodemanager port is available but I see no errors. Any ideas?
<Dec 28, 2009 9:16:59 AM> <Info> <QA10311> <WebProxyServer2> <Working directory is "C:\bea\user_projects\domains\QA10311">
<Dec 28, 2009 9:16:59 AM> <Info> <QA10311> <WebProxyServer2> <Server output log file is "C:\bea\user_projects\domains\QA10311\servers\WebProxyServer2\logs\WebProxyServer2.out">
<Dec 28, 2009 9:47:09 AM> <INFO> <Loading domains file: C:\bea\WLSERV~1.3\common\nodemanager\nodemanager.domains>
<Dec 28, 2009 9:47:16 AM> <INFO> <Loading identity key store: FileName=C:/bea/WLSERV~1.3/server\lib\DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Dec 28, 2009 9:47:18 AM> <INFO> <Loaded node manager configuration properties from 'C:\bea\WLSERV~1.3\common\nodemanager\nodemanager.properties'>
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <ACR2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\ACR2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <CommandNet2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\CommandNet2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <FPCE2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\FPCE2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:21 AM> <Info> <QA10311> <MQServer2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\MQServer2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <PortalServer2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\PortalServer2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <ShoppingCart2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\ShoppingCart2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <webapp2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\webapp2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <WebProxyServer2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\WebProxyServer2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:22 AM> <Info> <QA10311> <WUA2> <Startup configuration properties loaded from "C:\bea\user_projects\domains\QA10311\servers\WUA2\data\nodemanager\startup.properties">
<Dec 28, 2009 9:47:27 AM> <INFO> <Secure socket listener started on port 5556, host qaportal2.FCQA.COM>

Hi,
Entries in nodemanager.properties looks fine, but how are you testing this functionality?
If you will stop servers monitored by nodemanager, stop nodemanager and then reboot physical server, on startup if nodemanager is configured as daemon it will start nm process but will do nothing to managed server process.
To test "CrashRecoveryEnabled=true" kill the managed server pid and then stop/kill nodemanager process. Then perform physical server reboot, now if nm is configured as daemon process server reboot will start nodemanager. Once NM is up first thing it check is the server it was monitoring earlier and if under directory $WL_HOME/servers/<server>/data/nodemanager it finds "pid" and "lck" file it checks if that pid exists and if not then go for a restart of weblogic server instance.
This has worked for me...!
Thanks,
Ranjan
http://www.middlewaresupport.wordpress.com

Node Manager not starting servicess after server reboot

We have WLS 10.1.3.5 with IDM/OAM(two domains in same server as its TEST environment) on Oracle Linux 5(64bit),when ever server reboots node manger is not starting managed servers.
I have enabled CrashRecoveryEnabled=true
Here is nodemanager.properties.
[oracle@oam nodemanager]$ cat nodemanager.properties
#Changed NM Listen Port
#Tue Mar 05 03:12:20 IST 2013
DomainsFile=/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager/nodemanager.domains
LogLimit=0
DomainsDirRemoteSharingEnabled=false
PropertiesVersion=10.3
AuthenticationEnabled=true
NodeManagerHome=/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager
javaHome=/u01/Oracle/Middleware/jdk160_24
JavaHome=/u01/Oracle/Middleware/jdk160_24/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=false
LogCount=1
StopScriptEnabled=false
DomainRegistrationEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=true
StartScriptEnabled=true
LogFile=/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50
here is my domain information in server.
=========================
[oracle@oam nodemanager]$ cat nodemanager.domains
#Domains and directories created by Configuration Wizard
#Tue Mar 05 05:24:37 IST 2013
OAMDomain=/u01/Oracle/Middleware/user_projects/domains/OAMDomain
IDMDomain=/u01/Oracle/Middleware/user_projects/domains/IDMDomain

Hi,
Entries in nodemanager.properties looks fine, but how are you testing this functionality?
If you will stop servers monitored by nodemanager, stop nodemanager and then reboot physical server, on startup if nodemanager is configured as daemon it will start nm process but will do nothing to managed server process.
To test "CrashRecoveryEnabled=true" kill the managed server pid and then stop/kill nodemanager process. Then perform physical server reboot, now if nm is configured as daemon process server reboot will start nodemanager. Once NM is up first thing it check is the server it was monitoring earlier and if under directory $WL_HOME/servers/<server>/data/nodemanager it finds "pid" and "lck" file it checks if that pid exists and if not then go for a restart of weblogic server instance.
This has worked for me...!
Thanks,
Ranjan
http://www.middlewaresupport.wordpress.com

ACCOUNTING HIERARCHY MANAGER NOT STARTED,APPLET ERROR JBO-27022

제품 : FIN_GL
작성날짜 : 2004-05-20
ACCOUNTING HIERARCHY MANAGER NOT STARTED,APPLET ERROR JBO-27022
===============================================================
Problem Description
Account Hierarchy Manager 실행시 아래와 같이 JBO-27022 error발생하면서,
해당 기능을 사용할 수 없을 경우에 대한 원인 및 조치사항에 대해 언급한다.
Navigation PATH : GL Setup -> Accounts -> Manager
Error내용
oracle.jbo.AttributeLoadException: JBO-27022: java.sql.SQLException(?ㅈ)ㅇㅙ ?ㅙㅔ?
java.lang.String ?ㄿㅔ??ㅔ Java ㅀㅄㅐㅌㅈㄶ ㅋ???ㅔㅚ?ㄹ 13 ?ㅙㅅㄶㅍㅊ?ㄱㅌㄽ ㅀㄺ?ㅋ ㅇㅙㅅ￥ㅔㅚㅄㅏ ㅅㄵ
ㅍㅔㅓ?ㅔ?ㅍ?ㅄㅚㅄ?.
     at java.lang.Throwable.<init>(Compiled Code)
     at java.lang.Exception.<init>(Compiled Code)
     at java.lang.RuntimeException.<init>(RuntimeException.java:39)
     at oracle.jbo.JboException.<init>(JboException.java:354)
## Detail 0 ##
java.sql.SQLException: UTF8ㅀ? UCS2 ㅀㄳ?ㄱ ㅌㄽㅇㅙ ㅊㄿ?ㄿㅔㅛ ㅌ? ㅎ?ㅍ?ㅄㅚㅄ?: failUTF8Conv
     at java.lang.Throwable.<init>(Compiled Code)
     at java.lang.Exception.<init>(Compiled Code)
     at java.sql.SQLException.<init>(SQLException.java:43)
Solution Description
GL setup->financial->flexfileds->key -> segments 화면의
Structures region에서 Title에 입력된 Data가 20 chars(한글은 10글자)를 넘으면
위와 같은 Error가 발생한다.

Can you get us somehow a stack? maybe this can help narrowing down the real problem.
thx clemens

Path problem when use node manager to start Managed Server

Hi, I have met a problem:
My managed server's root directory is /usr/local/bea/cluster1/usr_projects/mydomain
My Node Manager 's default directory is /usr/local/bea/cluster1/weblogic700/common/nodemanager
I have some config file located under managed server's root directory and log
file is also should be generated in managed server's root directory.
But when I try to use node manager to startup/shutdown the Managed Server, I found
weblogic can not find the config file if I don't move the config file from Managed
server's root directory to node manager server's default directory. and also log
file was created under node manager's default directory.
It seems that managed server's default directory has been changed to node manager's
default directory.
What shall I do if I want Managed Server's default directory to be kept?

Hi Lumin,
If the weblogic version is 8.1, there is a RootDirectory field in Remote
Start tab which should resolve your problem. Just enter the absolute
directory pathname on the machine where you are starting your managed server
and the current working directory of the managed server will be this
RootDirectory. Before 8.1, the RootDirectory was used for finding the config
file for weblogic but it was never used to change the working directory of
the managed server created by the node manager.
cheers,
gaurav.
"lumin" <[email protected]> wrote in message
news:3eb7a4ae$[email protected]..
>
Hi, I have met a problem:
My managed server's root directory is/usr/local/bea/cluster1/usr_projects/mydomain
My Node Manager 's default directory is/usr/local/bea/cluster1/weblogic700/common/nodemanager
>
I have some config file located under managed server's root directory andlog
file is also should be generated in managed server's root directory.
But when I try to use node manager to startup/shutdown the Managed Server,I found
weblogic can not find the config file if I don't move the config file fromManaged
server's root directory to node manager server's default directory. andalso log
file was created under node manager's default directory.
It seems that managed server's default directory has been changed to nodemanager's
default directory.
What shall I do if I want Managed Server's default directory to be kept?

Node Manager not shutting down the Managed Servers

I have WebLogic Server 8.1 SP4 installed on around 9 separate boxes - 3 servers form one WLS domain of one admin + two managed servers. The managed servers are started using Node Manager from the admin console.
On one domain start/shutdown using Admin Console works just fine however the other two domains the managed servers startup fine but shutdown fails - or takes over 15 minutes. I see the following error in the Node Manager logs
NMMessage: Command read failed 'Connection reset' on socket/ <ip_address>
Any ideas on what I can do to fix this ? The servers which will not shutdown have EJB's installed on them. Dont think this could be related as shutting down the application results in the same problem as listed above.
Thanks,
Kevin.

The name of remote machine configured on admin server it must to be resolved on same machine in /etc/hosts:
Example:
On the admin server is configured the unix machine N1234
On the machine N1234 in the file nodemanager.hosts it must be present the IP of admin server
On the admin server in /etc/hosts it must be present
IP_N1234 N1234

Node Manager Not Reachable

Hi Everyone,
I'm very new to WebLogic, so I apologize if this is a dumb question.
I'm trying to install WebLogic as server for OBIEE. My environment is pretty simple. I have two Oracle Red Hat Linux 32-bit physical machines, let's call them WLS_SERVER and OBIEE_SERVER. I installed OBIEE on the OBIEE_SERVER and tested it using the out of the box OC4J container and that worked fine.
Then I installed WebLogic 11g on the WLS_SERVER machine and I ran the config.sh script. I used the configuration GUI to register the two physical machines and an additional server object called BIServer. Out of the box, web logic comes with AdminServer server object. I heard it's a bad practice to deploy applications to the AdminServer, hence why I created the BIServer object. I associated BIServer with the OBIEE_SERVER machine.
I went to start the BIServer and I got the following error:
"For server BIServer, the Node Manager associated with machine OBIEE_SERVER is not reachable."
How do I setup a Node Manager on the OBIEE_SERVER? Do I also need to physical put the WebLogic install file on the OBIEE_SERVER box and also run the installer there? Or there a copy paste thing I do? What am I missing here?
I appreciate the help!
-Joe

I'm trying to install WebLogic as server for OBIEEI have not heard about OBIEE, but if your aim is to use weblogic as a server for hosting this then you definitely need to install weblogic on the physical box where OBIEE will be hosted.
I associated BIServer with the OBIEE_SERVER machine.
I went to start the BIServer and I got the following error:
"For server BIServer, the Node Manager associated with machine OBIEE_SERVER is not reachable."Again not sure what is BIServer but I think you are missing a number of steps here. If you are trying to run BIServer
on weblogic you definitely need to install WLS on this physical machine [OBIEE_SERVER]. You will then need to configure a node manager on OBIEE_SERVER machine to enable it to start remotely from the admin machine.
Link for Node Manager. Since you are in Linux you can try setting up script based node manager.
http://download.oracle.com/docs/cd/E13222_01/wls/docs92/server_start/nodemgr.html#wp1099239

Native IO Disabled when using node manager to start managed server

Hi,
I am able to start my node manager and the managed server from the Admin Console. OS is AIX, WebLogic 9.2 MP3. The managed server starts OK, but Native IO is disabled even though it is enabled in the Admin Console for that manager server:
####<Jul 28, 2009 8:25:00 AM CDT> <Info> <Socket> <pstps09.statefarm.com> <PIA2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1248787500274> <BEA-000447> <Native IO Disabled. Using Java IO.>
I set NativeVersionEnabled to False in nodemanager.properties as specified in the BEA documentation, but can get Native IO to work. The log file clearly indicates that the JDK version is 32-bit:
java.fullversion = J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20071007 (JIT enabled)
Any ideas how to fix this or is this working as designed for AIX platforms?
Thanks!
-Mary Manchukian

The following document asks to set NativeVersionEnabled to False:
http://edocs.bea.com/wls/docs61/adminguide/remotestart.html#1043956:
The Node Manager is available for use only on Windows and UNIX platforms. Native libraries are available for running the Node Manager on Windows, Solaris, HP-UX, AIX and Red Hat Linux operating systems. For UNIX operating systems other than Solaris and HP UX, you will need to use the following argument on the java command line when starting the Node Manager:
-Dweblogic.nodemanager.nativeVersionEnabled=false
If we do not set NativeVersionEnabled to False, the managed server doesn't start at all and fails with the error:
java.io.IOException: Server failed to start up. See server output log for more details.
at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:296)
at weblogic.nodemanager.server.Handler.handleStart(Handler.java:511)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:115)
at weblogic.nodemanager.server.Handler.run(Handler.java:66)
at java.lang.Thread.run(Thread.java:810)
All I want is to be able to start the managed server from the Admin Console > Remote Start tab.
Thanks,
-Mary

RAC node is not starting

I have 2 node RAC instance in 2 virtual machines. My node 2 ASM instance is not starting after the reboot but node 1 is working fine.
I got below output from Node 2.
srvctl status asm
PRCR-1070 : Failed to check if resource ora.asm is registered
Cannot communicate with crsd
alertlog
[/u01/app/11.2.0/grid/bin/cssdagent(3782)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/11.2.0/grid/log/ol5-112-rac2/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2015-03-13 16:38:23.671
[ohasd(3315)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/11.2.0/grid/log/ol5-112-rac2/ohasd/ohasd.log.
2015-03-13 16:38:24.264
[ohasd(3315)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:35.915
[cssd(4326)]CRS-1713:CSSD daemon is started in clustered mode
2015-03-13 16:38:41.195
[cssd(4326)]CRS-1707:Lease acquisition for node ol5-112-rac2 number 2 completed
2015-03-13 16:38:41.259
[cssd(4326)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DISK1; details in /u01/app/11.2.0/grid/log/ol5-112-rac2/cssd/ocssd.log.
2015-03-13 16:38:41.562
[crsd(4285)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/ol5-112-rac2/crsd/crsd.log.
2015-03-13 16:38:42.217
[ohasd(3315)]CRS-2765:Resource 'ora.crsd' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:43.313
[crsd(4357)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/ol5-112-rac2/crsd/crsd.log.
2015-03-13 16:38:44.262
[ohasd(3315)]CRS-2765:Resource 'ora.crsd' has failed on server 'ol5-112-rac2'.
2015-03-13 16:38:44.716
[ohasd(3315)]CRS-2765:Resource 'ora.diskmon' has failed on server 'ol5-112-rac2'.
crsd log
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmo: kgfoCheckMount returned [7]
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmo: The ASM instance is down
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprioo: No OCR/OLR devices are usable
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmcl: asmhandle is NULL
2015-03-13 16:39:02.140: [ OCRRAW][1266976496]proprinit: Could not open raw device
2015-03-13 16:39:02.140: [ OCRASM][1266976496]proprasmcl: asmhandle is NULL
2015-03-13 16:39:02.140: [ OCRAPI][1266976496]a_init:16!: Backend init unsuccessful : [26]
2015-03-13 16:39:02.140: [ CRSOCR][1266976496] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2015-03-13 16:39:02.140: [    CRSD][1266976496][PANIC] CRSD exiting: Could not init OCR, code: 26
2015-03-13 16:39:02.140: [    CRSD][1266976496] Done
Pls help

Hi Levi,
I got below output from cssd.log
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos op : sgipcnTcpConnect
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos dep : No route to host (113)
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos loc : connect
2015-03-15 10:38:48.248: [ GIPCNET][1214789952]gipcmodNetworkProcessConnect: slos info: addr '192.168.1.101:42729'
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssscSelect: conn complete ctx 0x24d7aa0 endp 0x10d4
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmeventhndlr: node(1), endp(0x10d4) failed, probe((nil)) ninf->endp (0x1000010d4) CONNCOMPLETE
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmDiscHelper: ol5-112-rac1, node(1) connection failed, endp (0x10d4), probe(0x100000000), ninf->endp 0x10d4
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmDiscHelper: node 1 clean up, endp (0x10d4), init state 0, cur state 0
2015-03-15 10:38:48.248: [GIPCXCPT][1214789952]gipcInternalDissociate: obj 0x27c8050 [00000000000010d4] { gipcEndpoint : localAddr 'gipc://ol5-112-rac2:de93-de83-5e0c-a373#192.168.1.102#50439', remoteAddr 'gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x8061a, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2015-03-15 10:38:48.248: [GIPCXCPT][1214789952]gipcDissociateF [clssnmDiscHelper : clssnm.c : 3215]: EXCEPTION[ ret gipcretFail (1) ] failed to dissociate obj 0x27c8050 [00000000000010d4] { gipcEndpoint : localAddr 'gipc://ol5-112-rac2:de93-de83-5e0c-a373#192.168.1.102#50439', remoteAddr 'gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x8061a, usrFlags 0x0 }, flags 0x0
2015-03-15 10:38:48.248: [    CSSD][1214789952]clssnmDiscEndp: gipcDestroy 0x10d4
2015-03-15 10:38:48.414: [    CSSD][1147648320]clssnmvDHBValidateNCopy: node 1, ol5-112-rac1, has a disk HB, but no network HB, DHB has rcfg 320087493, wrtcnt, 207342, LATS 355744, lastSeqNo 207342, uniqueness 1426244111, timestamp 1426396128/31164744
2015-03-15 10:38:48.414: [    CSSD][1214789952]clssnmconnect: connecting to addr gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729
2015-03-15 10:38:48.414: [    CSSD][1214789952]clssscConnect: endp 0x10e0 - cookie 0x24d7aa0 - addr gipc://ol5-112-rac1:nm_ol5-112-scan#192.168.1.101#42729
2015-03-15 10:38:48.414: [    CSSD][1214789952]clssnmconnect: connecting to node(1), endp(0x10e0), flags 0x10002
2015-03-15 10:38:48.710: [    CSSD][1181219136]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmRcfgMgrThread: Local Join
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: set curtime (356544) for my node
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: scanning 32 nodes
2015-03-15 10:38:49.209: [    CSSD][1206397248]clssnmLocalJoinEvent: Node ol5-112-rac1, number 1, is in an existing cluster with disk state 3
2015-03-15 10:38:49.210: [    CSSD][1206397248]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
Thanks,
Suranga

11gR2 Cluster node does not start, start of ora.gipcd fails

Hi
I have a clusert node which just does not start, and i do know understand what the problem is.
Does anyone have an idea what the problem cound be?
I am using Oracle 11.2.0.1.0 on OEL.
I appreciate any help, thank you.
Regards Bernhard
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE OFFLINE
ora.crsd
      1        ONLINE INTERMEDIATE node001
ora.cssd
      1        ONLINE OFFLINE
ora.cssdmonitor
      1        ONLINE OFFLINE
ora.ctssd
      1        ONLINE OFFLINE
ora.diskmon
      1        ONLINE OFFLINE
ora.drivers.acfs
      1        ONLINE ONLINE       node001
ora.evmd
      1        ONLINE OFFLINE
ora.gipcd
      1        ONLINE OFFLINE
ora.gpnpd
      1        ONLINE OFFLINE
ora.mdnsd
      1        ONLINE ONLINE       node001                              alertnode001.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:43:52.511
[/opt/grid/product/11gR2/grid/bin/cssdmonitor(5144)]CRS-5822:Agent '/opt/grid/product/11gR2/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-06-29 08:48:49.713
[ohasd(4346)]CRS-2112:The OLR service started on node node001.
2012-06-29 08:48:50.010
[ohasd(4346)]CRS-8017:location: /etc/oracle/lastgasp has 50 reboot advisory log files, 0 were announced and 0 errors occurred
2012-06-29 08:48:55.134
[ohasd(4346)]CRS-2772:Server 'node001' has been assigned to pool 'Free'.
[client(4634)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4652)]CRS-10001:ACFS-9322: done.
2012-06-29 08:50:59.362
[ohasd(4346)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.oracssdmonitor_root.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:48:51.003: [    AGFW][1810312288] Starting the agent: /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/
2012-06-29 08:48:51.003: [   AGENT][1810312288] Agent framework initialized, Process Id = 4414
2012-06-29 08:48:51.003: [ USRTHRD][1810312288] to enter agent main
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: setting priority to 4
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] *** Agent Framework Started ***
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_vmondisv: Compatible vendor clusterware not in use
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_thrdspawn: spawn OMON succ
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: spawn omon succ
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: default values for omon, initrate 1000, pollrate 500.
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: starting omon
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:55.041: [    AGFW][1810312288] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Inited lsf context 0x1ae1cdd0
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Initing CLS Framework messaging
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] New Framework state: 2
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] M2M is starting...
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] m_pClscCtx=0x1ae21d70m_pUgblm=0x1ae754b0
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] Starting send thread
2012-06-29 08:48:55.052: [ CRSCOMM][1119893824] clsIpc: sendWork thread started.
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] IPC Client thread started listening
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] init data sent from server
2012-06-29 08:48:55.057: [CLSFRAME][1810312288] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:node001
2012-06-29 08:48:55.058: [CLSFRAME][1810312288] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:node001
2012-06-29 08:48:55.066: [CLSFRAME][1810312288] Starting thread model named: MultiThread
2012-06-29 08:48:55.073: [CLSFRAME][1810312288] Starting thread model named: SingleThread
2012-06-29 08:48:55.076: [CLSFRAME][1810312288] Starting thread model named: SingleThreadT
2012-06-29 08:48:55.080: [CLSFRAME][1810312288] New Framework state: 3
2012-06-29 08:48:55.080: [    AGFW][1810312288] Agent Framework started successfully
2012-06-29 08:48:55.080: [    AGFW][1182832960] Agfw engine module has enabled...
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] Module Enabling is complete
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] New Framework state: 6
2012-06-29 08:48:55.080: [    AGFW][1182832960] Agent is started with userid: root , expected user: root
2012-06-29 08:48:55.081: [    AGFW][1182832960] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-06-29 08:48:55.088: [    AGFW][1182832960] Agent received the message: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.089: [    AGFW][1182832960] Added new restype: ora.cssdmonitor.type
2012-06-29 08:48:55.089: [    AGFW][1182832960] Agent sending last reply for: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.090: [    AGFW][1182832960] Agent received the message: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.090: [    AGFW][1182832960] Added new resource: ora.cssdmonitor 1 1 to the agfw
2012-06-29 08:48:55.091: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.091: [    AGFW][1182832960] Agent received the message: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.091: [    AGFW][1182832960] Preparing CHECK command for: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [    AGFW][1172343104] Executing command: check for resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_INITRATE, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_POLLRATE, value 500
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr ORA_OPROCD_MODE, value
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr PROCD_TIMEOUT, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr LOGGING_LEVEL, value 1
2012-06-29 08:48:55.096: [    AGFW][1172343104] check for resource: ora.cssdmonitor 1 1 completed with status: OFFLINE
2012-06-29 08:48:55.096: [CRSTIMER][1203812672] Timer Thread Starting.
2012-06-29 08:48:55.096: [    AGFW][1182832960] ora.cssdmonitor 1 1 state changed from: UNKNOWN to: OFFLINE
2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.096: [    AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:28
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent received the message: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent sending last reply for: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.100: [    AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:34
2012-06-29 08:48:55.101: [    AGFW][1182832960] ora.cssdmonitor 1 1 marked as deleted.
2012-06-29 08:48:55.101: [    AGFW][1182832960] Deleting the resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.101: [    AGFW][1182832960] Agent is commiting suicide.
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: CSSD Agent was asked to exit with exit code 1
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_becomeactive: Signaling active state
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_term: still threads up (1)
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: Agent is done.
2012-06-29 08:48:55.101: [    AGFW][1182832960] Agent is exiting with exit code: 1
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29Edited by: Bernhard W on 03-Jul-2012 02:14

I am really getting desperate about this.
The ASM persmissions are on both nodes the same, so this should not be the problem and i can not find any problem on the network.
The log files of the healthy node do not say anything about the node which does not startup now, the last log entry was from 6 months ago, when i shut that node down. There seems to be no communication between the two nodes now.
I tried to manually start the ora.cssd but somehow ora.gipcd just does not start and i can not find the problem.
[oracle@dmrs11srvcla001 ~]$ crsctl stat res -t -init
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE OFFLINE
ora.crsd
      1        ONLINE INTERMEDIATE dmrs11srvcla001
ora.cssd
      1        ONLINE OFFLINE
ora.cssdmonitor
      1        ONLINE ONLINE       dmrs11srvcla001
ora.ctssd
      1        ONLINE OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE ONLINE       dmrs11srvcla001
ora.evmd
      1        ONLINE OFFLINE
ora.gipcd
      1        ONLINE OFFLINE
ora.gpnpd
      1        ONLINE OFFLINE
ora.mdnsd
      1        ONLINE ONLINE       dmrs11srvcla001
[oracle@dmrs11srvcla001 ~]$ crsctl start res ora.cssd -init
CRS-2672: Attempting to start 'ora.gipcd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gipcd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gipcd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gipcd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gpnpd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gpnpd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dmrs11srvcla001'
CRS-2672: Attempting to start 'ora.diskmon' on 'dmrs11srvcla001'
CRS-2676: Start of 'ora.diskmon' on 'dmrs11srvcla001' succeededalertnode001.log
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:40.789
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:41.654
[ohasd(4351)]CRS-2765:Resource 'ora.mdnsd' has failed on server 'node001'.
2012-07-02 17:11:43.404
[cssd(4997)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:11:46.634
[cssd(4997)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:11:46.745
[ohasd(4351)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'node001'.
2012-07-02 17:12:42.384
[/opt/grid/product/11gR2/grid/bin/orarootagent.bin(4488)]CRS-5818:Aborted command 'start for resource: ora.diskmon 1 1' for resource 'ora.diskmon'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-07-02 17:12:46.390
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.diskmon'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:14:50.768
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:55.819
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:58.354
[cssd(5137)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:16:58.425
[cssd(5137)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:25:23.194
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-02 17:25:23.200
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-02 17:26:57.384
[/opt/grid/product/11gR2/grid/bin/cssdagent(5116)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-02 17:27:02.514
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:37:11.971
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.evmd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 02:13:57.821
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:14:37.915
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:44:44.630
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:14:51.303
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:44:57.981
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:15:04.655
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:42:45.015
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:50.068
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:52.369
[cssd(10209)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-03 07:44:52.434
[cssd(10209)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-03 07:45:11.331
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-03 07:54:51.401
[/opt/grid/product/11gR2/grid/bin/cssdagent(10188)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-03 07:54:56.524
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.ohasd.log
2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [    AGFW][1347250496] Received state change for ora.crsd 1 1 [old state = PARTIAL, new state = OFFLINE]
2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23258]
2012-07-03 07:45:11.330: [    AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [   CRSPE][1357756736] State change received from node001 for ora.crsd 1 1
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Processing PE command id=139. Description: [Resource State Change (ora.crsd 1 1) : 0x1c284d40]
2012-07-03 07:45:11.331: [   CRSPE][1357756736] RI [ora.crsd 1 1] new external state [OFFLINE] old value: [INTERMEDIATE] on node001 label = []
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Resource Resource Instance ID[ora.crsd 1 1]. Values:
STATE=OFFLINE
TARGET=ONLINE
LAST_SERVER=node001
CURRENT_RCOUNT=1
LAST_RESTART=1341292505
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=29
STATE_CHANGE_VERS=29
LAST_FAULT=1341292504
DEGREE_ID=1
ID=ora.crsd 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:none
ReadLocks:|STATE INITED||INITIAL CHECK DONE| has failed!
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Processing unplanned state change for [ora.crsd 1 1]
2012-07-03 07:45:11.331: [   CRSPE][1357756736] ora.crsd 1 1: uptime exceeds uptime threshold , resetting restart count
2012-07-03 07:45:11.331: [   CRSPE][1357756736] Local Recovery picked for [ora.crsd 1 1]
2012-07-03 07:45:11.333: [   CRSPE][1357756736] Sending message to agfw: id = 23260
2012-07-03 07:45:11.333: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:11.333: [   CRSPE][1357756736] CRS-2672: Attempting to start 'ora.crsd' on 'node001'
2012-07-03 07:45:11.333: [    AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260 to the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.349: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.350: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.351: [   CRSPE][1357756736] RI [ora.crsd 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on node001 label = []
2012-07-03 07:45:12.351: [   CRSPE][1357756736] CRS-2676: Start of 'ora.crsd' on 'node001' succeeded
2012-07-03 07:45:12.351: [   CRSPE][1357756736] PE Command [ Resource State Change (ora.crsd 1 1) : 0x1c284d40 ] has completed
2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [    AGFW][1347250496] Agfw received reply from PE for resource state change for ora.crsd 1 1
2012-07-03 07:53:17.220: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [    AGFW][1347250496] Received state change for ora.diskmon 1 1 [old state = ONLINE, new state = PLANNED_OFFLINE]
2012-07-03 07:53:17.221: [    AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23497]
2012-07-03 07:53:17.221: [    AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [   CRSPE][1357756736] State change received from node001 for ora.diskmon 1 1
2012-07-03 07:53:17.221: [   CRSPE][1357756736] Processing PE command id=140. Description: [Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0]
2012-07-03 07:53:17.221: [   CRSPE][1357756736] RI [ora.diskmon 1 1] new external state [OFFLINE] old value: [ONLINE] on node001 label = []
2012-07-03 07:53:17.221: [   CRSPE][1357756736] RI [ora.diskmon 1 1] new target state: [OFFLINE] old value: [ONLINE]
2012-07-03 07:53:17.222: [ CRSOCR][1351452992] Multi Write Batch processing...
2012-07-03 07:53:17.222: [   CRSPE][1357756736] Resource Resource Instance ID[ora.diskmon 1 1]. Values:
STATE=OFFLINE
TARGET=OFFLINE
LAST_SERVER=node001
CURRENT_RCOUNT=0
LAST_RESTART=1341294297
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=2
STATE_CHANGE_VERS=5
LAST_FAULT=1341242723
DEGREE_ID=1
ID=ora.diskmon 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:START of [ora.cssd 1 1] on [node001] : 0x2aaab000ba10
ReadLocks:|STATE INITED| has failed!
2012-07-03 07:53:17.222: [   CRSPE][1357756736] Processing unplanned state change for [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [   CRSPE][1357756736] Target is not ONLINE, not recovering [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [   CRSPE][1357756736] PE Command [ Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0 ] has completed
2012-07-03 07:53:17.222: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.222: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.223: [    AGFW][1347250496] Agfw received reply from PE for resource state change for ora.diskmon 1 1
2012-07-03 07:53:17.223: [ CRSOCR][1351452992] Multi Write Batch done.
2012-07-03 07:54:56.523: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.523: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [    AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [   CRSPE][1357756736]Created alert : (:CRSPE00111:) : Start action timed out!
2012-07-03 07:54:56.524: [   CRSPE][1357756736] Start action failed with error code: 3
2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [   CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [    AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [   CRSPE][1357756736] CRS-2674: Start of 'ora.cssd' on 'node001' failed
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
     MESSAGE:
     TextMessage[CRS-2674: Start of 'ora.cssd' on 'node001' failed]
     MSGTYPE:
     TextMessage[1]
     OBJID:
     TextMessage[ora.cssd]
     WAIT:
     TextMessage[0]
2012-07-03 07:54:56.525: [   CRSPE][1357756736] Sending message to agfw: id = 23555
2012-07-03 07:54:56.525: [   CRSPE][1357756736] CRS-2679: Attempting to clean 'ora.cssd' on 'node001'
2012-07-03 07:54:56.525: [    AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:54:56.525: [    AGFW][1347250496] Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is being stopped. Will be restarted when agent stops completly
2012-07-03 07:54:56.525: [    AGFW][1347250496] Could not forward message [RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555] to agent. /opt/grid/product/11gR2/grid/bin/cssdagent_root is not running
2012-07-03 07:54:56.525: [    AGFW][1347250496] Can not start Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is waiting to be restarted.
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
     MESSAGE:
     TextMessage[CRS-2679: Attempting to clean 'ora.cssd' on 'node001']
     MSGTYPE:
     TextMessage[3]
     OBJID:
     TextMessage[ora.cssd]
     WAIT:
     TextMessage[0]
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 14
2012-07-03 07:55:01.628: [ CRSCOMM][1340946752] IPC listener connection to member 14 has been removed
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [   CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:01.628: [    AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:01.628: [    AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:01.628: [    AGFW][1347250496] Restarting the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.628: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10188] stopped!
2012-07-03 07:55:01.628: [ CRSCOMM][1347250496] removeConnection: Member 14 does not exist.
2012-07-03 07:55:01.628: [    AGFW][1347250496] Starting the agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id: root and incarnation:7
2012-07-03 07:55:01.632: [    AGFW][1347250496] Starting the HB [Interval = 30000, misscount = 10kill allowed=0] for agent: /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New IPC Member:{Relative|Node:0|Process:15|Type:3}:AGENT
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New process connected to us ID:{Relative|Node:0|Process:15|Type:3} Info:AGENT
2012-07-03 07:55:01.837: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.837: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root with pid:10530 connected to server.
2012-07-03 07:55:01.837: [    AGFW][1347250496] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [    AGFW][1347250496] Agfw Proxy Server sending message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [    AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.839: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.851: [    AGFW][1347250496] Received the reply to the message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.854: [    AGFW][1347250496] Received the reply to the message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.861: [    AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [    AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.862: [    AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [   CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.862: [    AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.863: [    AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [   CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.863: [    AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.863: [    AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [   CRSPE][1357756736] CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded
2012-07-03 07:55:01.863: [   CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
2012-07-03 07:55:01.863: [UiServer][1361959232] Container [ Name: ORDER
     MESSAGE:
     TextMessage[CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded]
     MSGTYPE:
     TextMessage[3]
     OBJID:
     TextMessage[ora.cssd]
     WAIT:
     TextMessage[0]
2012-07-03 07:55:01.864: [   CRSPE][1357756736] PE Command [ Start Resource : 0x2aaab0009d00 ] has completed
2012-07-03 07:55:01.864: [    AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [   CRSPE][1357756736] UI Command [Start Resource : 0x2aaab0009d00] is replying to sender.
2012-07-03 07:55:01.864: [    AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [UiServer][1361959232] Container [ Name: UI_DATA
     ora.cssd:
     TextMessage[215]
2012-07-03 07:55:01.864: [UiServer][1361959232] Done for ctx=0x2aaab004ece0
2012-07-03 07:55:01.869: [UiServer][1364060480] Closed: remote end failed/disc.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 15
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] IPC listener connection to member 15 has been removed
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.792: [   CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:03.792: [    AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:03.792: [    AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:03.792: [    AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10530] stopped!
2012-07-03 07:55:03.792: [ CRSCOMM][1347250496] removeConnection: Member 15 does not exist.

EPMA Prcess Manager not starting

Hi John,
I had gone thru the document you mentioned in the other post and increased the DimensionServerStartTime from 600 to 1200 and also to 1800 later but i am encountering the following error.
Error 1053: The service did not respond to the start or control request in a timely fashion
Please let me know what to be done

Hyperion EPMA Server service does not start and displays an error "Some services stop automatically if they are not in use by other services or programs" when trying to start the service . This issue occurs only in Windows 2008 server.
The event viewer logs will have the following entry
Service cannot be started. Hyperion.dimensionserver.Interface.Exceptions. EPMAServiceException:Cannot initialize the session Manager. -->System.Net.SocketException:No Connection could be made because the target machine actively refused it 127.0.0.1:5255
Changes
This problem occurs in an IPv6 enabled server.
Cause
When IPv6 is enabled on the EPMA server, the "Hyperion EPMA Server" service starts a process called EPMA_NetJNIBridge which communicates using IPv6. The problem is that the "Hyperion EPMA Server" is only listening using IPv4.
Solution
This issue has been identified as a code bug in 11.1.2.1.00 and is currently scheduled to be fixed in 11.1.2.2.000.
As a temporary solution, add the IPv6 loopback address to the host file on the server.
Example:
::1 localhost
::1 is the IPv6 loopback address. It is the IPv4 equivelent of "127.0.0.1 localhost"
Cheers
John
http://john-goodwin.blogspot.com/

Using Node Manager to start Managed Server in WLS6.1

I get a following exception when i am trying to start a managed server on a
          different host thru the Admin Console and was wondering if anybody can help
          me figure out what is going wrong.
          Thanks
          Kunjan
          <Oct 18, 2001 4:04:26 PM EDT> <Info> <NodeManager@labatt:5555>
          <BaseProcessControl: saving process id of Weblogic Managed server 'nmwls04',
          pid: 21339>
          <Oct 18, 2001 4:04:25 PM EDT> <Error> <NodeManager@labatt:5555>
          <SecureSocketListener: Socket accept failed on labatt:5555>
          Starting WebLogic Server ....
          Connecting to http://localhost:8001...
          The WebLogic Server did not start up properly.
          Exception raised:
          weblogic.management.configuration.ConfigurationException: connecting to
          http://localhost:8001/wl_management_internal2/Bootstrap - with nested
          exception:
          [java.net.ConnectException: Tried all: '1' addresses, but could not
          connect over HTTP to server: 'localhost', port: '8001']
          java.net.ConnectException: Tried all: '1' addresses, but could not
          connect over HTTP to server: 'localhost', port: '8001'
          at weblogic.net.http.HttpClient.openServer(HttpClient.java:203)
          at weblogic.net.http.HttpClient.openServer(HttpClient.java:254)
          at weblogic.net.http.HttpClient.<init>(HttpClient.java:117)
          at weblogic.net.http.HttpClient.New(HttpClient.java:149)
          at
          weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:109)
          at
          weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:27
          9)
          at
          java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:230)
          at weblogic.management.Admin.getBootstrapLocalServer(Admin.java:990)
          at weblogic.management.Admin.initialize(Admin.java:320)
          at weblogic.t3.srvr.T3Srvr.initialize(T3Srvr.java:359)
          at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:202)
          at weblogic.Server.main(Server.java:35)
          --------------- nested within: ------------------
          weblogic.management.configuration.ConfigurationException: connecting
          to http://localhost:8001/wl_management_internal2/Bootstrap - with nested
          exception:
          [java.net.ConnectException: Tried all: '1' addresses, but could not
          connect over HTTP to server: 'localhost', port: '8001']
          at weblogic.management.Admin.getBootstrapLocalServer(Admin.java:1081)
          at weblogic.management.Admin.initialize(Admin.java:320)
          at weblogic.t3.srvr.T3Srvr.initialize(T3Srvr.java:359)
          at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:202)
          at weblogic.Server.main(Server.java:35)
          Reason: Fatal initialization exception
          Child exited


          Make sure you set the listen Address of the Admin server. By default it listens
          on localhost and that's why localhost is sent to the nodemanager. Look in the
          NodeManagerLogs directory at the config file. The entry weblogic.management.server
          shouldn't have localhost, but the host of the admin server.
          andy
          Tom Barnes <[email protected]> wrote:
          >The NodeManager is trying to contact "localhost" to get config info.
          >Try explicitly configuring the host name of the server
          >that the NodeManager uses to bootstrap itself. (Not sure where this is
          >done.)
          >
          >Kunjan Jhaveri wrote:
          >
          >> I get a following exception when i am trying to start a managed server
          >on a
          >> different host thru the Admin Console and was wondering if anybody
          >can help
          >> me figure out what is going wrong.
          >>
          >> Thanks
          >> Kunjan
          >>
          >> <Oct 18, 2001 4:04:26 PM EDT> <Info> <NodeManager@labatt:5555>
          >> <BaseProcessControl: saving process id of Weblogic Managed server 'nmwls04',
          >> pid: 21339>
          >> <Oct 18, 2001 4:04:25 PM EDT> <Error> <NodeManager@labatt:5555>
          >> <SecureSocketListener: Socket accept failed on labatt:5555>
          >> Starting WebLogic Server ....
          >> Connecting to http://localhost:8001...
          >>
          >> ***************************************************************************
          >> The WebLogic Server did not start up properly.
          >> Exception raised:
          >> weblogic.management.configuration.ConfigurationException: connecting
          >to
          >> http://localhost:8001/wl_management_internal2/Bootstrap - with nested
          >> exception:
          >> [java.net.ConnectException: Tried all: '1' addresses, but could
          >not
          >> connect over HTTP to server: 'localhost', port: '8001']
          >> java.net.ConnectException: Tried all: '1' addresses, but could
          >not
          >> connect over HTTP to server: 'localhost', port: '8001'
          >> at weblogic.net.http.HttpClient.openServer(HttpClient.java:203)
          >> at weblogic.net.http.HttpClient.openServer(HttpClient.java:254)
          >> at weblogic.net.http.HttpClient.<init>(HttpClient.java:117)
          >> at weblogic.net.http.HttpClient.New(HttpClient.java:149)
          >> at
          >> weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:109)
          >> at
          >> weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:27
          >> 9)
          >> at
          >> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:230)
          >> at weblogic.management.Admin.getBootstrapLocalServer(Admin.java:990)
          >> at weblogic.management.Admin.initialize(Admin.java:320)
          >> at weblogic.t3.srvr.T3Srvr.initialize(T3Srvr.java:359)
          >> at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:202)
          >> at weblogic.Server.main(Server.java:35)
          >> --------------- nested within: ------------------
          >> weblogic.management.configuration.ConfigurationException: connecting
          >> to http://localhost:8001/wl_management_internal2/Bootstrap - with nested
          >> exception:
          >> [java.net.ConnectException: Tried all: '1' addresses, but could
          >not
          >> connect over HTTP to server: 'localhost', port: '8001']
          >> at weblogic.management.Admin.getBootstrapLocalServer(Admin.java:1081)
          >> at weblogic.management.Admin.initialize(Admin.java:320)
          >> at weblogic.t3.srvr.T3Srvr.initialize(T3Srvr.java:359)
          >> at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:202)
          >> at weblogic.Server.main(Server.java:35)
          >> Reason: Fatal initialization exception
          >>
          >> ***************************************************************************
          >> Child exited
          >

Node manager not starting

Similar Messages

Maybe you are looking for