Nodemanager does not restart managed server after Linux OS reboot

Question

Running Linux 2.6.18, Sun JDK 1.6.0_14, and WebLogic 10.3, we have a correctly functioning nodemanager watching a single managed server. If I kill -9 the managed server, the nodemanager fires up a new one immediately. If, however, we reboot the box, the nodemanager will start cleanly and will not start the managed server. It appears to believe the node should remain down and reports its status as UNKNOWN.
I've seen where this is a known problem on Windows boxes, easily gotten around by supplying the -Xrs parameter to the startup parms for the managed server. (Section 5 of the general nodemanager config doc.) I don't see, however, where there's an equally clean solution for Linux.
Things I've tried:
+ When I set the server to DEBUG, it shows zero sign of receiving a shutdown signal. It's dumping loads of cluster messages, etc, into the log and then suddenly the log just ends so it's dying via kill -9. If the nodemanager is still running at that time, it might try to restart the managed server and log it somewhere as permanently failed.
+ The nodemanager only logs to INFO level, and logs no evidence of deciding to leave the node down. (DEBUG is an invalid LogLevel, so that's all I've got.)
+ A restart signal sent from our AdminServer (on another Linux box) successfully starts the managed node through the nodemanager. Nothing unusual occurs or is logged.
Things I'm curious about:
+ Is there some way to see in what state the nodemanager believes the managed server to be?
+ Is there some value in some doc somewhere that might say the server should not be restarted?
+ Is there not a log of NodeManager client actions? I don't see anything in domain.log or AdminServer.log except when the NM is down.
Thanks,
Kevin
ListenPort=5556
ListenBacklog=50
SecureListener=true
AuthenticationEnabled=true
NativeVersionEnabled=false
CrashRecoveryEnabled=true
JavaHome=/appl/java/jre
StartScriptEnabled=false
StopScriptEnabled=false
StartScriptName=startWebLogic.sh
StopScriptName=
LogFile=/appl/bea/wlserver_10.3/common/nodemanager/logs/nodemanager.log
LogLevel=INFO
LogLimit=0
LogCount=1

Answer 1

can you set NativeVersionEnabled to true. We have faced this in windows which was fixed by changing this property. Some notes from that issue:
Faced this interesting issue in Windows 2008 environment. Managed Servers were started by node manager and it was expected that on a system restart after a crash, the node manager will restore the state of the managed servers to the state it was before the crash. Node Manger was configured to run as a windows service. Hence it started after the server restart. But the individual managed servers were not getting started. Interestingly, for the managed servers started by the node manager there were no <Managed_Server_Name>.lck file in the <Server>/data/NodeManger folder.
To fix this always set the following properties in the nodemanager.properties on the node maager home directory.
CrashRecoveryEnabled=true /* This by default is set to false. This needs to be turned for auto restart after crash to work */
NativeVersionEnabled=true /* This is by default set to true and it should be set as such. If you turn this off on platforms for which weblogic provide native JNI libraries ( e.g. as in Windows 2008) then this wont create the .lck file and hence node manager wont restart the managed server after the crash.
Note: As per official documentation .lck files are required for weblogic node managers to do auto restart after system crash
From Official documentation:
After the system is restarted, Node Manager checks each managed domain specified in the nodemanager.domains file to determine if there are any server instances that were not cleanly shutdown. This is determined by the presence of any lock files which are created by Node Manager when a WebLogic Server process is created. This lock file contains the process identifier for WebLogic Server startup script. If the lock file exists, but the process ID is not running, Node Manager will attempt to automatically restart the server.
If the process is running, Node Manager performs an additional check to access the management servlet running in the process to verify that the process corresponding to the process ID is a WebLogic Server instance.
After the system is restarted, Node Manager checks each managed domain specified in the nodemanager.domains file to determine if there are any server instances that were not cleanly shutdown. This is determined by the presence of any lock files which are created by Node Manager when a WebLogic Server process is created. This lock file contains the process identifier for WebLogic Server startup script. If the lock file exists, but the process ID is not running, Node Manager will attempt to automatically restart the server.If the process is running, Node Manager performs an additional check to access the management servlet running in the process to verify that the process corresponding to the process ID is a WebLogic Server instance.
*/