Cluster node does not shutdown after "received shutdown"

Question

Hi,
We put together an automated restart process that restarts cluster nodes across multiple servers. To shutdown a node, we use the Coherence MBeanConnector and invoke stop on object: name=Management,nodeId=<member id>. This works for most cases where member's log output shows "received shutdown", partition transfer messages and after the last primary partitions have been transferred the VM exits.
For one node however, the VM did not exit. From looking at the log file for this particular node, the primary partitions were transferred, the distributedCache thread stops showing output, but the Cluster thread continues to show activity.
Note that this node was the last VM to stop on the given server.
Has anyone seen this before or ideas on why this particular node did not exit after receiving the shutdown message?
Thanks!
Marcel.

Answer 1

Hi Marcel -
Please take a thread dump (via "kill -3" or "ctrl-break") on the VM that does not stop correctly. Coherence does not shut the VM down; it simply shuts itself down. If a non-daemon thread is running on the VM, then it may not exit. However, we won't know that until we see the thread dump.
Peace,
Cameron Purdy | Oracle Coherence