Managed server not communicating back to admin server - JMX issue?

Question

Hi,
I have a domain that consisted of:
1) 2 managed servers deploying an EAR for online use on server A
2) 1 managed server being used as a Queue Server for the JMS Module/JMS Server on server A
3) 2 managed servers deploying an EAR for online use on server B
4) 1 admin server to control the domain on server C
Servers A, B & C are running win2k3 and I'm using WLS 9.2.3
We've been shifting our infrastructure and I was tasked with relocating the Queue server to a new win2k8 box, server D. To achieve this 3 options occurred to me (having configured the new machine in my domain):
1) Change the IP address and machine for the Queue Server and then start it up on the new machine
2) Create a new Queue server on the new machine, and re-point the targets of the JMS Server and JMS Module to this new Queue server instead of the old one.
3) Create a new Queue server, new JMS Module and subdeployment and JMS Server, and all the JMS Queues (there are c.25)
I discounted option 3 due to the sheer amount of changes involved and the scope for error. I tested approaches 1 & 2 using a test domain on the new box first and all went well - the JMS queues could be populated and consumed following the migration to the new machine. For Production I went with approach 1 which was simplest.
However, despite the Queue server clearly working on the new machine (we've tested populating and consuming documents at both ends), for some reason it does not report back to the queue server - information such as it's health (e.g. OK, although it does report its state (e.g. RUNNING). Furthermore, the JMS queues no longer report their content - they all simply say "There are no items to display".
Looking in the Queue_Server_1.out file, I can see the following error occurred:
<26-Jun-2011 12:53:52 o'clock BST> <Warning> <Log Management> <BEA-170011> <The LogBroadcaster on this server failed to broadcast log messages to the admin server. The Admin server may not be running. Message broadcasts to the admin server will be disabled.>
<26-Jun-2011 12:55:18 o'clock BST> <Warning> <JMX> <BEA-149509> <Unable to establish JMX Connectivity with the Adminstration Server Administration_Server at service:jmx:admin://20.XXX.XXX.137:9002/jndi/weblogic.management.mbeanservers.domainruntime.
java.io.IOException
     at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:151)
     at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:79)
     at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:326)
     at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:247)
     at weblogic.management.mbeanservers.runtime.internal.RegisterWithDomainRuntimeService.getDomainMBeanServerConnection(RegisterWithDomainRuntimeService.java:199)
     Truncated. see log file for complete stacktrace
javax.naming.CommunicationException [Root exception is java.net.ConnectException: admin://20.XXX.XXX.137:9002: Destination unreachable; nested exception is:
     java.net.ConnectException: Connection timed out: connect; No available router to destination]
     at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:49)
     at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:773)
     at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:365)
     at weblogic.jndi.Environment.getContext(Environment.java:307)
     at weblogic.jndi.Environment.getContext(Environment.java:277)
     Truncated. see log file for complete stacktrace
java.net.ConnectException: admin://20.XXX.XXX.137:9002: Destination unreachable; nested exception is:
     java.net.ConnectException: Connection timed out: connect; No available router to destination
     at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:204)
     at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
     at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:350)
     at weblogic.jndi.Environment.getContext(Environment.java:307)
     at weblogic.jndi.Environment.getContext(Environment.java:277)
     Truncated. see log file for complete stacktrace
java.rmi.ConnectException: Destination unreachable; nested exception is:
     java.net.ConnectException: Connection timed out: connect; No available router to destination
     at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:475)
     at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
     at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
     at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
     at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
     Truncated. see log file for complete stacktrace
Now the IP address of the admin server itself is OK (I've XXX out part of the number as it's my client's machines) but it looks like it's trying to connect using the 'admin' protocol, when it should connect using t3s shouldn't it? Any ideas?

Answer 1

Hi there, thanks for your post. I can confirm that the IP and port can be reached as I've had the networks guys look at this. The issue seems to me to be related to the protocol used to establish the connection.