Clustered listener gets nothing back from APEX after upgrade to 4.1.1

Question

Hi,
I'm not sure if that subject is entirely accurate, but that appears to be the effect we are experiencing. We have had an SR open for this since April and seem to have the engineers stumped. Maybe someone on here has some insight that may help...
Environment: Seven APEX instances
APEX 4.1.1 (problem started after upgrading from 4.0.x) running on an 11g RAC.
Listener version 1.1.3.243.11.40 running on WebLogic 10.3.2.0
Listeners are in 2x clusters on two different WebLogic domains spread over 4 different servers.
Problem:
In two of the apex instances, a couple of days after upgrading to 4.1.1, the listeners starting returning blank, empty pages to the browser. We went through all the usual debugging steps (/i/ directory, connectivity, etc) and nothing fixed the problem. Finally the only thing that fixes the problem is to shut down one of the two clustered WL managed servers running the listeners, reducing the listener to a single instance per APEX instance. Like clockwork, if the second managed server is brought back up, within minutes-to-hours, the listener starts returning blank pages to the browser. Just a couple of days ago, a third apex instance and its listeners has started to manifest the problem and had to be half shut down. We are beginning to think there is a bug in APEX 4.1.1 that deadlocks the listener (or something) when it is clustered...
Other data points:
The listenerAdmin and listenerStatus pages continue to work and report perfect health for all listeners involved.
The APEX management console (p=4550:1) is also affected by the problem, not just apps.
We ran the "apex_verify" script on all problem instances with no errors.
We have hard rebooted all affected servers, with no change in the problem.
This identical environment was running perfectly stable, clustered for many months on APEX 4.0 (and before).
The only thing that keeps our APEX apps running is to run the listeners single instance (un-clustered).
Thanks in advance for any help that anyone can provide on this very annoying problem!
Brian
Edited by: bpoarnet on Jun 12, 2012 12:51 PM

Answer 1

Udo wrote:
So it's somewhere within your WLS setup.Mmm... I won't discount anything, but I doubt it. If that were the case, how could this exact environment have managed to work flawlessly for the past 3 years on these clusters, the listener being upgraded several times through several versions of apex? Only directly after we upgraded apex (not the listener) to 4.1.1 did our troubles start. My suspicion is that there is a bug or a change in 4.1.1 (or maybe 4.1 as we applied 4.1.1 right after 4.1) that has manifested this problem for us.
>
They definitely are not being used equally, but none of them are being slammed. Usage is fairly light in all cases.... which sounds like if there is some kind of load balancing algorithm different from round robin in your scenario.Ahh, I misinterpreted your question. You were asking if the load is balanced fairly between nodes on the same cluster. I took you to be asking if our separate, unrelated apex instances get the same use. So, the answer then is "yes" hits to a single listener WLS cluster do appear to be getting assigned equally to the different nodes, and the clusters are installed as per the documentation with round-robin scheduling.
####<Jun 11, 2012 11:07:31 AM EDT> <Warning> <Socket> <hostname> <server-name> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <1339427251735> <BEA-000450> <Socket 5 internal data record unavailable (probable closure due idle timeout), event received 17>That could probably have a lot of reasons, but none of them is specific to APEX Listener. Google gives a lot of hits ranging from memory issues to missing classes, and ... duplicate configuration parameters, e.g. app names, jdbc pool names, etc.... and of course, I'm not convinced that this has much to do with the problem at hand. It just happened to be in the log at around the time that someone noticed the problem happening in one case.
Could it be your APEX Listener is deployed multiple times on the same node using the same name?Good thought, but definitely no. We figured out that issue a couple of years ago, back when you had to hack the listener WAR file to make it use a non-temporary config.
Lastly, I just want to thank you again Udo for patiently digging into this. We now have three SR's open with Oracle on this (RAC team, APEX team, and WebLogic team) and we have gotten nowhere yet with any of them...