Failover Cluster testing - not working

Question

Hello
We are trying to perform failover testing on OC4J clusters. We have two nodes clustered. On each node we have installed SOA suite - four OC4J instances (home, oc4J_soa, oc4j_wsm, and oc4j_esbdt).
This is how we are performing the test:
We have deployed servlet on both oc4j_soa with different application names but with same context-root. We have observed following behavior.
1) In mod_oc4j.conf file , we have "Oc4jSelectMethod roundrobin:local".
So when we hit http://<loadbalancer>/<context-root-for-servlet> it works fine in roundrobin fashion. We have set the HTML page title as node name in servlets.
And so we see Node 1 title one time and Node 2 title the other time.
2) Now if I undeploy the servlet from Node 1 and hit http://<loadbalancer>/<context-root-for-servlet> It doen't work one time (gives HTTP 500 error) and it works the other time.
ie. it is still sending one request to Node 1 and other request to Node 2.
well, it should not do this right ? if my application is down/not available on Node 1 , all request should go to Node 2.
3) Also what i have observed is , when we hit the Nodes' http url directly instead of going through the loadbalancer it crisscross the requests.
ie. http://NODE 1/<context-root-for-servlet> - this one always goes to Node 2
and http://NODE 2/<context-root-for-servlet> - always goes to Node 1
This is something weird.
Anybody has any idea , please ? I am not sure why are we getting unexpected behavior mentioned 2) and 3 )
Please let me know if you need anything about config details.
Thanks
/Mishit

Answer 1

2) I hope there is a Hardware LBR front ending this architecture ... If yes then HW LBR have intelligent death detection mechanisms where in if Node 1 crashes it stops serving requests to the failed node until it is back online .... so this setting is more at the LBR then at mod_oc4j.conf
3) If load balancing is configured correctly, I dont think u shud be getting this issue..
To test load balancing you can do as below:
- Ensure Virtual host configuration is done in Apache of both nodes
- Ensure Virtual host entry is added to /etc/hosts
Like on Node 1 /etc/hosts
<Node 2 IP> <Virtual hostname>
Similarly on Node 2 /etc/hosts
<Node 1 IP> < Virtual hostname>
- Give Virtual URL to your LBR guy and he will do the settings in LBR
- Bring down node 1 keeping node 2 alive & check below:
http://<virtualURL>/contextroot
http://<node2>/contextroot
- Similarly bring down node 2 keeping node 1 alive & check below:
http://<virtualURL>/contextroot
http://<node1>/contextroot