Hung WebLogic Server in Adobe LiveCycle ES Update 1 (8.2.1)

Question

Hi. Please feel free to move this post to a better forum if it doesn't belong in the WebLogic install forum. I'm posting my findings to help the end user communty and Adobe support staff, so they can better assist users.  I hope the information helps other avoid the problems because this problem took a LONG time to fix with no real insight by vendors.  The solution happened through luck when I upgraded the JDK for better troubleshooting.
Problem Description:
Occasionally in a production environment one of my managed WebLogic servers running LC ES Update 1 (8.2.1) would hang.  A restart of the managed server was required each time it was hung. The restart was happening almost daily which isn't acceptable in an enterprise environment.
Symptoms:
* I would see stalled methods in my client code waiting for LC ES Update 1 to respond when it hangs. I would notice stuck threads in my client code too.
* FormsIVS and OutputIVS were deployed to all managed servers as test/verification tools, but the hung WebLogic instance wouldn't show the IVS page.
* If you visited the WebLogic admin console, then you normally see a RUNNING, OK health state. When the managed WebLogic server hung I would see RUNNING, <blank> status.
* I couldn't click on the hung managed servers Monitoring tab to see what it stuck on.
* I couldn't do garbage collections on the hung server using a WLST thread dumps.
Solution:
Upgrade from WebLogic 9.2 MP3 default JRockit 27.4 (1.5.0_12) to JRockit 28.0 (1.5.0_24). Here are some high level steps on what I had to do:
1. Copied the JRockit JDK to the box
2. Set the JAVA_HOME to the new JRockit JDK location
3. Updated setDomainEnv.cmd, config.xml, commEnv.cmd to the new JAVA_HOME location
4. Shut down all servers (admin, node manager, managed servers)
5. Edit the Server Start classpath that had any reference to the old Java location
6. Reinstall the Windows services to get them to use the new JDK.
Environment/Hardware:
Windows 2003 R2 SP2 (32-bit), Oracle RAC 10.2.0.4
8 Xeon dual core 2.8 GHz CPU, 16 GB RAM
Adobe LiveCycle Environment:
Adobe LiveCycle ES Update 1 (8.2.1) + QF 3.19, QF 3.72, QF 3.76, WebLogic 9.2 MP3 (32-bit), JRockit 27.4 (1.5.0_12) (32-bit)
  * Only had Forms ES, Output ES
WebLogic 9.2 MP3 Configuration:
1 admin server, 1 node manager, 4 managed servers on the same physical box in a vertical cluster

Answer 1

Hi. Please feel free to move this post to a better forum if it doesn't belong in the WebLogic install forum. I'm posting my findings to help the end user communty and Adobe support staff, so they can better assist users.  I hope the information helps other avoid the problems because this problem took a LONG time to fix with no real insight by vendors.  The solution happened through luck when I upgraded the JDK for better troubleshooting.
Problem Description:
Occasionally in a production environment one of my managed WebLogic servers running LC ES Update 1 (8.2.1) would hang.  A restart of the managed server was required each time it was hung. The restart was happening almost daily which isn't acceptable in an enterprise environment.
Symptoms:
* I would see stalled methods in my client code waiting for LC ES Update 1 to respond when it hangs. I would notice stuck threads in my client code too.
* FormsIVS and OutputIVS were deployed to all managed servers as test/verification tools, but the hung WebLogic instance wouldn't show the IVS page.
* If you visited the WebLogic admin console, then you normally see a RUNNING, OK health state. When the managed WebLogic server hung I would see RUNNING, <blank> status.
* I couldn't click on the hung managed servers Monitoring tab to see what it stuck on.
* I couldn't do garbage collections on the hung server using a WLST thread dumps.
Solution:
Upgrade from WebLogic 9.2 MP3 default JRockit 27.4 (1.5.0_12) to JRockit 28.0 (1.5.0_24). Here are some high level steps on what I had to do:
1. Copied the JRockit JDK to the box
2. Set the JAVA_HOME to the new JRockit JDK location
3. Updated setDomainEnv.cmd, config.xml, commEnv.cmd to the new JAVA_HOME location
4. Shut down all servers (admin, node manager, managed servers)
5. Edit the Server Start classpath that had any reference to the old Java location
6. Reinstall the Windows services to get them to use the new JDK.
Environment/Hardware:
Windows 2003 R2 SP2 (32-bit), Oracle RAC 10.2.0.4
8 Xeon dual core 2.8 GHz CPU, 16 GB RAM
Adobe LiveCycle Environment:
Adobe LiveCycle ES Update 1 (8.2.1) + QF 3.19, QF 3.72, QF 3.76, WebLogic 9.2 MP3 (32-bit), JRockit 27.4 (1.5.0_12) (32-bit)
  * Only had Forms ES, Output ES
WebLogic 9.2 MP3 Configuration:
1 admin server, 1 node manager, 4 managed servers on the same physical box in a vertical cluster