Weblogic Server 10.0 Workmanager

Question

I configurated in admin server a default work manager called weblogic.kernel.Default with a Maximum Threads
Constraint called MaxThreadsConstraint-0 with value of 300 and I also configurated a Capacity
Constraint with value of 450. The workamanager was configured to ignore stuck threads.
Today, almost every applications were using the workmanager went to unvailable state because the workmanager was
overloaded.
(a) The tab Monitoring -> Threads of one server shows me some interesting things:
Active Execute Threads          - 304
Execute Thread Total           - 311
Execute Thread Idle Count     - 2
Queue Length               - 334
Pending User Request Count     - 150
Hogging Thread Count          - 300
Standby Thread Count          - 7
(b) Meanwhile, the tab Monitoring -> Health of the same server shows me some informations of overloaded wormanager:
[Workmanager:002912] OverloadManager CapacityConstraint-0 rejected request as current length 631 exceeds max capacity of 450]
From explanation above, I have some questions
1) How is accounted the amount of thread (a) ? I understand that
(Execute Thread Total) = (Active Execute Threads) + (Standby Thread Count). How's possible to have
(Execute Thread Total) + (Pending User Request Count) = 311 + 150 = 461 > 540 = MaxThreadsConstraint-0 ?
2) How is possible to have 631 (b) threads? Where's this number come from?
3) Every applications went to unvailable state were deployed in the same cluster. I've two clusters. One cluster have many
applications deployed and the other have only one application deployed. The last one, the deployed application stayed
active while all applications
deployed in the first cluster stayed unvailable. Is there some explication to weblogic to keep to respond requests from the
cluster where the application stayed "available" while the other cluster every application stayed unavailable?

Answer 1

First of all, when you go to monitor tab you see all the threads from self tuning thread pool. This includes all the threads in the server, not only the ones allocated to your workmanager.
This is how i think its working...
1) All your threads are hung..you have 300 hogging threads , that reaches your max thread constraints on your work manager.. weblogic is not going to allow any more requests from that work set to take up threads.. and capacity is both executing and queued requests..you got 150 pending requests and 300 hogging.. so a total of 450, your capacity.
2) Its not 631 threads, its 631 requests, weblogic must be reporting all the threads in the queue at the time you monitored it.
3) The reason for one cluster to go down would be that those servers in that cluster were waiting for some resource. As you have different applications in the two clusters...there must be some resource that cluster1 uses which was not available. you can see what those threads were doing by getting thread dumps when this happens.