Mapping Stuck Threads to Proxy Services
Hello OSB Experts:
Is there a way I can map the information of a thread dump showing stuck threads to the proxy service that those threads are executing?
I am facing a situation that seems to be usual: one of several backends is performing poorly, producing stuck threads in the OSB app. servers. After a while, the performance of the whole OSB is affected because of the number of threads in stuck state.
I would like to be able to pinpoint the backend with problems in order to solve the problem before the health of the whole OSB is affected.
Can this be done without 3rd party tools?
Thanks
Sebastián
No, it is not possible to get the the proxy service name from the thread dump. The OSB stuff is handled under a weblogic wrapper thread.
One of the main reason for stuck threads is some of the proxy service calls taking a lot of time through the business services and there is no appropriate timeout set for the calls on the business service.
You have couple of options to find out the culprit. You can enable tracing on the proxy services and the business services and capture the metrics that should help you find a pattern for the performance. The numbers will help you find which backend is taking more time.
You can get nmore details on OSB Monitoring @ http://docs.oracle.com/cd/E23943_01/admin.1111/e15867/monitoring_ops.htm#OSBAG472
Secondly, always set some realistic timeout value (like 30 seconds/60 seconds depending on the sla of the back-end) on the business service. Leaving it to default (that is 0) is like configuring a indefinite timeout (in which case it uses the default weblogic connection timeout setting of 10/15 minutes).
This way the threads are not stuck waiting for the response and dont cause issues.
Additionally you use OSB alerts to notify you for service behavior discrepancies w.r.t response times, number of errors etc .. Refer http://docs.oracle.com/cd/E23943_01/admin.1111/e15867/monitoring_console.htm#CACDCDAH for more details.
Hope this information helps.
Thanks,
Patrick
It is considered good etiquette to reward answerers with points (as "helpful" - 5 pts - or "correct" - 10pts).+
https://forums.oracle.com/forums/ann.jspa?annID=893
Similar Messages
-
Stuck Thread - JMS Business proxy Service (ALSB 2.5)
Hi All,
We are having a jms proxy service which uses a JMS Business Service to send messages to a jms queue on weblogic 8.1 sp6. The alsb domain runs on weblogic 9.2.The jms proxy service is made single threaded by configuring a suitable work manager.
Now and then, we get a stuck thread issue in production and the proxy service stops consuming the messaging off the jms queue in wls 9.2. These are the errors in the logs:
<[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1235437298516> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "695" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@5321caf", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-70 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, waiting, priority=1, DAEMON> {
-- Waiting for notification on: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
jrockit.vm.Threads.waitForSignal(Native Method)
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Native Method)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:73)
^-- Lock released while waiting: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:100)
weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
weblogic.jms.dispatcher.DispatcherImpl_816_WLStub.dispatchSyncTranFuture(Unknown Source)
weblogic.jms.dispatcher.DispatcherInteropAdapter.dispatchSyncTranFuture(DispatcherInteropAdapter.java:80)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:230)
weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:51)
weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1047)
weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)
weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)
weblogic.jms.client.JMSProducer.send(JMSProducer.java:352)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1009)
weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:157)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:795)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:500)
com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:612)
jrockit.reflect.CompiledMethodInvoker.invoke0(Unknown Source)
jrockit.reflect.CompiledMethodInvoker.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:241)
$Proxy39.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:137)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:437)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:328)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:503)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:415)
stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:106)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:327)
com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:33)
com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:100)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:73)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:47)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:57)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:141)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:646)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:645)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:627)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:264)
com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:108)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:292)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4012)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
jrockit.vm.RNI.c2java(Native Method)
Any one know the root cause of this issue or ways to prevent this?
Also Is there a way to make the proxy service to reconsume messages off the queue without a server restart?
Thanks,
AtheekHi atheek, did you find the solution??
I already got the same/identic issue, do you mind giving me an aid?
Really thanks
Regards,
Mr.K -
Oracle Service Bus "ThreadPool has stuck threads"
Hi all,
I have JMS Proxy service that listen to the queue, with small quantity of message every second, it works great.
But after put message with bigger quantity and faster insertion on queue, the proxy service cannot
pull the message anymore, the error message is "STUCK THREAD", it makes the weblogic console give warning status.
This is the Dump Thread Stacks :
===== FULL THREAD DUMP ===============
Tue Jun 8 11:30:24 2010
BEA JRockit(R) R27.6.1-20-104573-1.6.0_05-20081016-1236-linux-ia32
"Main Thread" id=1 idx=0x4 tid=7168 prio=5 alive, in native, waiting
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:811)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:459)
at weblogic/Server.main(Server.java:67)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=7169 prio=5 alive, in native, daemon
"(GC Main Thread)" id=3 idx=0xc tid=7170 prio=5 alive, in native, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=7171 prio=5 alive, in native, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=7172 prio=5 alive, in native, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=7173 prio=5 alive, in native, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=7174 prio=5 alive, in native, daemon
"(GC Worker Thread 5)" id=? idx=0x20 tid=7175 prio=5 alive, in native, daemon
"(GC Worker Thread 6)" id=? idx=0x24 tid=7176 prio=5 alive, in native, daemon
"(GC Worker Thread 7)" id=? idx=0x28 tid=7177 prio=5 alive, in native, daemon
"(GC Worker Thread 8)" id=? idx=0x2c tid=7178 prio=5 alive, in native, daemon
"(Code Generation Thread 1)" id=4 idx=0x30 tid=7179 prio=5 alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x34 tid=7180 prio=5 alive, in native, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x38 tid=7181 prio=10 alive, in native, daemon
Is there any tuning that I need to do on Jrockit?
Really thanks in advance.
Regards,
Mr.KHi Eric,
Thank you for the link, I'll try for increasing it.
Hi Manoj,
hope it'll give you some additional information for helping me.
####<Jun 8, 2010 10:43:44 AM WIT> <Error> <WebLogicServer> <cspproduction> <AdminServerOSB> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1275968624958> <BEA-000337> <[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "653" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@39c79ac", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-50 "[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: weblogic.rjvm.ResponseImpl@39cf5b1[fat lock]
java.lang.Object.wait(Object.java:???)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:80)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:127)
weblogic.messaging.dispatcher.DispatcherProxy.unmarshalResponse(DispatcherProxy.java:236)
weblogic.messaging.dispatcher.DispatcherProxy.dispatchSyncTranFuture(DispatcherProxy.java:133)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:333)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncNoTran(DispatcherWrapperState.java:384)
weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1151)
weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:679)
weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:541)
weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:366)
weblogic.jms.client.JMSProducer.send(JMSProducer.java:353)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1050)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:107)
weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:235)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.sendMessage(JmsOutboundMessageContext.java:823)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:745)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.security.Security.runAs(Security.java:60)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:502)
com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:680)
sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:56)
$Proxy61.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:136)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:477)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:491)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:405)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:508)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:486)
stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:83)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:343)
com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:31)
com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:140)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:51)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:65)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:178)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:39)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:487)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:248)
com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:79)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:277)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:100)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:148)
$Proxy79.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4526)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4022)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3708)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
That is error message from Oracle service bus console, it stated there is JMS Proxy service that got stuck.
On previous post, I already attached the dump trace, it explained something about jrockit. But I still don't get the way for tuning it properly.
Regards,
Mr.K -
File occasionally stuck in staging on File Poller Proxy Service
Hi All,
I'm having some problems with a File Poller Proxy Service. The Proxy reads xml files in a polling shared directory and puts the message in a queue. The files are small (<1kbytes).
The problem is that occasionally the file gets stuck in the staging directory and never gets to the jms queue. If I cut the file and paste it again in the polling directory, it gets processed fine.
Any ideas?
Thanks.I have one cluster on the web logic server with 4 managed servers in it.
The proxy runs on ms1, and the staging directory is on a different physical server shared folder. -
In OSB 10,3, a Java Callout runs in the same thread of the Proxy Service?
Hello,
a I have a Java Callout within the Proxy Service, request pipeline.
Does it runs in the same thread of the Proxy?
If NOT,
which work manager is the Java Callout pinned to?
can I set the work manager I want?
Thank you,
regards
F.CostaThanks a lot.
Pointer to great info.
However doesn't mention Java Callout threading, nor the documentation does.
Anybody with a YES or NO, please...
F.Costa -
Mapping error code from Business service to Proxy
I am trying to get hands on with 11g and following the book Getting started with Oralce 11G soa suite. The example I have described is based on chapter 11 and chapter 14. I have a proxy service and business service. The business service calls a "ValidateCC" composite which validates the credit card information passed and is proxied by the proxy service. The validation of CC is done by a stored procedure, which returns an error ora-20001 if the credit card info is not found in the database.
Testing PROXY service from OSB console:
Here is the REQUEST message sent to proxy service:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
</soap:Header>
<soapenv:Body>
<cca:creditcardStatusRequest xmlns:cca="http://www.globalcompany.com/ns/CCAuthorizationService">
<cca:CCNumber>2234-1234-1234-1234</cca:CCNumber>
</cca:creditcardStatusRequest>
</soapenv:Body>
</soapenv:Envelope>
Here is the response message received:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<soapenv:Fault>
<faultcode>soapenv:Server</faultcode>
<faultstring>BEA-380001: Internal Server Error</faultstring>
<detail>
<con:fault xmlns:con="http://www.bea.com/wli/sb/context">
<con:errorCode>BEA-380001</con:errorCode>
<con:reason>Internal Server Error</con:reason>
<con:location>
<con:node>RouteTo_validationForCC</con:node>
<con:path>response-pipeline</con:path>
</con:location>
</con:fault>
</detail>
</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>
Testing Business service from OSB console
Response received for the same message:
<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
<env:Header/>
<env:Body>
<env:Fault>
<faultcode>env:Server</faultcode>
<faultstring>
Exception occured when binding was invoked.
Exception occured during invocation of JCA binding: "JCA Binding execute of Reference operation 'validateCC' failed due to: Stored procedure invocation error.
Error while trying to prepare and execute the VALIDATECC API.
An error occurred while preparing and executing the VALIDATECC API. Cause: java.sql.SQLException: ORA-20001: UNKNOWN CREDIT CARD
ORA-06512: at "SOADEMO.VALIDATECC", line 12
ORA-06512: at line 1
The invoked JCA adapter raised a resource exception.
Please examine the above error message carefully to determine a resolution.
</faultstring>
<faultactor/>
<detail>
<exception>
ORA-20001: UNKNOWN CREDIT CARD
ORA-06512: at "SOADEMO.VALIDATECC", line 12
ORA-06512: at line 1
</exception>
</detail>
</env:Fault>
</env:Body>
</env:Envelope>
What i need:
I would like to have the exception 20001 populated in faultcode of the response from proxy message. A down stream BPEL process checks the faultcode and processes the message as needed.
My OSB message flow contains :
ValidateCredit +> envelope
PipelinePairNode1
Request Pipeline
Report and Validate
Response Pipeline
RouteTo_validationForCC
Service Error Handler
stage1 with Alert action.
My question is how to map the actual error occured in the cally be business service handler to the Service error handler in stead of throwing a generic BEA error. Please let me knowThank You Swgt for your reply.
This is what I did:
When I'm in Edit mode, in the OSB Web Console, I go to my proxy service, click on the Edit Message Flow icon, in the Map of Message Flow window I left-click on my Route icon (which as you already mentioned routes to my Business Service), select Add Route Error Handler (this option only appears if you're in Edit Mode already), click on the Error Handler icon, Add Stage, click on the Stage icon, Edit Stage, click on Add an Action, Flow Control -> Reply. Here I select the Radio Button "With Failure" and then Save All and Activate the change for the session.
Is this correct?
I tried this. I no longer get the BEA-380001 error code. However, the fault returned is not a bindingFault as I would have expected, but rather a remoteFault. hence my fault policy never detects it, doesn't rethrow it, and my BPEL error handling never kicks in.
Is there a way to change this to work more or less in the sense of the tutorial?
Does using OSB to connect to the service automatically make it a remote fault, instead of a bindingFault?
Here's the SOAP message handled by my BPEL Component.
<messages>
<input>
<invokeCCStatusService_execute_InputVariable>
<part name="part1">
<creditcardStatusRequest>
<CCNumber>2234-1234-1234-1234</CCNumber>
</creditcardStatusRequest>
</part>
</invokeCCStatusService_execute_InputVariable>
</input>
<fault>
<bpelFault>
<faultType>0</faultType>
<remoteFault>
<part name="summary">
<summary>Beim Aufrufen des Bindings ist eine Exception aufgetreten.
Beim Aufrufen des JCA-Bindings ist eine Exception
aufgetreten: "JCA Binding execute of Reference operation
'validateCC' failed due to: Fehler beim Aufrufen von
gespeicherter Prozedur. Beim Versuch, die VALIDATECC-API
vorzubereiten und auszuführen, ist ein Fehler aufgetreten.
Beim Vorbereiten und Ausführen der VALIDATECC-API ist ein
Fehler aufgetreten. Ursache: java.sql.SQLException:
ORA-20001: UNKNOWN CREDIT CARD ORA-06512: in
"SOADEMO.VALIDATECC", Zeile 12 ORA-06512: in Zeile 1 Prüfen
Sie, ob die API in der Datenbank definiert ist und die
Parameter der API-Signatur entsprechen. Diese Exception wird
als nicht wiederholbar betrachtet. Sie ist wahrscheinlich auf
einen Modellierungsfehler zurückzuführen. Um den Fehler
stattdessen als wiederholbar zu klassifizieren, fügen Sie die
Eigenschaft nonRetriableErrorCodes mit dem Wert "-20001" zum
Deployment-Deskriptor hinzu (d.h. weblogic-ra.xml). Um einen
wiederholbaren Fault automatisch zu wiederholen, legen Sie
folgende Eigenschaften von composite.xml für diesen Aufruf
fest: jca.retry.interval, jca.retry.count und
jca.retry.backoff. Alle Eigenschaften sind Ganzzahlen. ". Der
aufgerufene JCA-Adapter hat eine Ressourcen-Exception
ausgelöst. Prüfen Sie die obige Fehlermeldung sorgfältig, um
eine Lösung zu finden.</summary>
</part>
<part name="detail">
<detail>
<exception>ORA-20001: UNKNOWN CREDIT CARD ORA-06512: in
"SOADEMO.VALIDATECC", Zeile 12 ORA-06512: in Zeile 1</exception>
</detail>
</part>
<part name="code">
<code>{http://schemas.xmlsoap.org/soap/envelope/}Server</code>
</part>
</remoteFault>
</bpelFault>
</fault>
<faultType>
<message>0</message>
</faultType>
</messages> -
Report and Alert don't executed in the error handler with a JMS proxy service
Hi,
I'm working with OSB 11.1.1.4.0 and I'm facing a problem with the error handler of my JMS proxy service.
My error handler contains two main elements :
- a Report action
- an Alert with a JMS destination defined
When the business service failed the message remain in the queue. It's the expected behavior.
The problem is :
- the report is missing
- the alert is raised but missing in the JMS destination defined in the Alert setting.
I've noticed if I add a Reply with Failure or Success, Report and Alert are successfully executed.
Unfortunately the message don't remain in the JMS queue.
What can I do in order to have my report/alert done and the JMS transaction rollback in order to keep the message in the queue ?
I've already met this problem in the past with alsb v3 and my solution was to add Service Callout in the error handler in order to externalize these actions.
This solution is not convenient and I hope another solution is possible.Inventorying workgroups can be difficult, especially when it comes to remote access and network security. Because workgroups are not centrally managed, some of the items discussed in this
wiki article on preparing your workgroup environment may require you to visit each machine individually.
For non-domain credentials, you do not use the <systemname>\<user> format, you simply enter the user name. Regarding how to enter the credentials, if you have an account that uses the same username and password on all machines and is an administrator
on all of those machines, then you can enter that in the All computers credentials page of the wizard. You can also do this if they are different user names. However, if some machines have an account with the same user name, such as Administrator,
but different passwords on each machine, you will need to use the Manually enter computer names discovery method, and then enter the information for each group or each machine.
As you can tell, workgroup environments can quickly negate any benefit that the agentless inventory nature of MAP provides.
Please remember to click "Mark as Answer" on the post that helps you, and to click
"Unmark as Answer" if a marked post does not actually answer your question. Please
VOTE as HELPFUL if the post helps you. This can be beneficial to other community members reading the thread. -
Stuck thread in Oracle Weblogic server
Dear All,
We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
WatchTime: Sep 23, 2011 10:28:29 AM IST
WatchDomainName: dev_domain
WatchServerName: psMS5
WatchSeverityLevel: Warning
WatchName: MS5-Watch
WatchRuleType: Log
WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Content-Length: 554
Cache-Control: no-cache
Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
Connection: Keep-Alive
Proxy-Client-IP: 192.168.9.179
X-Forwarded-For: 192.168.9.179
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 1373322659
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
java.io.File.exists(File.java:729)
weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
*^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
java.lang.Class.getDeclaredConstructors0(Class.java:???)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
java.lang.Class.getConstructor0(Class.java:2699)
java.lang.Class.newInstance0(Class.java:318)
java.lang.Class.newInstance(Class.java:305)
org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 60000
SMTPNotificationName: MS5-Health-NotificationI found something interesting to the question I posted from the below link:
http://www.mail-archive.com/[email protected]/msg03217.html -
OSB: Proxy Service and Dispatch Policy
Hi all,
I'm trying to use OSB (10.3) proxy service with dispatch policy set to WebLogic (10.3) work manager to limit maximum number of threads allocated for request to this proxy service.
It seems to me that whole dispatch policy setting is ignored in OSB. The situation is like this: I have simple Axis based web service with wait method that just waits for few seconds (based on request parameter). I use this service for testing (hm, so far just for trying to understand) OSB dispatch policy function.
Using soapUI I created a simple load test which uses 10 threads to call wait(10) - it means "wait for 10 seconds". Time limit for the whole load test is set to 20 seconds. It is clear that the total execution count is 2 x 10 = 20. So far, so good.
Then I created simple proxy service in OSB that just routes request to business service representing my Axis service with wait method. I set a dispatch policy for the proxy service to WorkManager-2threads (see below) and I expected that running the same load test with endpoint set to OSB would result in significant lower total execution count. I expected that because WebLogic should allocate 2 threads at most for all requests to this proxy service. However, that's not the case as the result is the same as in the first (Axis only) test. Just as there was no dispatch policy settings at all ...
Where is the problem?
This is the relevant part of my WebLogic configuration regarding work manager:
<max-threads-constraint>
<name>MaxThreadsConstraint-2</name>
<target>AdminServer</target>
<count>2</count>
<connection-pool-name></connection-pool-name>
</max-threads-constraint>
<work-manager>
<name>WorkManager-2threads</name>
<target>AdminServer</target>
<max-threads-constraint>MaxThreadsConstraint-2</max-threads-constraint>
<capacity xsi:nil="true"></capacity>
<ignore-stuck-threads>false</ignore-stuck-threads>
</work-manager>It's same problem to me. I do pressure test by loadrunner,I deployed two separate proxy service,under same concurrent user,I get same TPS from the two proxy service.but when I add low priority concurrent user,low priority TPS up.
I set the Route option as you say,but weblogic hanged immediately,and can't be accessed by the console. -
Different workmanager for different proxy service
Hi All,
I want to use two seperate workmanager for my two proxy service, one of them will be ignore stuck thread = true and alternate will be false.
but when I am using them all time its using only one.
I am testing one more scenario, i have another proxy service which is using another Oracle AQ for polling, and i configured the dispatch policy default.
the expected behaviour was to see the stuck thread. but when I monitor the thread list. I am finding the second thread regarding DEQUQUE also comes in same workload manager.
Is there any way to use different workload manager for different services.
Regards,
AbhishekHi Ashish,
I Can give you one more option you no need to create different Material master records .
1. You main tain Material Usage Indicater at order level path is Spro ->IMG> Sales and distribution-->Sales --> Sales documents ---> Sales documents Header ---> Define Usage Indicators.
You can Define Material usage 1.Delux rooms 2.Non Delux rooms.
2. You take field Material Usage in to Price condition table, put the table in the access sequence of Service price condition type and finally maintain condition record with Material Usage key combination.(Here you can maintain different rates for delux and nondelux.
3. Now if you enter Material Usage as Delux while creating order the system will take the price based on the key field, If you enter Non delux it will pick up specific price.
4.You can put Material usage field in incompletionlog
I hope it will Help you
Regard,
Murali. -
Hi,
My application keeps on hanging intermittently and when i check the weblogic output i see fat lock and stuck thread errors. I hope someone can help me analyse and make more sense of the following log, Please
<Aug 19, 2013 12:00:06 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'EPMServer0' has triggered at Aug 19, 2013 12:00:06 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: DATE = Aug 19, 2013 12:00:06 AM EDT SERVER = EPMServer0 MESSAGE = [STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "649" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 649292 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 137
Cache-Control: no-cache
Cookie: JSESSIONID=qk79SRVbLc3870MPGQ8pY6Q1B7GZL6VW2TDKDZw1mgcB03shyGLc!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19626 "[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = SGSINBMPFTAPP1 TXID = CONTEXTID = TIMESTAMP = 1376884806884
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 600000
>
<Aug 19, 2013 12:01:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "709" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 709304 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 137
Cache-Control: no-cache
Cookie: JSESSIONID=qk79SRVbLc3870MPGQ8pY6Q1B7GZL6VW2TDKDZw1mgcB03shyGLc!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19626 "[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:06:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "610" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 610316 ms
POST /HyperionPlanning/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 196
Cache-Control: no-cache
Cookie: JSESSIONID=q8vvSRXLVvKTwD8M3mJhhJKHJ5YNGD0vLz6SgHyzc0TrhC6jV9pT!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19652 "[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.hyperion.css.cache.Cache$ObjMonitor@31c747c2[fat lock]
com.hyperion.css.cache.Cache.getDeltaUpdates(Cache.java:89)
com.hyperion.css.cache.CacheManager.updateDeltaCache(CacheManager.java:964)
com.hyperion.css.facade.CSSFacade.processDeltaUpdate(CSSFacade.java:87)
com.hyperion.css.facade.CSSUserProvisioningFacade.getProvisionedApps(CSSUserProvisioningFacade.java:495)
com.hyperion.planning.security.HspCSS.listAppsForUser(HspCSS.java:127)
com.hyperion.planning.HspSmartViewProvider.listApplications(HspSmartViewProvider.java:1290)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1510)
oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:407)
oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:328)
oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1763)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:223)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:143)
HspSmartViewServlet.doPost(HspSmartViewServlet.java:137)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
HspValidationFilter.doFilter(HspValidationFilter.java:9)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:07:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "670" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 670319 ms
POST /HyperionPlanning/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 196
Cache-Control: no-cache
Cookie: JSESSIONID=q8vvSRXLVvKTwD8M3mJhhJKHJ5YNGD0vLz6SgHyzc0TrhC6jV9pT!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19652 "[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.hyperion.css.cache.Cache$ObjMonitor@31c747c2[fat lock]
com.hyperion.css.cache.Cache.getDeltaUpdates(Cache.java:89)
com.hyperion.css.cache.CacheManager.updateDeltaCache(CacheManager.java:964)
com.hyperion.css.facade.CSSFacade.processDeltaUpdate(CSSFacade.java:87)
com.hyperion.css.facade.CSSUserProvisioningFacade.getProvisionedApps(CSSUserProvisioningFacade.java:495)
com.hyperion.planning.security.HspCSS.listAppsForUser(HspCSS.java:127)
com.hyperion.planning.HspSmartViewProvider.listApplications(HspSmartViewProvider.java:1290)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1510)
oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:407)
oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:328)
oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1763)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:223)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:143)
HspSmartViewServlet.doPost(HspSmartViewServlet.java:137)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
HspValidationFilter.doFilter(HspValidationFilter.java:9)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:09:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 629117 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 136
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19098 "[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:10:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "642" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 642309 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 135
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19657 "[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:10:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "689" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 689128 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 136
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19098 "[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:11:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "702" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 702315 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 135
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19657 "[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:11:06 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'EPMServer0' has triggered at Aug 19, 2013 12:11:06 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: DATE = Aug 19, 2013 12:11:06 AM EDT SERVER = EPMServer0 MESSAGE = [STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "702" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 702315 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 135
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19657 "[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasHi
I see many threads blocked trying to get a database connection "weblogic.jdbc.common.internal.ConnectionPool.reserve(" ...Isuggest you to check connection pools to see if they are overloaded. -
hi we got the stuck thread in OSB domain after some time server goto warinig state, pls help me to resoleve this issue.
####<Apr 29, 2011 3:39:00 PM GMT+01:00> <Error> <WebLogicServer> <bbnjlapp88.online.blockbuster.com> <RentalServicesBus01> <[
STANDBY] ExecuteThread: '124' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1304087940677> <BEA-0
00337> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "621" seconds working
on the request "weblogic.servlet.internal.ServletRequestImpl@1bd0fdd[
POST /bbonline/ExchangeServices HTTP/1.0
Content-Type: text/xml
SOAPAction: ""
User-Agent: Java1.4.2_13
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Content-Length: 1309
Connection: Keep-Alive
Proxy-Client-IP: 10.194.219.78
X-Forwarded-For: 10.194.219.78
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: ijrElMGFX4CCS2HlqAk1Kk0MsiY
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Object.wait(Native Method)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:87)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:127)
weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:108)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
weblogic.transaction.internal.CoordinatorImpl_1030_WLStub.commit(Unknown Source)
weblogic.transaction.internal.TransactionImpl$1.run(TransactionImpl.java:325)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.transaction.internal.SecureAction.runAction(SecureAction.java:71)
weblogic.transaction.internal.TransactionImpl.commit(TransactionImpl.java:321)
weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:243)
weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:230)
weblogic.deployment.jms.WrappedTransactionalSession.delistFromTransaction(WrappedTransactionalSession.java:262)
weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:254)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.sendMessage(JmsOutboundMessageContext.java:880)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:799)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.security.Security.runAs(Security.java:61)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:518)
com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:680)
sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
$Proxy61.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:543)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:478)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:544)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:422)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:583)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:498)
stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128)
com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:339)
com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
com.bea.wli.sb.transports.CoLocatedMessageContext$1.run(CoLocatedMessageContext.java:159)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.security.Security.runAs(Security.java:61)
com.bea.wli.sb.transports.CoLocatedMessageContext.send(CoLocatedMessageContext.java:154)
com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:568)
sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
$Proxy60.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:543)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:478)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:544)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:422)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:583)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:498)
stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128)
com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:339)
com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:279)
com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:277)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.securedInvoke(HttpTransportServlet.java:276)
com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.service(HttpTransportServlet.java:237)
com.bea.wli.sb.transports.http.HttpTransportServlet.service(HttpTransportServlet.java:133)
weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
####<Apr 29, 2011 3:40:00 PM GMT+01:00> <Warning> <Common> <bbnjlapp88.online.blockbuster.com> <RentalServicesBus01> <[ACTIVE
] ExecuteThread: '117' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1304088000926> <BEA-000622> <
Ignoring attempt to release resource "JMSSessionHolder for: null" to pool "ALSB_JMS_SessionPool_-624021009" after the pool ha
s been shutdown.>
####<Apr 29, 2011 3:40:00 PM GMT+01:00> <Warning> <Common> <bbnjlapp88.online.blockbuster.com> <RentalServicesBus01> <[ACTIVE
] ExecuteThread: '67' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1304088000926> <BEA-000622> <I
gnoring attempt to release resource "JMSSessionHolder for: null" to pool "ALSB_JMS_SessionPool_-624021009" after the pool has
been shutdown.>http://download.oracle.com/docs/cd/E17904_01/apirefs.1111/e10660/toc.htm
-
Stuck Threads on weblogic 10.3.2 Holding lock HttpURLConnection
We are having issue with STUCK THREADS on weblogic 10.3.2. Basically my application makes a lot of web service calls over HTTPS and once in a while I see few STUCK THREADS and most of the time they clear but sometimes I have to bounce my server to clean.
I have a thread dump, according to this its holding lock at weblogic.net.http.HttpURLConnection.getInputStream
Any help to resolve this problem is highly appreciated.
ThreadDump:
<May 24, 2011 9:59:08 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@b607d75", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-75 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@bec563c[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:236)
^-- Holding lock: java.io.BufferedInputStream@bec5d9f[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:452)
^-- Holding lock: weblogic.net.http.HttpsClient@bec5202[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:328)
weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
^-- Holding lock: weblogic.net.http.SOAPHttpsURLConnection@bd3b209[thin lock]
weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:939)
com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:213)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:116)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:101)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@bd3afa8[thin lock]
com.sun.xml.ws.client.Stub.process(Stub.java:238)
com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:102)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:116)
$Proxy107.jobSubmit(Unknown Source)
com.somejob.invokeService(JobOperation.java:53)
com.somejobws.operations.ticket.execute(ticket.java:30)
com.somejobutil.Somehelper.process(Somehelper.java:174)
com.somejobutil.Somehelper.processJob(Somehelper.java:147)
com.somejobutil.Somehelper.processMDB(Somehelper.java:91)
com.somejobejb.ProcessMDB.onMessage(ProcessMDB.java:28)
sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy67.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4564)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4060)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3746)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}Yes, we have the same problem, stuck thread on weblogic.net.http.HttpsURLConnection.getResponseCode() (actually implemented by weblogic.net.http.SOAPHttpsURLConnection which also appears to be deprecated ???)
When we use the class javax.net.ssl.HttpsURLConnection the getResponseCode() method works perfectly well, so it looks like the weblogic implementation is to blame.
This is becoming rather urgent now because we're looking to migrate from OC4J to Weblogic and this issue has therefore become a blocker. -
Need help with STUCK Threads. What's causing them?
In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
I also see a note in the trace about something holding a lock:
Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
<Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy309.getComponentSnapShot(Unknown Source)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
>
<Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy309.getComponentSnapShot(Unknown Source)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 600000
>
Edited by: Brutus35 on Aug 5, 2011 8:04 AMmy impression is that you have trouble on the DB,
monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
you can use SQLDeveloper for that
anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash -
Error while invoking web services from Proxy services in OSB
I have a Proxy Service of message type in Oracel Service Bus 10gR3 and it passes the incoming messages from a JMS Queue to Business Service of Web Services type. This Web Service is hosted in Glass Fish v3 and it's binding style is RPC. The JRE used in both the servers are 1.6.0_20. The JMS message is of Text type. The Web Services at the Glass Fish throws MessageCreationException for any content of the JMS message. However, it works fine if Business Service's test client is invoked from the OSB'S Admin Console. However, the same will not work if Proxy Service's test client is called.
The detailed exception stack is given below :
Thanks for the help.
- Raj
com.sun.xml.ws.protocol.soap.MessageCreationException: Couldn't create SOAP message due to exception: XML reader error: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
at com.sun.xml.ws.encoding.SOAPBindingCodec.decode(SOAPBindingCodec.java:359)
at com.sun.xml.ws.transport.http.HttpAdapter.decodePacket(HttpAdapter.java:318)
at com.sun.xml.ws.transport.http.HttpAdapter.access$500(HttpAdapter.java:92)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:501)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:285)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:143)
at org.glassfish.webservices.Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:116)
at org.glassfish.webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:87)
at org.glassfish.webservices.EjbWebServiceServlet.dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
at org.glassfish.webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:127)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:195)
at com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:139)
at com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376)
at com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:329)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:100)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
at java.lang.Thread.run(Thread.java:619)
Caused by: com.sun.xml.ws.streaming.XMLStreamReaderException: XML reader error: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
at com.sun.xml.ws.streaming.XMLStreamReaderUtil.wrapException(XMLStreamReaderUtil.java:267)
at com.sun.xml.ws.streaming.XMLStreamReaderUtil.next(XMLStreamReaderUtil.java:95)
at com.sun.xml.ws.streaming.XMLStreamReaderUtil.nextContent(XMLStreamReaderUtil.java:110)
at com.sun.xml.ws.streaming.XMLStreamReaderUtil.nextElementContent(XMLStreamReaderUtil.java:100)
at com.sun.xml.ws.encoding.StreamSOAPCodec.decode(StreamSOAPCodec.java:175)
at com.sun.xml.ws.encoding.StreamSOAPCodec.decode(StreamSOAPCodec.java:303)
at com.sun.xml.ws.encoding.StreamSOAPCodec.decode(StreamSOAPCodec.java:129)
at com.sun.xml.ws.encoding.SOAPBindingCodec.decode(SOAPBindingCodec.java:354)Let me explain what I am trying to achieve. Our application receives data from multiple sources - from mainframe using FTP, JMS Queue and BPM application using Web Services. The data from all these sources are sending to a Web Service running in GlassFish V3. To handle this I have created three Proxy Services and one Business Service on OSB. The Business Service is of type Web Service using the same WSDL from the target Web Service.
The Proxy Services are configured as follows:
Proxy 1 (type - Messaging service) listens to the JMS Queue and receives JMS Text Message -> Business Service(B1) (Web Service).
Proxy 2 (type - Messaging service Polls files from the FTP Server -> Business Service(B1) (Web Service).
Proxy 3 (type - Web service) is invoked by the BPM application and sends a String -> Business Service(B1) (Web Service).
All the Proxies route to the same Business Service - B1 which calls the Web Service in GlassFish. The only Proxy that works fine is Proxy 3 which is created using the same WSDL of the Web Service. In the other two proxies it thows the same exception in GlassFish as I mentioned before.
com.sun.xml.ws.protocol.soap.MessageCreationException: Couldn't create SOAP message due to exception: XML reader error: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog at [row,col {unknown-source}]: [1,0] at com.sun.xml.ws.encoding.SOAPBindingCodec.decode(SOAPBindingCodec.java:359)
Is this configuration correct to achieve my requirements?
Once again thanks for the help.
- Raj
Maybe you are looking for
-
In Mail 7.3 on my Retina MacBook Pro on OS 10.9.4, how can I type from the keyboard into a mail message using the Apple Symbols font? When I select the Apple Symbols font from the list accessed from under Format/"Show fonts"/collection-all fonts/Appl
-
Since the latest upgrade to 31.0, there are times when I will receive an e-mail, but when I send a reply, I get a response saying the message is undeliverable because of an unknown user, or an unknown host. These are messages, I'm replying to and hav
-
Purchased songs no longer 'authorized' following hard drive crash?
Hello- I recently had a hard drive failure in my PowerBook and had to replace the drive. My iTunes library was backed up to an external drive, but after moving all of my songs to the new one and reinstalling software, etc, I'm now getting a message w
-
Document template - text lines
Hi, I have created a multine virtual attribute which is filled with text lines (from a PO). I want to show these lines in a document template. It works but the lines are shown like this: Line1Line2Line3 In the document template (word document) I want
-
Hello, Suppose you have a JTree of objects and you want to display and edit their data into a separate panel with the same vertical size. Both the JTree and the panel are inside a scrollpane. How to synchronize the panel with the jtree contents ? I w