Resources in UNKNOWN state after failover
Hi,
I'm new to RAC and trying to understand how to bring a node back online. See crsctl output below.
I've noted that the listener, ons and the database on the db01 node are all in an UNKNOWN state. The vip state is INTERMEDIATE.
The cause was activating a virtual NIC on the db01 node. The vNIC was deactivated but not before a failover.
I see oc4j is offline also which I believe is contributing to OEM dbconsole being unnavailable.
Please advise/suggest the best way forward to remedy this.
Mr C
[oracle@db01 ~]$ $ORACLE_HOME/bin/crsctl status resource -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.DATA.dg
ONLINE ONLINE db01
ONLINE ONLINE db02
ora.FRA.dg
ONLINE ONLINE db01
ONLINE ONLINE db02
ora.LISTENER.lsnr
ONLINE UNKNOWN db01
ONLINE ONLINE db02
ora.VOTE.dg
ONLINE ONLINE db01
ONLINE ONLINE db02
ora.asm
ONLINE ONLINE db01 Started
ONLINE ONLINE db02 Started
ora.eons
ONLINE ONLINE db01
ONLINE ONLINE db02
ora.gsd
OFFLINE OFFLINE db01
OFFLINE OFFLINE db02
ora.net1.network
ONLINE ONLINE db01
ONLINE ONLINE db02
ora.ons
ONLINE UNKNOWN db01
ONLINE ONLINE db02
ora.registry.acfs
ONLINE ONLINE db01
ONLINE ONLINE db02
Cluster Resources
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE db02
ora.LISTENER_SCAN2.lsnr
1 ONLINE UNKNOWN db01
ora.LISTENER_SCAN3.lsnr
1 ONLINE UNKNOWN db01
ora.e1oradb.db
*1 OFFLINE UNKNOWN db01*
2 ONLINE ONLINE db02 Open
ora.db01.vip
*1 ONLINE INTERMEDIATE db02 FAILED OVER*
ora.db02.vip
1 ONLINE ONLINE db02
ora.oc4j
*1 OFFLINE OFFLINE*
ora.scan1.vip
1 ONLINE ONLINE db02
ora.scan2.vip
1 ONLINE ONLINE db02
ora.scan3.vip
1 ONLINE ONLINE db02
[oracle@db01 ~]$
Thanks Sebastian,
I was a little busy so only now do I have the time to reply.
Another DBA was able to troubleshoot the root cause of the issue, which I will put here for the benefit of all.
Essentially the hosts file (see snippet below) contained multiple entries for the node name, both on the "localhost" line and also in the "Public" section. A quick edit solved the problem.
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 db02.placeholder.com.au db02 localhost.localdomain
localhost
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
# Public
ww.xx.yy.zzz db01.placeholder.com.au db01
ww.xx.yy.zzz db02.placeholder.com.au db02
# Private
ww.xx.yy.z db01-priv.placeholder.com.au db01-priv
ww.xx.yy.z db02-priv.placeholder.com.au db02-priv
# Virtual
ww.xx.yy.zzz db01-vip.placeholder.com.au db01-vip
ww.xx.yy.zzz db02-vip.placeholder.com.au db02-vip
Mr C
Similar Messages
-
Composites in "Unknown" state after managed soa server restart
Hi,
I am using SOA suite 11.1.1.6 version on 64 bit OEL Linux. I am getting following error in managed server output log files:
Feb 20, 2013 5:59:32 AM IST> <Error> <oracle.integration.platform> <SOA-20003> <Unable to register service.
oracle.fabric.common.FabricException: Error in getting XML input stream: oramds:/deployed-composites/WirelessEnterprise/svcEnrichSEARequest_rev1.0/svcEnrichSEARequest.wsdl: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/deployed-composites/WirelessEnterprise/svcEnrichSEARequest_rev1.0/svcEnrichSEARequest.wsdl does not exist.
Though this composites gets loaded, the composites dependent on this composite goes into "Unknown" state. Other composites are properly loaded on server restart.
Please let us know if anybody has any solution to this problem.Hi,
Thanks for your help.
we have narrowed down to problem now.
The actual problem occurs while redeploying same composite on same revision.
E.g 3 composites A, B and C are deployed on and in "Up" state.
B And C composites depends on A.
B composite composite.xml has following entry
<import namespace="http://xmlns.oracle.com/A"
location="oramds:/deployed-composites/Enterprise/A_rev1.0/A.wsdl"
importType="wsdl"/>
same entry exist for C composite.xml as well.
Now whenever composite A is redeployed with same revision (1.0 in our case), and managed servers are restarted, B and C composites goes in to "Unknown" state.
Moving A.wsdl to /apps/wsdl/* in MDS works but it duplicates the A.wsdl file beacause it also exists in MDS @ /deployed-composites/Enterprise/A_rev1.0/A.wsdl
Please let us know if anyone has any solution.
Thanks,
Rupesh. -
Weblogic servers - Unknown state - Coherence error - OutOfMemory
Dear members,
Weblogic servers are going to unknown state after the failure of ORA-OSB-Deployments(which is used by Oracle Coherence) and OutOfMemory exception.
Server version - 10.3.6.0
Coherence Version - Oracle Coherence 3.7.1.1
Scenario - While retrieving the value from coherence cache via sb protocol.
Error Stack
com.bea.wli.sb.service.resultcache.ResultCacheException: An unexpected exception was thrown while using the result cache: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED) The exception message should provide more detail.
java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)
at com.bea.alsb.coherence.impl.CoherenceCache.get(CoherenceCache.java:83)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:201)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:68)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatchCachedResponse(PipelineContextImpl.java:730)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:611)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
Caused By: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
<Notice> <Diagnostics> <BEA-320068> <Watch 'UncheckedException' with severity 'Notice' on server 'SERVER_NAME' has triggered at 25-Mar-2015 14:04:50 o'clock GMT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
WatchData: DATE = 25-Mar-2015 14:03:55 o'clock GMT SERVER = SERVER_NAME MESSAGE = ExecuteRequest failed
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 4K.
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 4K
at weblogic.utils.io.Chunk.<init>(Chunk.java:318)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:153)
at weblogic.utils.io.ChunkedOutputStream.<init>(ChunkedOutputStream.java:28)
at weblogic.utils.io.ChunkedDataOutputStream.<init>(ChunkedDataOutputStream.java:15)
at weblogic.servlet.internal.ResponseHeaders$1.<init>(ResponseHeaders.java:397)
at weblogic.servlet.internal.ResponseHeaders.writeHeaders(ResponseHeaders.java:397)
at weblogic.servlet.internal.ServletResponseImpl.writeHeaders(ServletResponseImpl.java:1288)
at weblogic.servlet.internal.ServletOutputStreamImpl.sendHeaders(ServletOutputStreamImpl.java:281)
at weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:118)
at weblogic.servlet.internal.ServletOutputStreamImpl.commit(ServletOutputStreamImpl.java:491)
at weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1531)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1498)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Kindly help
Regards
RamSri - Thanks for your response.
Please find below the JVM memory setting of the managed server where the OSB application is deployed
-Xms:2048m -Xmx:2048m -Xns:700m -Xgc:gencon -XX:+UseNewHashFunction
I don't think there is any problem with the JVM settings as it is working in other domains. PFB full stack trace
<25-Mar-2015 13:36:20 o'clock GMT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '248' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "633" seconds working on the request "Workmanager: weblogic.kernel.Default, Version: 1, Scheduled=false, Started=true, Started time: 633240 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-819 "[STUCK] ExecuteThread: '248' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, parked, priority=1, DAEMON> {
java.util.concurrent.locks.LockSupport.park(LockSupport.java:154)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:957)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1278)
java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invoke(SBDispatcherBean.java:118)
com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invokeNoTxWithImmutable(SBDispatcherBean.java:518)
com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.__WL_invoke(Unknown Source)
weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21)
com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.invokeNoTxWithImmutable(Unknown Source)
com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:136)
weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:338)
weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:237)
com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_1036_WLStub.invokeNoTxWithImmutable(Unknown Source)
com.bea.wli.sb.transports.client.impl.SBClientConnection.invoke(SBClientConnection.java:152)
com.bea.wli.sb.transports.sb.SBOutboundConnection.invoke(SBOutboundConnection.java:159)
com.bea.wli.sb.transports.client.impl.SBConnectionBase.send(SBConnectionBase.java:67)
com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper$1.run(SBOutboundMessageContext.java:567)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:142)
weblogic.security.Security.runAs(Security.java:60)
com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper.invoke(SBOutboundMessageContext.java:583)
com.bea.wli.sb.transports.sb.SBOutboundMessageContext.send(SBOutboundMessageContext.java:178)
com.bea.wli.sb.transports.sb.SBTransportProvider.sendMessageAsync(SBTransportProvider.java:1017)
sun.reflect.GeneratedMethodAccessor29293.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:57)
com.sun.proxy.$Proxy127.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:135)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:477)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:444)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:528)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:432)
com.bea.alsb.flow.messaging.dispatcher.servicebus.TransportDispatcherDelegate.send(TransportDispatcherDelegate.java:54)
com.bea.alsb.flow.messaging.dispatcher.servicebus.TransactionalDispatcherDelegate.send(TransactionalDispatcherDelegate.java:47)
com.bea.alsb.flow.messaging.dispatcher.servicebus.ResultCacheDispatcherDelegate.send(ResultCacheDispatcherDelegate.java:47)
com.bea.alsb.flow.messaging.dispatcher.ServiceBusDispatcherHandler.send(ServiceBusDispatcherHandler.java:80)
com.bea.bpel.services.messaging.OutboundDispatcherServiceImpl.send(OutboundDispatcherServiceImpl.java:72)
com.bea.bpel.internal.dispatcher.OutboundDispatcherImpl.dispatch(OutboundDispatcherImpl.java:83)
com.bea.bpel.interpreter.internal.instruction.messaging.InvokeRequest.enter(InvokeRequest.java:63)
com.bea.bpel.interpreter.internal.process.ProcessNodeImpl.enter(ProcessNodeImpl.java:181)
com.bea.bpel.interpreter.internal.Interpreter.interpret(Interpreter.java:141)
com.bea.bpel.interpreter.internal.Interpreter.invoke(Interpreter.java:79)
com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:56)
com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:44)
com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEventJobScheduled(SchedulerServiceContextImpl.java:79)
com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEvent(SchedulerServiceContextImpl.java:65)
com.bea.alsb.flow.scheduler.SchedulerServiceImpl$1.run(SchedulerServiceImpl.java:33)
weblogic.work.ContextWrap.run(ContextWrap.java:39)
com.bea.alsb.platform.weblogic.WlsWorkManagerServiceImpl$WorkAdapter.run(WlsWorkManagerServiceImpl.java:152)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<25-Mar-2015 13:38:13 o'clock GMT> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 4K.
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 4K
at weblogic.utils.io.Chunk.<init>(Chunk.java:318)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:153)
at weblogic.servlet.internal.ChunkOutput.<init>(ChunkOutput.java:114)
at weblogic.servlet.internal.ChunkOutput.create(ChunkOutput.java:160)
at weblogic.servlet.internal.ServletOutputStreamImpl.<init>(ServletOutputStreamImpl.java:102)
Truncated. see log file for complete stacktrace
>
<25-Mar-2015 13:38:36 o'clock GMT> <Error> <OSB Transform> <BEA-382515> <Callout to java method "public static final void com.bt.util.logging.E2EStaticTransaction.startInboundCall(com.bt.util.logging.E2ETransaction)" resulted in exception: null
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor180625.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at stages.transform.runtime.JavaCalloutRuntimeStep$1.run(JavaCalloutRuntimeStep.java:183)
at stages.transform.runtime.JavaCalloutRuntimeStep.processMessage(JavaCalloutRuntimeStep.java:205)
Truncated. see log file for complete stacktrace
Caused By: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [C, size 2224
at com.bt.util.logging.TextFileLogDriver.log(TextFileLogDriver.java:539)
at com.bt.util.logging.LogDirector.logAsync(LogDirector.java:143)
at com.bt.util.logging.LogDirector.log(LogDirector.java:79)
at com.bt.util.logging.LogManager.log(LogManager.java:363)
at com.bt.util.logging.MessagesBundle.log(MessagesBundle.java:69)
Truncated. see log file for complete stacktrace
>
<25-Mar-2015 13:46:44 o'clock GMT> <Warning> <OSB Kernel> <BEA-398300> <Checking result cache for service BusinessService CoherenceCache caused an exception:
com.bea.wli.sb.service.resultcache.ResultCacheException: An unexpected exception was thrown while using the result cache: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED) The exception message should provide more detail.
java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)
at com.bea.alsb.coherence.impl.CoherenceCache.get(CoherenceCache.java:83)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:201)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:68)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatchCachedResponse(PipelineContextImpl.java:730)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:611)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:101)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:597)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:595)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:594)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean$SBRequestTask.run(SBDispatcherBean.java:822)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invoke(SBDispatcherBean.java:282)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invokeNoTxWithImmutable(SBDispatcherBean.java:522)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_1036_WLStub.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.client.impl.SBClientConnection.invoke(SBClientConnection.java:169)
at com.bea.wli.sb.transports.sb.SBOutboundConnection.invoke(SBOutboundConnection.java:165)
at com.bea.wli.sb.transports.client.impl.SBConnectionBase.send(SBConnectionBase.java:81)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper$1.run(SBOutboundMessageContext.java:567)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.security.Security.runAs(Security.java:61)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper.invoke(SBOutboundMessageContext.java:606)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext.send(SBOutboundMessageContext.java:266)
at com.bea.wli.sb.transports.sb.SBTransportProvider.sendMessageAsync(SBTransportProvider.java:1026)
at sun.reflect.GeneratedMethodAccessor29293.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
at com.sun.proxy.$Proxy127.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:445)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:582)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:449)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransportDispatcherDelegate.send(TransportDispatcherDelegate.java:66)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransactionalDispatcherDelegate.send(TransactionalDispatcherDelegate.java:51)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.ResultCacheDispatcherDelegate.send(ResultCacheDispatcherDelegate.java:50)
at com.bea.alsb.flow.messaging.dispatcher.ServiceBusDispatcherHandler.send(ServiceBusDispatcherHandler.java:93)
at com.bea.bpel.services.messaging.OutboundDispatcherServiceImpl.send(OutboundDispatcherServiceImpl.java:73)
at com.bea.bpel.internal.dispatcher.OutboundDispatcherImpl.dispatch(OutboundDispatcherImpl.java:91)
at com.bea.bpel.interpreter.internal.instruction.messaging.InvokeRequest.enter(InvokeRequest.java:90)
at com.bea.bpel.interpreter.internal.process.ProcessNodeImpl.enter(ProcessNodeImpl.java:181)
at com.bea.bpel.interpreter.internal.Interpreter.interpret(Interpreter.java:155)
at com.bea.bpel.interpreter.internal.Interpreter.invoke(Interpreter.java:87)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:66)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:44)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEventJobScheduled(SchedulerServiceContextImpl.java:87)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEvent(SchedulerServiceContextImpl.java:68)
at com.bea.alsb.flow.scheduler.SchedulerServiceImpl$1.run(SchedulerServiceImpl.java:33)
at weblogic.work.ContextWrap.run(ContextWrap.java:41)
at com.bea.alsb.platform.weblogic.WlsWorkManagerServiceImpl$WorkAdapter.run(WlsWorkManagerServiceImpl.java:152)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
com.bea.wli.sb.service.resultcache.ResultCacheException: An unexpected exception was thrown while using the result cache: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED) The exception message should provide more detail.
java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)
at com.bea.alsb.coherence.impl.CoherenceCache.get(CoherenceCache.java:83)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:201)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:68)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatchCachedResponse(PipelineContextImpl.java:730)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:611)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:101)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:597)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:595)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:594)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean$SBRequestTask.run(SBDispatcherBean.java:822)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invoke(SBDispatcherBean.java:282)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invokeNoTxWithImmutable(SBDispatcherBean.java:522)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_1036_WLStub.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.client.impl.SBClientConnection.invoke(SBClientConnection.java:169)
at com.bea.wli.sb.transports.sb.SBOutboundConnection.invoke(SBOutboundConnection.java:165)
at com.bea.wli.sb.transports.client.impl.SBConnectionBase.send(SBConnectionBase.java:81)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper$1.run(SBOutboundMessageContext.java:567)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.security.Security.runAs(Security.java:61)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper.invoke(SBOutboundMessageContext.java:606)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext.send(SBOutboundMessageContext.java:266)
at com.bea.wli.sb.transports.sb.SBTransportProvider.sendMessageAsync(SBTransportProvider.java:1026)
at sun.reflect.GeneratedMethodAccessor29293.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
at com.sun.proxy.$Proxy127.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:445)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:582)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:449)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransportDispatcherDelegate.send(TransportDispatcherDelegate.java:66)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransactionalDispatcherDelegate.send(TransactionalDispatcherDelegate.java:51)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.ResultCacheDispatcherDelegate.send(ResultCacheDispatcherDelegate.java:50)
at com.bea.alsb.flow.messaging.dispatcher.ServiceBusDispatcherHandler.send(ServiceBusDispatcherHandler.java:93)
at com.bea.bpel.services.messaging.OutboundDispatcherServiceImpl.send(OutboundDispatcherServiceImpl.java:73)
at com.bea.bpel.internal.dispatcher.OutboundDispatcherImpl.dispatch(OutboundDispatcherImpl.java:91)
at com.bea.bpel.interpreter.internal.instruction.messaging.InvokeRequest.enter(InvokeRequest.java:90)
at com.bea.bpel.interpreter.internal.process.ProcessNodeImpl.enter(ProcessNodeImpl.java:181)
at com.bea.bpel.interpreter.internal.Interpreter.interpret(Interpreter.java:155)
at com.bea.bpel.interpreter.internal.Interpreter.invoke(Interpreter.java:87)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:66)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:44)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEventJobScheduled(SchedulerServiceContextImpl.java:87)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEvent(SchedulerServiceContextImpl.java:68)
at com.bea.alsb.flow.scheduler.SchedulerServiceImpl$1.run(SchedulerServiceImpl.java:33)
at weblogic.work.ContextWrap.run(ContextWrap.java:41)
at com.bea.alsb.platform.weblogic.WlsWorkManagerServiceImpl$WorkAdapter.run(WlsWorkManagerServiceImpl.java:152)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:205)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:69)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1394)
Truncated. see log file for complete stacktrace
Caused By: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
Truncated. see log file for complete stacktrace
>
<25-Mar-2015 13:47:06 o'clock GMT> <Warning> <OSB Kernel> <BEA-398300> <Checking result cache for service BusinessService CoherenceCache caused an exception:
com.bea.wli.sb.service.resultcache.ResultCacheException: An unexpected exception was thrown while using the result cache: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED) The exception message should provide more detail.
java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)
at com.bea.alsb.coherence.impl.CoherenceCache.get(CoherenceCache.java:83)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:201)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:68)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatchCachedResponse(PipelineContextImpl.java:730)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:611)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:101)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:597)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:595)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:594)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean$SBRequestTask.run(SBDispatcherBean.java:822)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invoke(SBDispatcherBean.java:282)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invokeNoTxWithImmutable(SBDispatcherBean.java:522)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_1036_WLStub.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.client.impl.SBClientConnection.invoke(SBClientConnection.java:169)
at com.bea.wli.sb.transports.sb.SBOutboundConnection.invoke(SBOutboundConnection.java:165)
at com.bea.wli.sb.transports.client.impl.SBConnectionBase.send(SBConnectionBase.java:81)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper$1.run(SBOutboundMessageContext.java:567)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.security.Security.runAs(Security.java:61)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper.invoke(SBOutboundMessageContext.java:606)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext.send(SBOutboundMessageContext.java:266)
at com.bea.wli.sb.transports.sb.SBTransportProvider.sendMessageAsync(SBTransportProvider.java:1026)
at sun.reflect.GeneratedMethodAccessor29293.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
at com.sun.proxy.$Proxy127.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:445)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:582)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:449)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransportDispatcherDelegate.send(TransportDispatcherDelegate.java:66)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransactionalDispatcherDelegate.send(TransactionalDispatcherDelegate.java:51)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.ResultCacheDispatcherDelegate.send(ResultCacheDispatcherDelegate.java:50)
at com.bea.alsb.flow.messaging.dispatcher.ServiceBusDispatcherHandler.send(ServiceBusDispatcherHandler.java:93)
at com.bea.bpel.services.messaging.OutboundDispatcherServiceImpl.send(OutboundDispatcherServiceImpl.java:73)
at com.bea.bpel.internal.dispatcher.OutboundDispatcherImpl.dispatch(OutboundDispatcherImpl.java:91)
at com.bea.bpel.interpreter.internal.instruction.messaging.InvokeRequest.enter(InvokeRequest.java:90)
at com.bea.bpel.interpreter.internal.process.ProcessNodeImpl.enter(ProcessNodeImpl.java:181)
at com.bea.bpel.interpreter.internal.Interpreter.interpret(Interpreter.java:155)
at com.bea.bpel.interpreter.internal.Interpreter.invoke(Interpreter.java:87)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:66)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:44)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEventJobScheduled(SchedulerServiceContextImpl.java:87)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEvent(SchedulerServiceContextImpl.java:68)
at com.bea.alsb.flow.scheduler.SchedulerServiceImpl$1.run(SchedulerServiceImpl.java:33)
at weblogic.work.ContextWrap.run(ContextWrap.java:41)
at com.bea.alsb.platform.weblogic.WlsWorkManagerServiceImpl$WorkAdapter.run(WlsWorkManagerServiceImpl.java:152)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
com.bea.wli.sb.service.resultcache.ResultCacheException: An unexpected exception was thrown while using the result cache: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED) The exception message should provide more detail.
java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)
at com.bea.alsb.coherence.impl.CoherenceCache.get(CoherenceCache.java:83)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:201)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:68)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatchCachedResponse(PipelineContextImpl.java:730)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:611)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:101)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:597)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:595)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:594)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean$SBRequestTask.run(SBDispatcherBean.java:822)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invoke(SBDispatcherBean.java:282)
at com.bea.wli.sb.transports.sb.bean.SBDispatcherBean.invokeNoTxWithImmutable(SBDispatcherBean.java:522)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.sb.transports.sb.bean.SBDispatcher_nwlqik_EOImpl_1036_WLStub.invokeNoTxWithImmutable(Unknown Source)
at com.bea.wli.sb.transports.client.impl.SBClientConnection.invoke(SBClientConnection.java:169)
at com.bea.wli.sb.transports.sb.SBOutboundConnection.invoke(SBOutboundConnection.java:165)
at com.bea.wli.sb.transports.client.impl.SBConnectionBase.send(SBConnectionBase.java:81)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper$1.run(SBOutboundMessageContext.java:567)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.security.Security.runAs(Security.java:61)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext$SBInvokeHelper.invoke(SBOutboundMessageContext.java:606)
at com.bea.wli.sb.transports.sb.SBOutboundMessageContext.send(SBOutboundMessageContext.java:266)
at com.bea.wli.sb.transports.sb.SBTransportProvider.sendMessageAsync(SBTransportProvider.java:1026)
at sun.reflect.GeneratedMethodAccessor29293.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
at com.sun.proxy.$Proxy127.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:445)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:582)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:449)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransportDispatcherDelegate.send(TransportDispatcherDelegate.java:66)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.TransactionalDispatcherDelegate.send(TransactionalDispatcherDelegate.java:51)
at com.bea.alsb.flow.messaging.dispatcher.servicebus.ResultCacheDispatcherDelegate.send(ResultCacheDispatcherDelegate.java:50)
at com.bea.alsb.flow.messaging.dispatcher.ServiceBusDispatcherHandler.send(ServiceBusDispatcherHandler.java:93)
at com.bea.bpel.services.messaging.OutboundDispatcherServiceImpl.send(OutboundDispatcherServiceImpl.java:73)
at com.bea.bpel.internal.dispatcher.OutboundDispatcherImpl.dispatch(OutboundDispatcherImpl.java:91)
at com.bea.bpel.interpreter.internal.instruction.messaging.InvokeRequest.enter(InvokeRequest.java:90)
at com.bea.bpel.interpreter.internal.process.ProcessNodeImpl.enter(ProcessNodeImpl.java:181)
at com.bea.bpel.interpreter.internal.Interpreter.interpret(Interpreter.java:155)
at com.bea.bpel.interpreter.internal.Interpreter.invoke(Interpreter.java:87)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:66)
at com.bea.bpel.internal.dispatcher.EngineManagerImpl.dispatch(EngineManagerImpl.java:44)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEventJobScheduled(SchedulerServiceContextImpl.java:87)
at com.bea.bpel.internal.dispatcher.SchedulerServiceContextImpl.fireEvent(SchedulerServiceContextImpl.java:68)
at com.bea.alsb.flow.scheduler.SchedulerServiceImpl$1.run(SchedulerServiceImpl.java:33)
at weblogic.work.ContextWrap.run(ContextWrap.java:41)
at com.bea.alsb.platform.weblogic.WlsWorkManagerServiceImpl$WorkAdapter.run(WlsWorkManagerServiceImpl.java:152)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
at com.bea.wli.sb.service.resultcache.ResultCache.get(ResultCache.java:205)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.checkCache(ServiceResultCacheImpl.java:148)
at com.bea.wli.sb.service.resultcache.ServiceResultCacheImpl.processRequest(ServiceResultCacheImpl.java:69)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:560)
at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1394)
Truncated. see log file for complete stacktrace
Caused By: java.lang.RuntimeException: Failed to start Service "ORA-OSB-deployments" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)
at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)
Truncated. see log file for complete stacktrace
>
<25-Mar-2015 14:05:06 o'clock GMT> <Notice> <Diagnostics> <BEA-320068> <Watch 'UncheckedException' with severity 'Notice' on server 'SERVER_NAME' has triggered at 25-Mar-2015 14:04:50 o'clock GMT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
WatchData: DATE = 25-Mar-2015 14:03:55 o'clock GMT SERVER = SERVER_NAME MESSAGE = ExecuteRequest failed
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 4K.
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 4K
at weblogic.utils.io.Chunk.<init>(Chunk.java:318)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:153)
at weblogic.utils.io.ChunkedOutputStream.<init>(ChunkedOutputStream.java:28)
at weblogic.utils.io.ChunkedDataOutputStream.<init>(ChunkedDataOutputStream.java:15)
at weblogic.servlet.internal.ResponseHeaders$1.<init>(ResponseHeaders.java:397)
at weblogic.servlet.internal.ResponseHeaders.writeHeaders(ResponseHeaders.java:397)
at weblogic.servlet.internal.ServletResponseImpl.writeHeaders(ServletResponseImpl.java:1288)
at weblogic.servlet.internal.ServletOutputStreamImpl.sendHeaders(ServletOutputStreamImpl.java:281)
at weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:118)
at weblogic.servlet.internal.ServletOutputStreamImpl.commit(ServletOutputStreamImpl.java:491)
at weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1531)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1498)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
SUBSYSTEM = Kernel USERID = <WLS Kernel> SEVERITY = Error THREAD = [STUCK] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000802 MACHINE = blp10865069 TXID = CONTEXTID = 582d5d390322c8f7:-346f60f0:14ae3e7dc81:-7ffd-0000000000897f09 TIMESTAMP = 1427292235779
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000
Many thanks in advance
Ram -
Server 2012 File Server Cluster Shadow Copies Disappear Some Time After Failover
Hello,
I've seen similar questions posted on here before however I have yet to find a solution that worked for us so I'm adding my process in hopes someone can point out where I went wrong.
The problem: After failover, shadow copies are only available for a short time on the secondary server. Before the task to create new shadow copies happens the shadow copies are deleted. Failing back shows them missing on the primary server as
well when this happens.
We have a 2 node (hereafter server1 and server2) cluster with a quorum disk. There are 8 disk resources which are mapped to the cluster via iScsi. 4 of these disks are setup as storage and the other 4 are currently set up as shadow copy volumes
for their respective storage volume.
Previously we weren't using separate shadow copy volumes and seeing the same issue described in the topic title. I followed two other topics on here that seemed close and then setup the separate shadow copy volumes however it has yet to alleviate the
issue. These are the two other topics :
Topic 1: https://social.technet.microsoft.com/Forums/windowsserver/en-US/ba0d2568-53ac-4523-a49e-4e453d14627f/failover-cluster-server-file-server-role-is-clustered-shadow-copies-do-not-seem-to-travel-to?forum=winserverClustering
Topic 2: https://social.technet.microsoft.com/Forums/windowsserver/en-US/c884c31b-a50e-4c9d-96f3-119e347a61e8/shadow-copies-missing-after-failover-on-2008-r2-cluster
After reading both of those topics I did the following:
1) Add the 4 new volumes to the cluster for shadow copies
2) Made each storage volume dependent on it's shadow copy volume in FCM
3) Went to the currently active node directly and opened up "My Computer", I then went to the properties of each storage volume and set up shadow copies to go to the respective shadow copy volume drive letter with correct size for spacing, etc.
4) I then went back to FCM and right clicked on the corresponding storage volume and choose "Configure Shadow Copy" and set the schedule for 12:00 noon and 5:00 PM.
5) I noticed that on the nodes the task was created and that the task would failover between the nodes and appeared correct.
6) Everything appears to failover correctly, all volumes come up, drive letters are same, shadow copy storage settings are the same, and 4 scheduled tasks for shadow copy appear on the current node after failover.
Thinking everything was setup according to best practice I did some testing by changing file contents throughout the day making sure that previous versions were created as scheduled on server1. I then rebooted Server1 to simulate failure. Server2
picked up the role within about 10 seconds and files were avaiable. I checked and I could still see previous versions for the files after failover that were created on server1. Unfortunately that didn't last as the next day before noon I was going
to make more changes to files to ensure that not only could we see the shadow copies that were created when Server1 owned the file server role but also that the copies created on Server2 would be seen on failback. I was disappointed to discover that
the shadow copies were all gone and failing back didn't produce them either.
Does anyone have any insight into this issue? I must be missing a switch somewhere or perhaps this isn't even possible with our cluster type based on this: http://technet.microsoft.com/en-us/library/cc779378%28v=ws.10%29.aspx
Now here's an interesting part, shadow copies on 1 of our 4 volumes have been retained from both nodes through the testing, but I can't figure out what makes it different though I do suspect that perhaps the "Disk#s" in computer management / disk
management perhaps need to be the same between servers? For example, on server 1 the disk #s for cluster volume 1 might be "Disk4" but on server 2 the same volume might be called "Disk7", however I think that operations like this
and shadow copy are based on the disk GUID and perhaps this shouldn't matter.
Edit, checked on the disk numbers, I see no correlation between what I'm seeing in shadow copy and what is happening to the numbers. All other items, quotas, etc fail and work correctly despite these diffs:
Disk Numbers on Server 1:
Format: "shadow/storerelation volume = Disk Number"
aHome storage1 = 16
aShared storage2 = 09
sHome storage3 = 01
sShared storage4 = 04
aHome shadow1 = 10
aShared shadow2 = 11
sHome shadow3 = 02
sShared shadow4 = 05
Disk numbers on Server 2:
aHome storage1 = 16 (SAME)
aShared storage2 = 04 (DIFF)
sHome storage3 = 05 (DIFF)
sShared storage4 = 08 (DIFF)
aHome shadow1 = 10 (SAME)
aShared shadow2 = 11 (SAME)
sHome shadow3 = 06 (DIFF)
sShared shadow4 = 09 (DIFF)
Thanks in advance for your assistance/guidance on this matter!Hello Alex,
Thank you for your reply. I will go through your questions in order as best I can, though I'm not the backup expert here.
1) "Did you see any event ID when the VSS fail?
please offer us more information about your environment, such as what type backup you are using the soft ware based or hard ware VSS device."
I saw a number of events on inspection. Interestingly enough, the event ID 60 issues did not occur on the drive where shadow copies did remain after the two reboots. I'm putting my event notes in a code block to try to preserve formatting/readability.
I've written down events from both server 1 and 2 in this code block, documenting the first reboot causing the role to move to server 2 and then the second reboot going back to server 1:
JANUARY 2
9:34:20 PM - Server 1 - Event ID: 1074 - INFO - Source: User 32 - Standard reboot request from explorer.exe (Initiated by me)
9:34:21 PM - Server 1 - Event ID: 7036 - INFO - Source: Service Control Manager - "The Volume Shadow Copy service entered the running state."
9:34:21 PM - Server 1 - Event ID: 60 - ERROR - Source: volsnap - "The description for Event ID 60 from source volsnap cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
\Device\HarddiskVolumeShadowCopy49
F:
T:
The locale specific resource for the desired message is not present"
9:34:21 PM - Server 1 - Event ID 60 - ERROR - Source: volsnap - "The description for Event ID 60 from source volsnap cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
\Device\HarddiskVolumeShadowCopy1
H:
V:
The locale specific resource for the desired message is not present"
***The above event repeats with only the number changing, drive letters stay same, citing VolumeShadowCopy# numbers 6, 13, 18, 22, 27, 32, 38, 41, 45, 51,
9:34:21 PM - Server 1 - Event ID: 60 - ERROR - Source: volsnap - "The description for Event ID 60 from source volsnap cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
\Device\HarddiskVolumeShadowCopy4
E:
S:
The locale specific resource for the desired message is not present"
***The above event repeats with only the number changing, drive letters stay same, citing VolumeShadowCopy# numbers 5, 10, 19, 21, 25, 29, 37, 40, 46, 48, 48
9:34:28 PM - Server 1 - Event ID: 7036 - INFO - Source: Service Control Manager - "The NetBackup Legacy Network Service service entered the stopped state."
9:34:28 PM - Server 1 - Event ID: 7036 - INFO - Source: Service Control Manager - "The Volume Shadow Copy service entered the stopped state.""
9:34:29 PM - Server 1 - Event ID: 7036 - INFO - Source: Service Control Manager - "The NetBackup Client Service service entered the stopped state."
9:34:30 PM - Server 1 - Event ID: 7036 - INFO - Source: Service Control Manager - "The NetBackup Discovery Framework service entered the stopped state."
10:44:07 PM - Server 2 - Event ID: 7036 - INFO - Source: Service Control Manager - "The Volume Shadow Copy service entered the running state."
10:44:08 PM - Server 2 - Event ID: 7036 - INFO - Source: Service Control Manager - "The Microsoft Software Shadow Copy Provider service entered the running state."
10:45:01 PM - Server 2 - Event ID: 48 - ERROR - Source: bxois - "Target failed to respond in time to a NOP request."
10:45:01 PM - Server 2 - Event ID: 20 - ERROR - Source: bxois - "Connection to the target was lost. The initiator will attempt to retry the connection."
10:45:01 PM - Server 2 - Event ID: 153 - WARN - Source: disk - "The IO operation at logical block address 0x146d2c580 for Disk 7 was retried."
10:45:03 PM - Server 2 - Event ID: 34 - INFO - Source: bxois - "A connection to the target was lost, but Initiator successfully reconnected to the target. Dump data contains the target name."
JANUARY 3
At around 2:30 I reboot Server 2, seeing that shadow copy was missing after previous failure. Here are the relevant events from the flip back to server 1.
2:30:34 PM - Server 2 - Event ID: 60 - ERROR - Source: volsnap - "The description for Event ID 60 from source volsnap cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
\Device\HarddiskVolumeShadowCopy24
F:
T:
The locale specific resource for the desired message is not present"
2:30:34 PM - Server 2 - Event ID: 60 - ERROR - Source: volsnap - "The description for Event ID 60 from source volsnap cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
\Device\HarddiskVolumeShadowCopy23
E:
S:
The locale specific resource for the desired message is not present"
We are using Symantec NetBackup. The client agent is installed on both server1 and 2. We're backing them up based on the complete drive letter for each storage volume (this makes recovery easier). I believe this is what you would call "software
based VSS". We don't have the infrastructure/setup to do hardware based snapshots. The drives reside on a compellent san mapped to the cluster via iScsi.
2) "Confirm the following registry is exist:
- HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VSS\Settings"
The key is there, however the DWORD value is not, would that mean that the
default value is being used at this point? -
How to restore standby from primary after failover.
HI we have a dr setup but the standby is provided archives through an application not the datagaurd.
So for applying application patch I cant switchover the database.
I have to shutdown the primary database and apply failover to the standby database to apply patch.
Can you please tell how can I restore my standby database to actual primary after failover.
As our actual activity will be in 2 weeks time. We want to apply patch on saturday.
DR DRILL is on 8th nov.Your earlier statement "Can you please tell how can I restore my standby database to actual primary after failover"
means something different from "SO have to restore from cold backup of primary database"
What you seem to want is a procedure to recreate the Standby from a Backup of the Pirmary. (ie FROM Primary TO Standby).
You haven't specified your Oracle Version. DG has been available since 9i (was even backported to 8i) but I will presume that you are using 9i
You should follow the same process as used for the first creation of the standby, except that you don't have to repeat some of the setup steps (listener, tnsnames etc)
http://download.oracle.com/docs/cd/B10501_01/server.920/a96653/create_ps.htm#63563
Hemant K Chitale
http://hemantoracledba.blogspot.com -
Hello All,
I have a 2 node RAC cluster running on Oracle 11G on Windows 2003. Everything appears to be working fine, but I have some services in an unknown state.
If I stop all services using CRSCTL STOP CRS on any node, all services go offline as expected and OEM shows the fact that one of the nodes (instances) is no longer available.
If I then issue CRSCTL START CRS on any node all services start up OK except for the .INST and .DB services. Issuing the CRS_STAT -T the .INST service shows as 'online' but its state is 'unknown'. The .DB service for the database shows as 'offline'.
The stopped DB Instance does not start up automatically. If I try to start it using SRVCTL I get a CRS-0223 'Placement' error message . To get my stopped DB Instance to startup the only way I have found is to log on to sqlplus as SYS and issue the startup command. The instance then starts up with no problems
I'm confused as to why the above mentioned .INST and .DB services dont appear to be in a correct state yet my RAC cluster appears to function correctly. Any advice on this would be greatly appreciated .
Thanks,
ShaunI wouldn't use crsctl stop crs to stop the services, that shutdown the entire clusterware environment. Use "crs_stop -all" instead.
As for your issue, did you check the alert log just after clusterware tries to start up the db?
Check the alert log on the databases that are not coming up automatically. There's probably an error in there. -
Agent going to ready state after login
Hi guys,
I have UCCX 5.0(2) and I would like to force agent to go to state ready after login. There were some posts on this topic and they suggested to play with workflow:
1. add add action agent ready at startup - for this I get: "Error on Startup Event. Error Message: An Agent State Action may only be used on Answered and Dropped Events."
2. add macro - record putting agent in ready state (keystroke Ctrl +W). This I can do, but agent gets error during login: " Could not execute Macro Action. Unable to playback to windows program."
Did you have similar problems? Did you solve it and how? Or maybe you have some other idea how to force agent to go in ready state after login?
BR,
JelenaHi,
Problem An agent using Windows XP was able to start Agent Desktop, but was
not able to enter an active state.
Solution Windows XP can be configured so that the Internet Connection Firewall
(ICF) is active. ICF acts by keeping track of all traffic to and from the
computer; it will only allow information through that has originated from
that particular computer. If a message originates from outside the
computer, it will be discarded.
To solve this problem, either turn off ICF (requires someone with
administrator rights to the computer) or override the defaults to include
known “good” connections like the CAD servers.
And also, is it by any chance happening during a failover or failback scenario in a High Availability (HA) setup?
If its a HA setup, did you install this CAD after the HA setup is made i.e. the CAD has the entries for the ip address of the both the nodes.
Hope it helps,
Anand -
Cluster services UNKNOWN state
Hi,
I am having two node cluster database. I have some doubt
If cluster services will go UNKNOWN state in first node existing connection will failover to second node?
New connections will try to connect first node?user2017273 wrote:
Hi,
I am having two node cluster database. I have some doubtQuit doubting and TEST it for yourself. Also actually reading the documentation will help
>
If cluster services will go UNKNOWN state in first node existing connection will failover to second node?
Maybe...
New connections will try to connect first node?If nodex is down any connection attempt should go to the remaining nodes. -
New weblogic instances goto unknown state
Hi ,
When i create a new weblogic server using Admin console , the server at time shows up in UNKNOWN State, where as normal behaviour is to show in SHUTDOWN state. have you seen or know about this kind of behaviour before? The problem with this unknown state , is once i create and change properties , i cannot able to start the server from Admin Console.
But when i restart the admin server, these new servers goto "shutdown" state, after which i could able to start the servers fine.
But i had to restart admin server everytime i create a new server , which ends up in unknown state.
im using WLS 10.3.6 version with node manager configured and jvm args are -Xmn 256m -Xmx512m respectively, let me know if you need more info on this
Thanks in advance,
PatraHi ,
sorry for the late response.
FlyingBird - here are my answers to your questions.
I created managed servers - Admin server is already created long back.
yes,, both startWeblogic.sh and startNodeManager.sh are started. I could able to see the node manager is reachable from admin console and node manager process has started fine on the node.
Sharmela - I dont see any error reported.
Since the new managed servers that are created , are not started up yet, ( i.e. i could not able to start them as they end up in UNKNOWN state ), they does not have the directory structure created in the linux box. so , there is no <MSX>/logs dir created so , im clues less on what is going on.
i have like more than 60 managed servers on this admin console. this issue was not there when there were less managed servers ( ~ 20 but happens everytime i create a new managed server. Not sure, what is the issue. Hard part is , i dont find any logs for it to find a direction in troubleshooting. Appreciate your advises.
Let me know if you need more info.
Regards
PatraD -
Server for NFS deletes the partially copied file after failover
Hello,
I found the strange behaviour in Server for NFS in the microsoft foc cluster.
I have a foc cluster of two nodes(windows server 2k12r2) and server for nfs role is installed on both. now I did mount the export from the client(windows server 2k12r2) where client for nfs role is installed;
and a large file copying from the clients local drive to the mounted export is started.
Now during this copying operation I did switch the export to second node(moved it using the failover cluster manager wizard). and what I found is the copying interrupts with
error message "there is problem accessing file path" and on retry the copying starts all over again, I mean if the 50% was copied previously then
after retry it will start from 0%.
I watched the system events on second node with the "procmon" and I found; after the disk gets attached to the second node, the delete operation is get called for this file.
Now what I need to know is why this delete is been called and who made this called. In the stack of the delete operation I could track the nfssvr.sys but not sure what attributes of that file made it to call the
delete.Hi,
It takes us some time to create a failover cluster for testing.
We tried to copy a large folder which contains many files, and a large VHD file we created.
The result is the same - it will not restart the copy after failover. Thus as you said it is not caused by design and it should be an issue on your side which causes a Delete and redo the copy.
Is there any application or specific network settings could terminate a disconnected copy process? Currently I do not have much information about this.
Edit: I noticed that it is NFS share and in our test we are using SMB share. Will do the test again to see if any difference. -
Call remains in "conferenced" state after the call is released
We have a scenario where a lengthy recorded message is set up through a ICM call routing script. If we call the number we assigned to the script using our hard phones the call is disconnected as soon as the recorded message is finished playing.
When we use Cisco Administrator Desktop and the conference feature, the call remains in the "conferenced" state after the call has been released.
My assumption was that our reports would be smarter than CAD and would get it right. However, reporting agrees with CAD, it thinks the call remained on a conference call for the whole time.
In the script we set some variables for the call, play the message through "RunExtScript" node, set a call type, than use the "ReleaseCall" node to disconnect the call. Any ideas how we can configure the script other than this?Hi,
can you tell us more - do you have Translation Routing set up, perhaps?
Also, can you take a look at the Router Log (on the Distributor server).
And please swith the script to Monitoring mode to see where it fails.
G. -
Replica nodes going in 'UNKNOWN' state
Hello,
I have a 4 node cluster with 2 nodes assigned to 2 replica groups. When master node of one of my replicas failed, as mentioned in document, I expected my other node (replica) to become master but instead it went into UNKNOWN state. Hence I am unable to write data.
Kindly help.
-Rishabh Agrawal
Edited by: Rishabh Agrawal on Jun 25, 2012 5:05 AMHello Charles,
Following is the log u asked:
06-25-12 17:13:21:90 UTC+5:30 INFO [rg1-rn2] JE: Shutting down node rg1-rn2(2)
06-25-12 17:13:21:90 UTC+5:30 INFO [rg1-rn2] JE: Refreshed 0 monitors.
06-25-12 17:13:21:91 UTC+5:30 INFO [rg1-rn1] JE: Exiting inner Replica loop.
06-25-12 17:13:21:90 UTC+5:30 INFO [rg1-rn2] JE: Elections shutdown initiated
06-25-12 17:13:21:91 UTC+5:30 INFO [rg1-rn1] JE: Replica stats - Lag waits: 0 Lag wait time: 0ms. VLSN waits: 0 Lag wait time: 0ms.
06-25-12 17:13:21:90 UTC+5:30 INFO [rg1-rn2] JE: Elections shutdown completed
06-25-12 17:13:21:91 UTC+5:30 INFO [rg1-rn1] State change event: Mon Jun 25 17:13:21 IST 2012, State: UNKNOWN, Master: none
06-25-12 17:13:21:90 UTC+5:30 INFO [rg1-rn2] JE: Feeder manager soft shutdown.
06-25-12 17:13:21:91 UTC+5:30 INFO [rg1-rn1] JE: Election initiated; election #2
06-25-12 17:13:21:90 UTC+5:30 INFO [rg1-rn2] JE: Shutting down feeder for replica rg1-rn1 write time: 36ms Avg write time: 62us
06-25-12 17:13:21:91 UTC+5:30 INFO [rg1-rn1] JE: Election in progress. Waiting....
06-25-12 17:13:21:91 UTC+5:30 INFO [rg1-rn1] JE: Started election thread Mon Jun 25 17:13:21 IST 2012
06-25-12 17:14:02:22 UTC+5:30 INFO [admin1] [admin1] sn2: Service status: UNREACHABLE 06-25-12 17:14:02
06-25-12 17:14:02:22 UTC+5:30 INFO [admin1] [admin1] rg1-rn2: Service status: UNREACHABLE 06-25-12 17:14:00
-Rishabh
Edited by: Rishabh Agrawal on Jun 25, 2012 5:30 AM
Edited by: Rishabh Agrawal on Jun 25, 2012 5:31 AM -
Unknown state or association field with @Transformation
I have a class that uses a Transformation mapping for the IP address field. The IP address is the string dot notation in the class, but is a long integer in the database. My entity class looks like this:
@Entity
@Table(name="LOCATION_MAP")
public class LocationMap implements Serializable {
bq. @Transformation \\ @ReadTransformer(IPAttributeTransformer.class) \\ @WriteTransformer(IPAttributeTransformer.class) \\ @Column(name="IP_ADDR") \\ protected String ipAddr; \\ ...
IPAttributeTransformer is a class that implements both org.eclipse.persistence.mappings.transformers.AttributeTransfomer and org.eclipse.persistence.mappings.transformers.FieldTransformer.
When I attempt to execute the query, "select location.ipAddr from LocationMap location", it gives me the error Error compiling the query [(select location.ipAddr from LocationMap location|http://select location.ipAddr from LocationMap location]), line 1, column 16: unknown state or association field (ipAddr) of class (com.example.LocationMap).
If I comment out the @Transformation, @ReadTransformer, and @WriteTransformer annotations, then I don't get the error, but of course I also don't get the transformed IP address.
This mapping worked with TopLink 10g and the XML descriptor file, but I am trying to migrate to 11g and JPA with annotations.
Josh Davis
Edited by: user603300 on Mar 12, 2009 12:22 PM, forum software tried to turn square brackets into URL's.Hello Ryan,
The example I have does the following:
@ReadTransformer(method="buildNormalHours")
@WriteTransformers({
@WriteTransformer(method="getStartTime", column=@Column(name="START_TIME")),
@WriteTransformer(method="getEndTime", column=@Column(name="END_TIME"))
@Property(name="attributeName", value="normalHours")
protected Time[] getNormalHours() {
return normalHours;
where buildNormalHours looks like:
/** IMPORTANT: This method builds the value but does not set it.
* The mapping will set it using method or direct access as defined in the descriptor.
public Time[] buildNormalHours(Record row, Session session) {
Time[] hours = new Time[2];
/** This conversion allows for the database type not to match, i.e. may be a Timestamp or String. */
hours[0] = (Time) session.getDatasourcePlatform().convertObject(row.get("START_TIME"), java.sql.Time.class);
hours[1] = (Time) session.getDatasourcePlatform().convertObject(row.get("END_TIME"), java.sql.Time.class);
return hours;
Another uses a transformer class for the read:
@ReadTransformer(transformerClass=org.eclipse.persistence.testing.models.jpa.advanced.AdvancedReadTransformer.class)
which extends the org.eclipse.persistence.mappings.transformers.AttributeTransformer class and implements a buildAttributeValue method.
Hope this helps,
Chris -
Unknown state or association field
Hi,
I have a table "client" that has many "request", both have an id attribute that´s the primary key. I need to query the request of a type that belong to a certain type of client. In the Request entity I have the following named query:
@NamedQuery(name="Request.findByTypeAndClient", query="Select r from Request where r.type = :type and r.client.clientType= :clientType")
The app server starts fine, but when I try just get to the index page I get the following error:
javax.faces.el.EvaluationException: javax.faces.FacesException: javax.faces.FacesException: Can't instantiate class: 'com.app.web.backing.Index'.. class com.app.web.backing.Index : javax.ejb.EJBException:
Exception Description: Unknown state or association field [clientType] of class [com.app.entity.Request].; nested exception is: Exception [TOPLINK-8030] (Oracle TopLink Essentials - 2006.8 (Build 060829)): oracle.toplink.essentials.exceptions.EJBQLException
Exception Description: Unknown state or association field [clientType] of class [com.app.entity.Request].
at com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:190)
Caused by: javax.ejb.EJBException: Exception Description: Unknown state or association field [clientType] of class [com.app.entity.Request].; nested exception is: Exception [TOPLINK-8030] (Oracle TopLink Essentials - 2006.8 (Build 060829)): oracle.toplink.essentials.exceptions.EJBQLException
Exception Description: *Unknown state or association field [clientType] of class [com.app.entity.Request].*
I've made sure I don't even use that query at the index page.
Why does it happen and how could I fix it?
I'm using jdev 10.1.3
Thanks in advanceThat could be a hint...
The Client entity has a list of Requests, and the Request entity keeps a reference to the Client it belongs to. Since what I'm trying to list are the requests, and not the clients, I was expecting I could use the Client reference in the Request entity to make the query. Isn't it possible? -
Unknown state or association field exception
I meet the above exception on my project, but not know where my problem lies. Please help. Thank you.
Here is the exception message:
Exception Description: Unknown state or association field [projectNoSub] of class [database.SfcWip].
Here is the entity class:
@Entity
@Table(name = "SFC_WIP")
@NamedQueries( {
@NamedQuery(name = "SfcWip.findByProjectNoSub", query = "SELECT s FROM SfcWip s WHERE s.sfcWipPK.projectNoSub = :projectNoSub"),
@NamedQuery(name = "SfcWip.findByPcbPartNo", query = "SELECT s FROM SfcWip s WHERE s.sfcWipPK.pcbPartNo = :pcbPartNo"),
@NamedQuery(name = "SfcWip.findByPcbQty", query = "SELECT s FROM SfcWip s WHERE s.pcbQty = :pcbQty"),
@NamedQuery(name = "SfcWip.findByFlowId", query = "SELECT s FROM SfcWip s WHERE s.flowId = :flowId"),
@NamedQuery(name = "SfcWip.findByStatus", query = "SELECT s FROM SfcWip s WHERE s.status = :status"),
@NamedQuery(name = "SfcWip.findByRemarks", query = "SELECT s FROM SfcWip s WHERE s.remarks = :remarks")
public class SfcWip implements Serializable {
* EmbeddedId primary key field
@EmbeddedId
protected SfcWipPK sfcWipPK;
@Column(name = "PCB_QTY")
private BigDecimal pcbQty;
@Column(name = "FLOW_ID")
private String flowId;
@Column(name = "STATUS")
private String status;
@Column(name = "REMARKS")
private String remarks;
@OneToMany(cascade = CascadeType.ALL, mappedBy = "sfcWip")
private Collection<SfcWiptran> sfcWiptranCollection;
/** Creates a new instance of SfcWip */
public SfcWip() {
public SfcWip(SfcWipPK sfcWipPK) {
this.sfcWipPK = sfcWipPK;
public SfcWip(String pcbPartNo, String projectNoSub) {
this.sfcWipPK = new SfcWipPK(pcbPartNo, projectNoSub);
public SfcWipPK getSfcWipPK() {
return this.sfcWipPK;
public void setSfcWipPK(SfcWipPK sfcWipPK) {
this.sfcWipPK = sfcWipPK;
public BigDecimal getPcbQty() {
return this.pcbQty;
public void setPcbQty(BigDecimal pcbQty) {
this.pcbQty = pcbQty;
public String getFlowId() {
return this.flowId;
public void setFlowId(String flowId) {
this.flowId = flowId;
public String getStatus() {
return this.status;
public void setStatus(String status) {
this.status = status;
public String getRemarks() {
return this.remarks;
public void setRemarks(String remarks) {
this.remarks = remarks;
public Collection<SfcWiptran> getSfcWiptranCollection() {
return this.sfcWiptranCollection;
public void setSfcWiptranCollection(Collection<SfcWiptran> sfcWiptranCollection) {
this.sfcWiptranCollection = sfcWiptranCollection;
@Override
public int hashCode() {
int hash = 0;
hash += (this.sfcWipPK != null ? this.sfcWipPK.hashCode() : 0);
return hash;
@Override
public boolean equals(Object object) {
// TODO: Warning - this method won't work in the case the id fields are not set
if (!(object instanceof SfcWip)) {
return false;
SfcWip other = (SfcWip)object;
if (this.sfcWipPK != other.sfcWipPK && (this.sfcWipPK == null || !this.sfcWipPK.equals(other.sfcWipPK))) return false;
return true;
@Override
public String toString() {
return "database.SfcWip[sfcWipPK=" + sfcWipPK + "]";
@Embeddable
public class SfcWipPK implements Serializable {
@Column(name = "PROJECT_NO_SUB", nullable = false)
private String projectNoSub;
@Column(name = "PCB_PART_NO", nullable = false)
private String pcbPartNo;
/** Creates a new instance of SfcWipPK */
public SfcWipPK() {
public SfcWipPK(String pcbPartNo, String projectNoSub) {
this.pcbPartNo = pcbPartNo;
this.projectNoSub = projectNoSub;
public String getProjectNoSub() {
return this.projectNoSub;
public void setProjectNoSub(String projectNoSub) {
this.projectNoSub = projectNoSub;
public String getPcbPartNo() {
return this.pcbPartNo;
public void setPcbPartNo(String pcbPartNo) {
this.pcbPartNo = pcbPartNo;
@Override
public int hashCode() {
int hash = 0;
hash += (this.pcbPartNo != null ? this.pcbPartNo.hashCode() : 0);
hash += (this.projectNoSub != null ? this.projectNoSub.hashCode() : 0);
return hash;
@Override
public boolean equals(Object object) {
// TODO: Warning - this method won't work in the case the id fields are not set
if (!(object instanceof SfcWipPK)) {
return false;
SfcWipPK other = (SfcWipPK)object;
if (this.pcbPartNo != other.pcbPartNo && (this.pcbPartNo == null || !this.pcbPartNo.equals(other.pcbPartNo))) return false;
if (this.projectNoSub != other.projectNoSub && (this.projectNoSub == null || !this.projectNoSub.equals(other.projectNoSub))) return false;
return true;
@Override
public String toString() {
return "database.SfcWipPK[pcbPartNo=" + pcbPartNo + ", projectNoSub=" + projectNoSub + "]";
}The problem lies in the NamedQuery that I found in the end.
Maybe you are looking for
-
How to restore a schema from a corrupted DB
Hi, I need your help. I am using a laptop and a desktop, both with XP SP2. I have installed Oracle XE on both of them. They are in a LAN. I have been using them for almost a year now... But since few days ago I cannot use the DB from my desktop (ther
-
Retrieve permision levels for all items in list programatically
Hi all, I need to retrieve all permission levels for all items in a list programatically using client object model in sharepoint 2010.Please send me the code sample. Regards, Praveen
-
Newsfeed error - The operation failed because the server could not access the distributed cache.
Recently installed SharePoint 2013 RTM, on the newsfeed page an error is displayed, and no entries display in the following or everyone tabs. "The operation failed because the server could not access the distributed cache." Reading through various po
-
No appropriate entry found in table ADRT
Hi, In XI T-Rfc (sm58) we get error --> No appropriate entry found in table ADRT Kindly, suggest. Akki.
-
Memory read error causes crash, cause unknown?
I've been getting the error below at various times, usually after I've been working for a while without saving. It appears to be random. Does anyone know what kinds of things might trigger this? Is there a fix? FormDesigner.exe - Application Error Th