CF8 Server Restarts Under Load - Seems to be locked threads

We are running a CF8 box on Win2k3 with around 400 active
datasources on a mix of SQL 2000 and SQL 2005 databases using JDBC
connections. We have the max running requests set to 12. Under
heavy load the request will start queueing. When they do, it's only
a matter of time before the service restarts. The standard output
logs are full of the following. Does that mean anything to anyone?
"obj-skimmer" daemon prio=5 tid=0x03bd5620 nid=0x1ba0 in
Object.wait() [0x5a30f000..0x5a30fd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15feeac8> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:47)
- locked <0x15feeac8> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)
"obj-skimmer" daemon prio=5 tid=0x03bd4940 nid=0x1628 in
Object.wait() [0x5a20f000..0x5a20fd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15feeb70> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:47)
- locked <0x15feeb70> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)
"obj-skimmer" daemon prio=5 tid=0x03bd3c98 nid=0xd84 in
Object.wait() [0x5a10f000..0x5a10fd94]
at java.lang.Object.wait(Native Method)
- waiting on <0x15feebc8> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at
coldfusion.server.j2ee.pool.PoolSkimmerThread.run(PoolSkimmerThread.java:47)
- locked <0x15feebc8> (a
coldfusion.server.j2ee.pool.PoolSkimmerThread)
at java.lang.Thread.run(Thread.java:534)

Hi John,Tia,
I thought I will peep into the forums before taking off for Diwali
Tia, as John correctly mentioned, it will be helpful if you can share your tomcat logs.
Meanwhile there are a few things you can try. RoboHelp server's default configuration is optimal for development/testing environment. There are a few settings you need to configure before you move to production environment.
1) Increase Java heap space: You can find details about this configuration in RoboHelp Server documentation. Here are a few links on how to do that:
)       http://www.adobe.com/products/robohelpserver/faq/ under header “How to Install Apache Tomcat?”
2)      http://help.adobe.com/en_US/RoboHelp/8.0/RoboHelpServer/WS5b3ccc516d4fbf351e63e3d11c0aaa9f d7-7ff4.html#WSa148d62006dae0de1172e0811dc117a846-8000
3)      http://blogs.adobe.com/techcomm/roboserver_8_gettingstarted.pdf under heading “Install Tomcat”
2) Increae java perm space :
You need to increase PermSpace available to your tomcat. You can do that using <tomcat-install-dir>/bin/tomcat6w.exe
Just ensure that your “Maximum memory pool” and “XX:MaxPermSize” should not add up to more than actual memory you can make available to tomcat.
The setting is as shown in this snapshot:
3) Ms Access database is not optimal for production environment. You should use SQL server or oracle database for production environment.
Hope this information helps
Tulika.

Similar Messages

  • Weblogic Server Crashes under load

    I am completely lost and do not know where to start for this problem.
    Once the server gets a heavy load it crashes. I try to ping it and
    does not respond even thought the process is running on Unix. I do
    not know where to start. Can someone please help me!!!
    Thanks. :)
    Here is the config:
    Sun Solaris 8 4 cpu @900 Mhz 32Gb mem
    java options are 512 min and max mem.
    I am using ThreadCount=50
    I thought of increasing these pools but it did not seem to help.
    Things I have noticed
    - Before it crashes I see this in the error log.
    It says starting weblogic? Why?
    At one point I have multiple weblogic servers running.
    ####<Nov 21, 2002 9:37:11 PM EST> <Notice> <WebLogicServer> <hostname>
    <PIA> <main> <system> <> <000327> <Starting WebLogic Admin Server
    "PIA" for domain "peoplesoft">
    ####<Nov 21, 2002 9:37:22 PM EST> <Notice> <Management> <hostname>
    <PIA> <main> <system> <> <141053> <Application Poller not started for
    production server.>
    ####<Nov 21, 2002 9:37:22 PM EST> <Notice> <WebLogicServer> <hostname>
    <PIA> <ListenThread> <system> <> <000202> <ListenThread listening on
    port 7003>
    ####<Nov 21, 2002 9:37:22 PM EST> <Notice> <WebLogicServer> <hostname>
    <PIA> <SSLListenThread> <system> <> <000202> <SSLListenThread
    listening on port 7004>
    - I have a lot of these errors but from what I know I can ignore
    them?!
    Didn't meet stated Content-Length, wrote: '986' bytes instead of
    stated: '13634' bytes.>
    - I dumped the Thread Heap before it crashed and got this
    "ExecuteThread: '6' for queue: 'default'" daemon prio=5 tid=0xb1a0b0
    nid=0x14 runnable [0xd3900000..0xd39019d8]
    at java.net.SocketInputStream.socketRead(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:86)
    at java.io.DataInputStream.readFully(DataInputStream.java:156)
    at bea.jolt.NwHdlr.openConnection(NwHdlr.java:323)
    at bea.jolt.CMgr.connect(CMgr.java:71)
    at bea.jolt.JoltSession.logon(JoltSession.java:245)
    at bea.jolt.JoltSession.<init>(JoltSession.java:125)
    at psft.pt8.net.JoltSessionWrapper.<init>(JoltSessionWrapper.java:66)
    at psft.pt8.net.JoltSessionPool.createConnection(JoltSessionPool.java:366)
    at psft.pt8.net.JoltSessionPool.getJoltSession(JoltSessionPool.java:215)
    at psft.pt8.net.NetSession.getJoltSession(NetSession.java:397)
    at psft.pt8.net.NetReqRepSvc.sendRequest(NetReqRepSvc.java:359)
    at psft.pt8.net.NetService.requestService(NetService.java:144)
    at psft.pt8.net.NetSession.connect(NetSession.java:211)
    at psft.pt8.net.NetSession.<init>(NetSession.java:150)
    at psft.pt8.jb.JBEntry.connectWithBlob(JBEntry.java:565)
    at psft.pt8.jb.JBEntry.connect(JBEntry.java:503)
    at psft.pt8.auth.PSAuthenticator.authenticate(PSAuthenticator.java:353)
    at psp.onLogin(psp.java:504)
    at psp.service(psp.java:313)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:265)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:200)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2456)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:1986)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
    "ExecuteThread: '5' for queue: 'default'" daemon prio=5 tid=0xe3dd40
    nid=0x13 runnable [0xd3a01000..0xd3a019d8]
    # HotSpot Virtual Machine Error, Internal Error
    # Please report this error at
    # http://java.sun.com/cgi-bin/bugreport.cgi
    Please help!!!! Thanks.

    Did you get resolution to your crash.
    As you can see below I am getting the same error.
    This is reproducible in a simple benchmark.
    It will start to error after exactly the same number of logins in a given configuration..
    Any help would be appreciated.
    Corey. pt 818 on sun v880 8cpu(750) w/32gb
    Mon Mar 03 19:55:55 MST 2003:<E> <ServletContext-General> Servlet failed with
    Exception
    java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start(Native Method)
    at bea.jolt.NwHdlr.start_threads(NwHdlr.java:2023)
    at bea.jolt.NwHdlr.openConnection(NwHdlr.java:940)
    at bea.jolt.CMgr.connect(CMgr.java:78)
    at bea.jolt.JoltSession.logon(JoltSession.java:238)
    at bea.jolt.JoltSession.<init>(JoltSession.java:117)
    at psft.pt8.net.JoltSessionWrapper.<init>(JoltSessionWrapper.java:66)
    at psft.pt8.net.JoltSessionPool.createConnection(JoltSessionPool.java:366)
    at psft.pt8.net.JoltSessionPool.getJoltSession(JoltSessionPool.java:215)
    at psft.pt8.net.NetSession.getJoltSession(NetSession.java:372)
    at psft.pt8.net.NetReqRepSvc.sendRequest(NetReqRepSvc.java:383)
    at psft.pt8.net.NetService.requestService(NetService.java:141)
    at psft.pt8.net.NetReqRepSvc.requestService(NetReqRepSvc.java:268)
    at psft.pt8.net.NetSession.connect(NetSession.java:179)
    at psft.pt8.net.NetSession.<init>(NetSession.java:142)
    at psft.pt8.jb.JBEntry.connectWithBlob(JBEntry.java:522)
    at psft.pt8.jb.JBEntry.connect(JBEntry.java:443)
    at psft.pt8.auth.PSAuthenticator.authenticate(PSAuthenticator.java:344)
    at iclientservlet.onLogin(iclientservlet.java:342)
    at iclientservlet.service(iclientservlet.java:231)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
    at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:915)
    at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:879)
    at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
    at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:365)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:253)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:129)

  • WL server failed to response under load

    Hi
    We have a quite strange situation on my sight. Under load our WL 10.3.2 server failed to response. We are using RestEasy with HttpClient version 3.1 to coordinate with web service deployed as WAR.
    What we have is a calculation process that run on 4 containers based on 4 physical machines and each of them send request to WL during calculation.
    Each run we see a messages from HttpClient like this:
    [THREAD1] INFO   I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server OUR_SERVER_NAME failed to respond
    [THREAD1] INFO   Retrying request
    The HttpClient make several requests until get necessary data.
    I want to understand why WL can refuse connections. I read about WL thread pool that process http request and understand that WL allocate separate thread to process web request and the numbers of threads is not bounded in default configuration. Also our server is configured Maximum Open Sockets: -1 which means that the number of open sockets is unlimited.
    From this thread I'd want to understand where the issue is? Is it on WL side or it's a problem of our business logic? Can you guys help to deeper investigate the situation.
    What should I check more in order to understand that our WL server is configured to work with as much requests as we want?

    Thanks for a good explanation!
    Linux server it's not in my competencies but what we have now is:
    root    soft    nofile  3072
    root    hard    nofile  65536
    So the upper number is really big. As I understood it's a config for whole processes in operation system.
    But I go further and find these numbers for our WL managed server.
    cat /proc/23085/limits
    Limit                     Soft Limit           Hard Limit           Units
    Max cpu time              unlimited            unlimited            seconds
    Max file size             unlimited            unlimited            bytes
    Max data size             unlimited            unlimited            bytes
    Max stack size            10485760             unlimited            bytes
    Max core file size        0                    unlimited            bytes
    Max resident set          unlimited            unlimited            bytes
    Max processes             578455               578455               processes
    Max open files            8192                 8192                 files
    Max locked memory         32768                32768                bytes
    Max address space         unlimited            unlimited            bytes
    Max file locks            unlimited            unlimited            locks
    Max pending signals       578455               578455               signals
    Max msgqueue size         819200               819200               bytes
    Max nice priority         0                    0
    Max realtime priority     0                    0
    Again it's only a config numbers. If continue moving in this direction I'd want to see the real numbers when the server is under load. So finding these numbers I would make an assumption that the issue is here. Don't you know is there a log file or history file where I can find a max open file descriptors for WL server?

  • JDK 1.3.1_08 Under load on linux (RH7.3)

    I have been running JDK 1.3.1 on linux under a development environment for quite a long time (2+ years) without issue, running a large Java application cinsisting of EJBs, and Web apps being run by BEA Weblogic. In an effort to cut costs we are migrating to linux for our production systems.
    During the testing phase of running our applications on a dual-SMP Xeon based server with 2 GB of RAM we had no issues. However under load the java child processes (threads to the JVM) are getting very low priorities (15-19) and have a status of "SN" sleeping, and low-priority. If you run pstack on the PPID of the java processes they seem to be 'stuck' in nanosleep.
    Does anyone out there have experiencing running large apps under high load on linux platforms?

    had no issues. However under load the java child
    processes (threads to the JVM) are getting very low
    priorities (15-19) and have a status of "SN" sleeping,
    and low-priority. Every Linux JVM that I know does not use native thread priorities (nice levels) at all, so if you are seeing child PIDs at the 15-19 range, someone most likely started the JVM at the exact same nice level. How many runtimes do you have running? What are your 5, 10, and 15 minute load averages? Are they under 2.0? Can you start your runtimes at a lower nice level?
    Does anyone out there have experiencing running large
    apps under high load on linux platforms?The `S' you see w/ ps is not particularly indicative that its stuck in this queue, just may be waiting for IO. Adjust your runtime nice level to 10 and under if your load is high and you have a higher IO priority runtime (as you have a dual-proc system).

  • CF8/JRun4 Cluster for Load Balancing

    Does anyone have an example of how to set up a CF8/JRun4
    cluster for load balancing?
    I have three servers:
    x004 - Linux - Apache2 (10.0.0.54,10.1.0.54)
    x020 - Linux - JRun4/CF8 (10.0.0.70,10.1.0.70)
    x021 - Linux - JRun4/CF8 (10.0.0.71,10.1.0.71)
    Every server in our network has two network cards. One
    network card is attached to 10.0.x.x which has a gateway to the
    internet and runs at 100Mbps and is firewalled, and the other is
    attached to 10.1.x.x which runs at 1Gbps and is internal with no
    gateway. I'm trying to set it up so web traffic arrives on
    10.0.0.54 into Apache and mod_jrun20 bootstraps a cluster named
    STST using 10.1.0.54 which consists of STST_x020 coldfusion server
    running on x020 and STST_x021 running on x021. I want the
    communications between JRun4 on x020 and x021 to occur on the
    10.1.x.x network and eventhough JRun and ColdFusion will only use
    the 10.1.x.x network I still need the 10.0.x.x network card
    attached for other purposes which require a gateway. I have
    installed JRun4/CF8 about 10 times already and it seems I have no
    control over what network JRun4 clusters on... sometimes it will
    communicate on one, sometimes the other and without being able to
    set which network is being used there always seems to be "network
    error" on at least one of the two CF8 servers. I was able to get
    everything working fine by disabling the network cards on the
    10.0.x.x network and re-installing everything... but as soon as I
    added the network cards back the whole thing was broken again.
    How is this supposed to work? Most of the examples are either
    no clustering or clustering on the same machine with Apache running
    on the same box... I don't see any clustering across machines
    examples.
    How do I install a connector on a web server which doesn't
    have JRun on it and get wsconfig to connect to a multi-machine
    cluster when wsconfig only accepts a single IP address as a host
    and the cluster is not listed?
    How do I get JRun to bind to a specific network card?
    Does this work if I choose a J2EE server other than JRun?
    Any help anyone can provide is greatly appreciated. I'm
    getting close to giving up which means staying on the non-clustered
    environment and figuring out how we can deal with scalability by
    switching to something else.

    The article at
    http://www.adobe.com/go/1e8e9170
    is specific to configuring two or more cluster nodes that reside on
    separate networks, e.g. 10.0.1.0/24 and 10.0.2.0/24. (The article
    doesn't state it, but you can only use unicast peers if your
    cluster nodes host a single instance of JRun or multiple instances
    of JRun in the same cluster domain. When performing unicast
    discovery, JRun looks for all Jini groups and not just the cluster
    group.)
    Anyhow, that's not your problem. The simplest solution is you
    haven't enabled the jrun.servlet.jrpp.JRunProxyService service. I'm
    most familiar with the Windows version of JRun, but I'm assuming
    the directory structure is similar across platforms. In
    <jrun_root>/servers/<name>/SERVER-INF/jrun.xml, set the
    deactivated attribute of the jrun.servlet.jrpp.JRunProxyService
    service to false and restart JRun. You should now see JRun
    listening on the appropriate port. (The default for the first
    manually created instance is 51000.) You can limit the proxy
    service to a single interface using the interface attribute.
    If you have enabled the proxy service, verify your security
    settings in <jrun_root>/lib/security.properties. It's usually
    best to limit access to specific hosts. Comment out the
    jrun.subnet.restriction parameter and set the jrun.trusted.hosts to
    the IP address of your web server, e.g. 10.1.0.54.
    Forcing all JRun processes/services to listen on a single
    interface isn't difficult, but it does require modifying quite a
    few configuration files by hand. If you need assistance with that,
    I can elaborate.
    Configuring the JRun module under Apache is pretty
    straightforward. If you're not using virtual hosts, it's very
    simple. If you are using virtual hosts, it's still simple, but your
    JRun configuration can be virtual host-specific.
    On your Apache server, you'll want to create a directory
    structure for the JRun module. I'll assume
    /opt/jrun/lib/wsconfig/1, but you can use anything you want. Once
    the directory structure is created, extract the appropriate JRun
    module from wsconfig.jar to the new directory. You're most likely
    interested in the Apache 2.0 module,
    wsconfig.jar/connectors/apache/intel-linux/prebuilt/mod_jrun20.so.
    Let's assume you've extracted the module to
    /opt/jrun/lib/wsconfig/1/mod_jrun20.so. Your Apache service account
    should have read, write, and execute permissions on the
    /opt/jrun/lib/wsconfig/1 directory.
    The JRun module configuration is normally appended to your
    current httpd.conf file by wsconfig. Here's a sample configuration:
    LoadModule jrun_module
    "/opt/jrun/lib/wsconfig/1/mod_jrun20.so"
    <IfModule mod_jrun20.c>
    JRunConfig Verbose false
    JRunConfig Apialloc false
    JRunConfig Ssl false
    JRunConfig Ignoresuffixmap false
    JRunConfig Serverstore
    "/opt/jrun/lib/wsconfig/1/jrunserver.store"
    JRunConfig Bootstrap 10.1.0.70:51000
    #JRunConfig Errorurl <optionally redirect to this URL on
    errors>
    #JRunConfig ProxyRetryInterval 600
    #JRunConfig ConnectTimeout 30
    #JRunConfig RecvTimeout 30
    #JRunConfig SendTimeout 30
    AddHandler jrun-handler .jsp .jws .cfm .cfml .cfc .cfr
    .cfswf
    </IfModule>
    You may also want to update your DirectoryIndex directive
    with an appropriate index page, e.g. index.cfm.
    After the first request to a page handled by the JRun module
    is received, the module will query the boostrap server,
    10.1.0.70:51000, for a list of cluster peers. If you've configured
    your cluster correctly, a line similar to following will be written
    to /opt/jrun/lib/wsconfig/1/jrunserver.store:
    proxyservers=10.1.0.70:51000;10.1.0.71:51000
    You can create/edit this file manually as well.
    Unfortunately, the bootstrap option only accepts one server. If
    your bootstrap server is down, the JRun module will use the values
    in jrunserver.store directly, if the file exists.
    Here's a complete list of JRun module options:
    metrics *
    debugger *
    ssl *
    verbose
    traceflags
    serverstore
    bootstrap
    errorurl
    apialloc
    ignoresuffixmap
    proxyretryinterval
    connecttimeout
    recvtimeout
    sendtimeout
    sslcalist
    Options flagged with an asterisk can only be configured at
    the Apache server level. All other options can be configured at the
    server level and/or the virtual host level. The usage of these
    options is in the JRun documentation, and the JRun module source
    code is included in wsconfig.jar. Keep in mind that versions of the
    JRun module shipped prior to ColdFusion 8 were coded to assign the
    connecttimeout and sendtimeout options to the socket connection
    timeout. Whichever option appeared last in your configuration ended
    up as the final value. This has been fixed in ColdFusion 8 and
    presumably the next release of the JRun updater.
    I think that's a good start. If you need more information or
    can't find what you need in the JRun or ColdFusion documentation,
    let me know.
    If you're looking for resiliency, I highly recommend
    expanding your configuration to include a second web server and a
    hardware load-balancer (preferably one that supports redudancy via
    multiple paths and devices, e.g. devices from Cisco, F5, or Foundry
    Networks). Often, however, running Apache on the ColdFusion
    server(s) provides adequate performance, and round-robin DNS
    records coupled with the ability to update DNS quickly in the event
    of a failure may be all you need for load-balancing and
    failover.

  • Reload of composite on server restart

    Hi,
    We have a processes which is invoking some 5 to 6 other bpel processes.On server restart this dependent process is getting loaded first ,i.e prior to the load of other bpel processes.So the reload is failing with :
    The composite OrderRespOAppProcessSrvc (1.0) is not available. This could happen because either the composite has been undeployed or soa-infra has not yet loaded this composite.
    Error occurred during deployment of component: OrderRespOAppProcessSrvc to service engine: implementation.bpel, for composite: OrderRespOAppProcessSrvc: ORABPEL-05250<br><br>Error deploying BPEL suitcase.<br>error while attempting to deploy the BPEL component file "D:\SOA_Middleware\user_projects\domains\dev_domain\servers\soa_server1\dc\soa_4b0a73b3-5a1e-4ab2-b7b7-41bb8f392586"; the exception reported is: java.lang.IllegalStateException: this is only a psuedo object<br><br>This error contained an exception thrown by the underlying deployment module.<br>Verify the exception trace in the log (with logging level set to debug mode).<br>
    oracle.sysman.emSDK.app.exception.EMSystemException
         at oracle.sysman.emai.model.sca.composite.core.impl.EMCompositeModel.handleErrorDetails(EMCompositeModel.java:83)
         at oracle.sysman.emai.model.sca.share.composite.CompositeCoreModel.initialize(CompositeCoreModel.java:122)
         at oracle.sysman.emai.model.sca.composite.core.impl.CompositeModel.initialize(CompositeModel.java:46)
         at oracle.sysman.emai.model.sca.EMEntityModel._initializeEntityData(EMEntityModel.java:28)
         at oracle.sysman.emai.model.sca.common.SCAEntityObject.initializeSCAData(SCAEntityObject.java:124)
         at oracle.sysman.emai.model.sca.composite.core.impl.EMCompositeModel.initializeSCAData(EMCompositeModel.java:31)
         at oracle.sysman.emai.model.sca.SCAObject.initializeData(SCAObject.java:62)
         at oracle.sysman.emSDK.model.EMObject.reset(EMObject.java:216)
         at oracle.sysman.core.model.util.ModelUtil$EMObjectResetter.initOrResetEMObject(ModelUtil.java:328)
         at oracle.sysman.core.model.util.ModelUtil$EMObjectResetter.access$000(ModelUtil.java:280)
         at oracle.sysman.core.model.util.ModelUtil.initializeMBEMObjects(ModelUtil.java:231)
         at oracle.sysman.core.model.util.ModelUtil.initializeAllEMObjects(ModelUtil.java:128)
         at oracle.sysman.core.model.util.ModelUtil.initializeAllEMObjects(ModelUtil.java:101)
         at oracle.sysman.emSDK.view.util.RefreshBean.refreshPageData(RefreshBean.java:91)
         at oracle.sysman.emSDK.view.util.RefreshBean.refreshPage(RefreshBean.java:73)
         at oracle.sysman.emai.view.sca.common.EMNavigationHandler.refreshPage(EMNavigationHandler.java:677)
         at oracle.sysman.emai.view.sca.composite.core.EMCompositeInstancesView.dataChanged(EMCompositeInstancesView.java:119)
         at oracle.sysman.emai.model.sca.share.composite.CompositeInstances._fireDataChangedEvent(CompositeInstances.java:194)
         at oracle.sysman.emai.model.sca.share.composite.CompositeInstances.getDataModel(CompositeInstances.java:158)
         at sun.reflect.GeneratedMethodAccessor971.invoke(Unknown Source)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at javax.el.BeanELResolver.getValue(BeanELResolver.java:261)
         at com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:173)
         at com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:200)
         at com.sun.el.parser.AstValue.getValue(Unknown Source)
         at com.sun.el.ValueExpressionImpl.getValue(Unknown Source)
         at org.apache.myfaces.trinidad.bean.FacesBeanImpl.getProperty(FacesBeanImpl.java:68)
         at org.apache.myfaces.trinidad.component.FacesBeanWrapper.getProperty(FacesBeanWrapper.java:58)
         at org.apache.myfaces.trinidad.component.UIXTable$RowKeyFacesBeanWrapper.getProperty(UIXTable.java:594)
         at org.apache.myfaces.trinidad.component.UIXComponentBase.getProperty(UIXComponentBase.java:1194)
         at org.apache.myfaces.trinidad.component.UIXIterator.getValue(UIXIterator.java:706)
         at org.apache.myfaces.trinidad.component.UIXCollection._flushCachedModel(UIXCollection.java:1386)
         at org.apache.myfaces.trinidad.component.UIXCollection.encodeBegin(UIXCollection.java:515)
         at javax.faces.component.UIComponent.encodeAll(UIComponent.java:928)
         at javax.faces.component.UIComponent.encodeAll(UIComponent.java:933)
         at org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
         at oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
         at oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
         at oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
    Is there any way to control the reload of processes when server is restarted.

    Hi sab2,
    There seems to be no reason for error if you have used correct Abstract WSDLs from the MDS in your composite.xml. The best way to check this is to do a Search for hostname in the entire project and ensure that the host-specific URLs exist only in the composite.xml that too in tags starting with *<binding.ws*. Any other WSDL references e.g. in composite namespaces, in Transformation files Source/target namespaces etc. can be replaced with the abstract WSDLs references.
    The task is not done yet. You need to further create the configuration plans corresponding to each server environment where your composite would be deployed e.g. DEV_config_plan.xml, PROD_config_plan.xml. These config plans would replace the concrete locations in your composite.xml at the deployment time.
    Also have another issue when we move the jars from dev to prod i see the endpoint on MDS is modified
    PROD endpoints but after the process is deployed it still has the Endpoint pointing to Dev URL.
    I dont see it any where on the MDS and the Composite WSDL.Are you using proper config plans? If not please use one.
    The Process we use to migrate the code from DEV to PROD is by exporting jars from the
    DEV instance and deploying them on PRD instance, does this have any effect on the endpoint pointing to the DEV URLI'm not sure about this approach as per the recommended approaches. You can however, use the same deployable JARs and deploy from the FMW console by selecting the appropriate configuration plan.
    Regards,
    Neeraj Sehgal

  • Missing data after server restart

    Hi,
    We restarted our MDM server at the OS level last night. Everything seemed okay after the restart. However, when our users came in this morning, they found that some data was missing from a table. We have a Material repository with 3 main tables. Some data which have previously been populated was not appearing in our Products Detail table, the largest of the three. A large number of rows in the table had columns that failed to load with data on the repository auto-start that occured after the server restart. The auto start value in the mds.ini file was set at 300 (5 minutes) as we had seen this iisue happen once before, and thought that may help.
    A stop and restart of the repository corrected the issue. Is there anything we could have done to prevent this? Is it worthwhile to try increasing the auto start delay again? Validating detail data after every server restart is not an effective option.  
    MDM version: 7.1 SP08
    OS: AIX  DB: Oracle 11g
    Thanks!
    Dave

    Hi Dave,
    I meant that when you load repository using Immediate data is loaded from .acl files (accelerator files) in a async manner.
    Whereas when you load by Update Indices new acl files are generated.
    Below men6tioned is a comparision between the 2 ways of laoding repository:
    Load Repository Immediate
    o Loads indexes from accelerator files instead of being regenerated by reading the data them from the DB and creating the index.
    o Automatically builds indexes stored in out-of-date accelerator files.
    o Is usually fast.
     Load Repository with Update Indices
    o Rebuilds all MDM indexes files by reading and indexing data stored in the DB. The results are flushed on disk as accelerator files.
    o Necessary by repository schema changes or if any error occurs.
    o Might take significant amount of time to complete.
    So when you observe some visual data impact in your repository it is good idea to rebuild indexes.
    I guess we cannot schedule this as default behavior is to load by immediate even through the inin parameter.
    as a best practice one can do this by fixing a time frame eg monthly depending on data/schema changes.
    Thanks,
    Ravi

  • Composite with dependency not working after soa server restart

    Hello,
    I have composite application that invokes another composite application. After deploying those work fine. But after restarting soa server the one having dependency does not work any more. To soa server log I get following:
    <Jul 1, 2010 11:45:29 AM EEST> <Error> <oracle.integration.platform> <SOA-20003> <Unable to register service.
    oracle.fabric.common.FabricException: Error in getting XML input stream: http://Yacico:8001/soa-infra/services/default/validationForCC/getStatusByCC?WSDL: Response: '503: Service Unavailable' for url: 'http://Yacico:8001/soa-infra/services/default/validationForCC/getStatusByCC?WSDL'
    at oracle.fabric.common.metadata.MetadataManagerImpl.getInputStreamFromAbsoluteURL(MetadataManagerImpl.java:276)
    Caused By: java.io.FileNotFoundException: Response: '503: Service Unavailable' for url: 'http://Yacico:8001/soa-infra/services/default/validationForCC/getStatusByCC?WSDL'
    <Jul 1, 2010 11:45:30 AM EEST> <Error> <oracle.integration.platform> <SOA-20020> <Deployment of composite "POProcessing" failed: Unable to find a WSDL that has a definition for service {http://oracle.com/sca/soapservice/POProcessing/POProcessing/receivePO}receivePO and port execute_pt. Please make sure that the port attribute for the binding defined in the composite file is correct by checking the namespace, service name, and port name. In addition, check that the WSDL associated with the binding namespace is imported and currently reachable (check the import nodes at the top of the composite file). Finally, validate the HTTP proxy settings for the server..>
    So POProcessing does not work any more after server restart. validationForCC works fine also after server restart. Url http://...validationForCC/getStatusByCC?WSDL points to wsdl file and is visible by browser.
    I use soa suite 11g patch set 2 (11.1.1.3) running on redhat enterprise linux 5.
    Any idea what is the problem?
    Is it somehow possible to configure which composites are started first during server startup?
    regards, Matti
    Edited by: user10197965 on Jul 1, 2010 2:28 AM

    Yes, I did that. I'm not all that happy about this as a solution either, but it's better than making multiple copies.
    We have since found out that his is a known bug and that it is fixed in some, but not all, deployments.
    -------- see below -------------
    Composites With WSDL Dependencies Fail To Deploy Following SOA Server Restart [ID 1272070.1]          
    Modified:Jul 19, 2012 Type:PROBLEM Status:MODERATED Priority:3                         
    In this Document
         Symptoms
         Cause
         Solution
         References
    This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process and therefore has not been subject to an independent technical review.
    Applies to:
    Oracle SOA Platform - Version 11.1.1.3.0 and later
    Information in this document applies to any platform.
    Symptoms
    A SOA Project has an external reference to a Web Service or a reference to another Composite.
    When the SOA Server is started, the Composite is trying to access the WSDL of it's referenced Web Service in order to load data structures. If SOA can not find the WSDL then the Composite fails to load/deploy.
    Once this happens the Composite can not be started, or shutdown, from Enterprise Manager Fusion Middleware Control Application.
    Related Error Messages:
    [ERROR] [SOA-20020] ... Unable to find a WSDL that has a definition for service ... Please make sure that the port attribute for the binding defined in the composite file is correct by checking the namespace, service name, and port name. In addition, check that the WSDL associated with the binding namespace is imported and currently reachable (check the import nodes at the top of the composite file). Finally, validate the HTTP proxy settings for the server.]
    javax.wsdl.WSDLException: WSDLException: faultCode=INVALID_WSDL: Error reading import of oramds
    Cause
    When SOA Server is restarting the Composite can not access the WSDL of it's referenced Web Service (it is not available).
    In the first scenario there are two Composites on the same server: CompositeA and CompositeB.
    In SOA Suite 11g there is no possibility to specify the load order for the composites. If CompositeA references CompositeB and if CompositeA is firstly loaded then it can not access the WSDL from CompositeB and the issue occurs.
    In the second situation there is one Composite on the server (CompositeA) which references an external WebService (ExternalWS). If the ExternalWS is not available when the SOA Server is starting then this issue occurs.
    Solution
    Solution 1
    Redeploy the affected Composite into the SOA Server.
    You can do that but it is not advisable in a production environment and in a development environment it will take a lot of time to redeploy the composite manually
    Solution 2
    Copy the abstract WSDL locally into the project.
    This is documented in these articles in our Knowledge base:
    •     Document:1155033.1 Node Restart Cause Composites To Become Unavailable. Response: '503: Service Unavailable'
    •     Document:1151973.1 Boot Order Of The Composites Upon Soa Suite Restart
    Steps:
    o     a. Copy and use the WSDL file in the Project
    o     b. Edit the WebService Adapter
    o     c. Change the "WSDL URL" to point to the WSDL copied into the project
    o     d. Redeploy the Project
    o     e. Make this configuration with all the Projects that have a references with other Web Services
    Solution 3
    Use shared artifacts in Metadata Service (MDS). A WSDL used by more than one composite is a shared artifact per definition. If the WSDL structure is changed (that is not happening frequently in a production environment), you will normally deploy a composite with a new version because overwriting it will break your production environment. Moreover, in a development environment you will need a proper process in place where different developers access the same artifacts within MDS. In addition, when you use external WSDLs (owned by third parties), you will have a proper process/agreement in place to be notified about modifications affecting your applications.
    1.     In order to use shared artifacts a MDS connection must be configured in jDeveloper:
    http://download.oracle.com/docs/cd/E14571_01/integration.1111/e10224/sca_lifecycle.htm#SOASE85488
    2.     Deploy the shared artifacts:
    o     a. Create a JAR profile and include the artifacts to share
    o     b. Create a SOA bundle that includes the JAR profile
    o     c. Deploy the SOA bundle to the application server
    http://download.oracle.com/docs/cd/E14571_01/integration.1111/e10224/sca_lifecycle.htm#SOASE85472
    If the shared artifacts (WSDLs) needed are from other composites this step can be skipped
    3.     Create a new WebService
    o     a. When completing the "WSDL URL" click on "Find existing WSDL's"
    o     b. Select "Resource Palette"
    o     c. Go to the SOA-MDS
    o     d. Select a WSDL from a Composite or the one deployed at step 2.
    Known Restriction 1
    The port and the location for the WSDL reference is not completed automatically by jDeveloper in the composite.xml file. This information must be inserted manually. In Bug:10287325 is raised this issue and it is available for SOA 11gPS2+.
    Known Restriction 2
    Another issue regarding MDS caching is raised in Bug:10218147 - the MDS cache is not refreshed when an artifact is deployed or deleted.
    In order to refresh this cache the server must be restarted. This Bug resolves the issue but in a production environment the server restart will be the preferred option. The reason for that is because in a production environment you never deploy single composites under the same version after artifacts (WSDLs) have been modified.
    The fix for the Bug:10218147 can be an acceptable solution in development environment where redeployment would be quicker than restarting the server.
    Bug:10218147 is available for SOA 11gPS2+.
    To find out more information about how to use Shared Metadata check the following documentation:
    Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite 11g
    41 Deploying SOA Composite Applications
    41.7.3 Deploying and Using Shared Metadata Across SOA Composite Applications in Oracle JDeveloper
    URL:
    http://download.oracle.com/docs/cd/E14571_01/integration.1111/e10224/sca_lifecycle.htm#CACFEAJJ
    Solution 4
    The last solution is to use an UDDI (Universal Description Discovery and Integration).
    Oracle recommends Oracle Service Registry 11g (OSR). The advantage is that you can use OSR and SOA Suite in a Shared WebLogic Domain.
    http://www.oracle.com/technetwork/middleware/registry/overview/index.html
    http://blogs.oracle.com/governance/2010/05/oracle_service_registry_11gr1.html
    To find out more information about Oracle Service Registry check the following documentation:
    Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite 11g
    33 Configuring Service and Reference Binding Components
    33.1.3 Changing the Endpoint Reference and Service Key for Oracle Service Registry Integration
    http://download.oracle.com/docs/cd/E15523_01/integration.1111/e10226/bc_config.htm#SOAAG37248
    Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite 11g
    A BPEL Process Activities and Services
    A.4 Publishing and Browsing the Oracle Service Registry
    http://download.oracle.com/docs/cd/E15523_01/integration.1111/e10224/bp_appx_ref.htm#SOASE85561
    Oracle Fusion Middleware Service Registry 11g
    http://download.oracle.com/otndocs/tech/soa/OSR11gR1ProductDocumentation.pdf
    Solution 5
    In case you have BPM Components in the Composites deployed check the following note:
    Document 1317803.1 Soa Suite Composite Fails To Deploy Upon Restart Of Managed Server
    The issue is caused by the Bug:11822470 SOA SUITE COMPOSITE FAILS TO DEPLOY UPON RESTART OF MANAGED SERVER
    References
    BUG:10218147 - WSDL CHANGES NEED SOA SUITE SERVER RESTART
    BUG:10278478 - WHEN SOA SERVER IS RESTARTED, SOME SOA COMPOSITES COULD NOT LOAD/BE DEPLOYED
    BUG:10287325 - ABSTRACT WSDL NOT AVAILABLE WHEN COMPOSITE STARTED
    BUG:10311698 - WHEN SOA SERVER IS RESTARTED, SOME SOA COMPOSITES CAN NOT BE STARTUP / SHUTDOWN
    BUG:11822470 - SOA SUITE COMPOSITE FAILS TO DEPLOY UPON RESTART OF MANAGED SERVER
    @ BUG:9267312 - MDS ARTIFACTS ARE STILL CACHED AFTER DELETING
    @ BUG:9708488 - AFTER SOA RESTART, ALL PROCESSES HAVE TO BE REDEPLOYED
    @ BUG:9749845 - SCHEMA CACHE STARTS EMPTY AFTER RE-START, BUT NOT AFTER DEPLOYMENT
    NOTE:1151973.1 - Boot Order Of The Composites Upon Soa Suite Restart
    NOTE:1155033.1 - Node Restart Cause Composites To Become Unavailable. Response: '503: Service Unavailable'
    NOTE:1317803.1 - Soa Suite Composite Fails To Deploy Upon Restart Of Managed Server
    Bug 11822470 : SOA SUITE COMPOSITE FAILS TO DEPLOY UPON RESTART OF MANAGED SERVER                    
                   Bug Attributes     
    Type     B - Defect     Fixed in Product Version     11.1.1.6
    Severity     2 - Severe Loss of Service     Product Version     11.1.1.4
    Status     80 - Development to QA/Fix Delivered Internal     Platform     912 - Microsoft Windows (32-bit)
    Created     Mar 1, 2011     Platform Version     2003
    Updated     Oct 12, 2012     Base Bug     N/A
    Database Version     N/A     Affects Platforms     Generic
    Product Source     Oracle          
    Abstract: SOA SUITE COMPOSITE FAILS TO DEPLOY UPON RESTART OF MANAGED SERVER
    *** 03/01/11 08:07 am ***
    Customer has a project consists of two services for interaction with the
    process and one reference to an external service.
    On deployment of the process to an Enterprise Environment, the process works
    as expected. However, upon a restart of the managed server, the process will
    then fail to deploy.
    Workaround used is to redeploy the project again.
    Bug 10278478 : WHEN SOA SERVER IS RESTARTED, SOME SOA COMPOSITES COULD NOT LOAD/BE DEPLOYED                    
                   Bug Attributes     
    Type     B - Defect     Fixed in Product Version     
    Severity     2 - Severe Loss of Service     Product Version     11.1.1.3.0
    Status     92 - Closed, Not a Bug     Platform     226 - Linux x86-64
    Created     Nov 10, 2010     Platform Version     RED HAT ENTERPRISE LINUX 5
    Updated     Dec 10, 2010     Base Bug     N/A
    Database Version     N/A     Affects Platforms     Generic
    Product Source     Oracle          
    Abstract: WHEN SOA SERVER IS RESTARTED, SOME SOA COMPOSITES COULD NOT LOAD/BE DEPLOYED
    Detailed Problem Description
    ====================
    When a SOA composite get dependencies with other SOA composites located on the same server, if the dependency is still not loaded, the SOA composite will not be loaded, and there is no possibilities for Entreprise Manager console to restart it (EM crash). The only way to make it working is to redeployed the SOA composite from jDeveloper that is not acceptable in a production environment.
    The composite that has the issue cannot be start-up or shut-down, although the Enterprise Manager do not show any errors with it.

  • Http problems under load

    Running WL 51 w. SP3, Solaris 2.7, java 1.2.1, and using loadrunner to
    simulate 160 users doing http GETs directly to the weblogic http server
    (port 8001). There are a number of errors showing up in the weblogic
    log that I don't understand.
    First, I get a few of:
    Tue Aug 08 18:13:45 GMT+00:00 2000:<E> <HTTP> Servlet request terminated
    with IOException:
    java.net.ProtocolException: Didn't meet stated Content-Length, wrote:
    '24576' bytes instead of stated: '99596' bytes.
    at java.lang.Throwable.fillInStackTrace(Native Method)
    at java.lang.Throwable.fillInStackTrace(Compiled Code)
    at java.lang.Throwable.<init>(Compiled Code)
    at java.lang.Exception.<init>(Compiled Code)
    at java.io.IOException.<init>(Compiled Code)
    at java.net.ProtocolException.<init>(Compiled Code)
    at
    weblogic.servlet.internal.ServletOutputStreamImpl.finish(Compiled Code)
    at
    weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled
    Code)
    at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled
    Code)
    at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
    at weblogic.kernel.ExecuteThread.run(Compiled Code)
    Then a bunch of:
    <W> <ListenThread> Connection rejected: 'Login timed out after: '5000'
    ms on socket ...
    followed by:
    Tue Aug 08 20:12:30 GMT+00:00 2000:<E> <Posix Performance Pack> Failure
    in processSockets()
    java.net.SocketException: Connection reset by peer: Connection reset by
    peer
    at java.net.SocketInputStream.socketRead(Native Method)
    at java.net.SocketInputStream.socketRead(Compiled Code)
    at java.net.SocketInputStream.read(Compiled Code)
    at weblogic.socket.PosixSocketMuxer.processSockets(Compiled
    Code)
    at
    weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:23)
    at weblogic.kernel.ExecuteThread.run(Compiled Code)
    There are 4 client machines each running 40 clients, and these errors
    seem to only affect one of the client machines.
    If I run the test just using that machine, it runs fine - it only seems
    to show up under load.
    I don't know if it's a coincidence or not, but the numbers seem too
    regular - I get (in order):
    1480 Connection rejected: messages
    40 <Posix Performance Pack> Failure in processSockets()
    40 Connection rejected: messages
    80 <Posix Performance Pack> Failure in processSockets()
    240 Connection rejected: messages
    30 <Posix Performance Pack> Failure in processSockets()
    Thoughts? Solutions?

    Chris Fullerton wrote:
    >
    Running WL 51 w. SP3, Solaris 2.7, java 1.2.1, and using loadrunner to
    simulate 160 users doing http GETs directly to the weblogic http server
    (port 8001). There are a number of errors showing up in the weblogic
    log that I don't understand.
    First, I get a few of:
    Tue Aug 08 18:13:45 GMT+00:00 2000:<E> <HTTP> Servlet request terminated
    with IOException:
    java.net.ProtocolException: Didn't meet stated Content-Length, wrote:
    '24576' bytes instead of stated: '99596' bytes.In the last week I tried to do same things. I wrote my own tester (a
    simple http-client,
    multi-thread etc.) to see whats happen' when 10000 clients POSTs an
    HTTP-form. And of course
    I got the same exception, because I forgot to set the correct
    Content-length in the CLIENT SIDE.
    My first HTTP-header looks like this:
    POST /pqr HTTP/1.0<CRLF>
    Content-type: application/x-www-form-urlencoded<CRLF>
    Content-length: 17<CRLF><CRLF>
    xml=<blabla></blabla>
    And it was really bad: I counted only the xml-like string, instead of
    the full data (incl. "xml=")
    When I corrected it was goood!
    So, I think the source of Your problem isn't in the WS-side: try to
    check your client...
    And Your client can't close the connection before sending all data!!!
    Best Regards/Üdvözlettel:
    Oláh, Károly
    i2c Systems & Consulting

  • Cache and/or Connection problems under load

    I have a Kodo web app that's been running just fine in
    production for many months now. However, recently the web
    traffic has shot up by a huge amount, literally overnight.
    But unfortunately, it's caused the app to fail very ungracefully
    under the strain.
    It's been a crazy few days, and I haven't been able to do
    very much analysis because of higher priorities. But from
    what I have been able to gleen, it now looks like Kodo is
    the most likely culprit. From what I've read in other messages
    here, it appears others may have been experiencing similar
    problems.
    My environment: Redhat Linux 8, Postgres 7.3.4 with the
    included JDBC3 driver, Apache 1.3.x, Tomcat 4.1.x and the
    webapp connector. Similar behavior was seen with Apache 2.x,
    Tomcat 4.1.x and the JK2 connector (that was on the new machine
    I setup to handle the new traffic, which, of course, died the
    night before).
    As I mentioned, this app has been running reliably for
    months with no problems. But when placed under heavy load,
    it appears to get into some sort of pathological state where
    it slows down dramatically (asymptotically?) to the point where
    it's effectively locked up. In one case, where the app was
    left running for several hours in this state, requests were
    taking 90 minutes to complete (normal is 1-5 seconds).
    From what I can deduce, there seem to be four things
    going on, three of which have been mentioned in recent threads
    here:
    1) Excessive memory consumption. When the app is
    operating normally, I see fairly flat memory usage for
    the JVM process. Under load, the JVM steadily expands
    until it hits its heap limit. I've gotten OutOfMemory
    exceptions with a heap size of 350MB, which should be plenty.
    2) Level 2 cache locking issues. I've seen dozens of
    threads waiting on a lock in the DataCache code. Not sure
    if there's a deadlock happening here or just that the
    threads are waiting on a lock that's being held for a long time.
    3) Database Connection leaks or contention. I see threads
    spinning in the DataSource code trying to get a connection.
    I also see dozens of connections from the Postgres side which
    seem to be sitting idle, but in the middle of a transaction.
    When things get bad, I also see exceptions being thrown because
    of timeouts waiting for a connection to become available. It's
    a web app, PMs should not be tied up for more than a few seconds.
    4) CPU usage pegged or nearly so for the JVM. I suspect
    this is related to #3. Something very bad is going on here.
    If I stop all inbound requests to the JVM when it's in this
    bad state, it will continue to burn CPU at 90%+ for a very
    long time. I think it will eventually finish what it's doing,
    but I haven't had the luxury of waiting for it. It's definitely
    not a linear slowdown proportional to the load.
    Attached are my kodo.properties file and some thread stack
    traces along with some comments. Any advice would be greatly
    appreciated. This is not a complicated app nor am I doing
    anything unusual. It doesn't seem logical that Kodo could
    breakdown so dramatically under load, so I'm hoping it's some
    sort of interaction thing that I can work around.
    Thanks.
    Ron Hitchens {mailto:[email protected]} RonSoft Technologies
    (510) 494-9597 (Home Office) http://www.ronsoft.com
    (707) 924-3878 (fax) Bit Twiddling At Its Finest
    "Born with a broken heart" -Kenny Wayne Shepard

    Please read prior posts regarding level 2 cache. It is unusable under stress
    as far I am concerned. Basically entire cache gets locked on any database
    read. Makes it very unscalable
    Are you using 2.5.3? It will request a connection from a pool every time it
    resolves reference to a PC even if it is cached in PM and therefore Kodo
    does not need to read any. As result if you iterate over 100 objects in your
    query and for each object resolve reference to another object (always the
    same) kodo will request 100 database connections from the pool (and note
    they issue rollback on every time they return a connection to the pool so
    getting connection might be fairly expensive)
    In conjunction with level 2 cache contention this causes application to go
    into a stupor.
    Try to go back to 2.5.2 (or may be 2.5.4 they promised in the near future
    with a workaround) or use "persistent-manager" connection retention if you
    discard PM after each HTTP invocation - it will take care of connection
    pooling issue. As far as L2 cache I was unable to find any work around so
    far - see if you might be better of without cache. You might if your object
    graph is not very complex
    "Ron Hitchens" <[email protected]> wrote in message
    news:[email protected]...
    >
    I have a Kodo web app that's been running just fine in
    production for many months now. However, recently the web
    traffic has shot up by a huge amount, literally overnight.
    But unfortunately, it's caused the app to fail very ungracefully
    under the strain.
    It's been a crazy few days, and I haven't been able to do
    very much analysis because of higher priorities. But from
    what I have been able to gleen, it now looks like Kodo is
    the most likely culprit. From what I've read in other messages
    here, it appears others may have been experiencing similar
    problems.
    My environment: Redhat Linux 8, Postgres 7.3.4 with the
    included JDBC3 driver, Apache 1.3.x, Tomcat 4.1.x and the
    webapp connector. Similar behavior was seen with Apache 2.x,
    Tomcat 4.1.x and the JK2 connector (that was on the new machine
    I setup to handle the new traffic, which, of course, died the
    night before).
    As I mentioned, this app has been running reliably for
    months with no problems. But when placed under heavy load,
    it appears to get into some sort of pathological state where
    it slows down dramatically (asymptotically?) to the point where
    it's effectively locked up. In one case, where the app was
    left running for several hours in this state, requests were
    taking 90 minutes to complete (normal is 1-5 seconds).
    From what I can deduce, there seem to be four things
    going on, three of which have been mentioned in recent threads
    here:
    1) Excessive memory consumption. When the app is
    operating normally, I see fairly flat memory usage for
    the JVM process. Under load, the JVM steadily expands
    until it hits its heap limit. I've gotten OutOfMemory
    exceptions with a heap size of 350MB, which should be plenty.
    2) Level 2 cache locking issues. I've seen dozens of
    threads waiting on a lock in the DataCache code. Not sure
    if there's a deadlock happening here or just that the
    threads are waiting on a lock that's being held for a long time.
    3) Database Connection leaks or contention. I see threads
    spinning in the DataSource code trying to get a connection.
    I also see dozens of connections from the Postgres side which
    seem to be sitting idle, but in the middle of a transaction.
    When things get bad, I also see exceptions being thrown because
    of timeouts waiting for a connection to become available. It's
    a web app, PMs should not be tied up for more than a few seconds.
    4) CPU usage pegged or nearly so for the JVM. I suspect
    this is related to #3. Something very bad is going on here.
    If I stop all inbound requests to the JVM when it's in this
    bad state, it will continue to burn CPU at 90%+ for a very
    long time. I think it will eventually finish what it's doing,
    but I haven't had the luxury of waiting for it. It's definitely
    not a linear slowdown proportional to the load.
    Attached are my kodo.properties file and some thread stack
    traces along with some comments. Any advice would be greatly
    appreciated. This is not a complicated app nor am I doing
    anything unusual. It doesn't seem logical that Kodo could
    breakdown so dramatically under load, so I'm hoping it's some
    sort of interaction thing that I can work around.
    Thanks.
    Ron Hitchens {mailto:[email protected]} RonSoft Technologies
    (510) 494-9597 (Home Office) http://www.ronsoft.com
    (707) 924-3878 (fax) Bit Twiddling At Its Finest
    "Born with a broken heart" -Kenny Wayne Shepard
    With cahce enabled, 2.5.3
    Here the app had recently slowed down and then effectively locked up.
    There where many outstanding web requests that were not receiving output.
    At this point most threads seemed to be waiting at the same location.
    There were a large number of active database connections and most of
    them had open transactions (according to pg_stat_activity). The app
    was not responding to any web requests.
    It would seem that db transactions had been started, then the thread
    got stuck for a long time on a synchronization lock in the cache lookup.
    Below are two randomly chosen thread stack dumps.
    Thread-72[1] where
    [1] java.lang.Object.wait (native method)
    [2] java.lang.Object.wait (Object.java:429)
    [3]oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquir
    e (WriterPreferenceReadWriteLock.java:169)
    [4]com.solarmetric.kodo.runtime.datacache.AbstractCacheImpl.acquireReadLock
    (AbstractCacheImpl.java:384)
    [5]com.solarmetric.kodo.runtime.datacache.TimedDataCache.acquireReadLock
    (TimedDataCache.java:256)
    [6] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:595)
    [7] com.solarmetric.kodo.runtime.StateManagerImpl.loadFields(StateManagerImpl.java:2,330)
    [8] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:897)
    [9] com.europeasap.data.City.jdoGetname (null)
    [10] com.europeasap.data.City.getName (City.java:39)
    [11] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:922)
    [12] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
    [13] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
    [14] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
    [15] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
    [16] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [17]
    org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
    [18] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
    [19] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
    [20] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
    [21] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
    [22] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
    [23]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [24] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [25] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [26] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    [27]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [28] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    [29]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [30] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [31] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [32] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
    [33] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
    [34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [35] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
    [36]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [37] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
    [38]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [39] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [40] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [41] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
    [42]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [43] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [44] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [45] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
    [46] org.apache.catalina.connector.warp.WarpConnection.run (null)
    [47] java.lang.Thread.run (Thread.java:534)
    Thread-64[1] where
    [1] java.lang.Object.wait (native method)
    [2] java.lang.Object.wait (Object.java:429)
    [3]oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquir
    e (WriterPreferenceReadWriteLock.java:169)
    [4]com.solarmetric.kodo.runtime.datacache.AbstractCacheImpl.acquireReadLock
    (AbstractCacheImpl.java:384)
    [5]com.solarmetric.kodo.runtime.datacache.TimedDataCache.acquireReadLock
    (TimedDataCache.java:256)
    [6] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:595)
    [7] com.solarmetric.kodo.runtime.StateManagerImpl.loadField(StateManagerImpl.java:2,248)
    [8] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:899)
    [9] com.europeasap.data.HotelPrices.jdoGetseasonalPrices (null)
    [10] com.europeasap.data.HotelPrices.normalizeIndex(HotelPrices.java:113)
    [11] com.europeasap.data.HotelPrices.getCost (HotelPrices.java:45)
    [12] com.europeasap.logic.CostHelper.findLowestHotel(CostHelper.java:181)
    [13] com.europeasap.logic.CostHelper.computeBasePackageCost(CostHelper.java:297)
    [14] com.europeasap.logic.CostHelper.computeFinalPackageCost(CostHelper.java:246)
    [15] com.europeasap.form.CustomerBookingForm.updateDisplayCosts(CustomerBookingForm.java:1,440)
    [16] com.europeasap.form.CustomerBookingForm.updateCustomizeDisplayInfo(CustomerBookingForm.java:1,407)
    [17] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:68)
    [18] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
    [19] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [20]
    org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
    [21] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
    [22] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
    [23] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
    [24] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
    [25] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
    [26]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [27] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [28] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [29] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    [30]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [31] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    [32]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [33] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [34] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [35] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
    [36] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
    [37]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [38] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
    [39]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [40] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
    [41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [42] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [43] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [44] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
    [45]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [46] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [47] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [48] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
    [49] org.apache.catalina.connector.warp.WarpConnection.run (null)
    [50] java.lang.Thread.run (Thread.java:534)
    while running slow, 2.5.3
    At this point, the app had been running several hours normally, then
    apparently slowed down and locked up while I was away. When looking
    at the app threads and database activity, everything appeared idle.
    No transactions seemed to be open in the db. But the app was not
    behaving normally. Web requests that did not make use of JDO worked
    fine (but slow). But requests that hit the db either blocked or were
    very slow to respond.
    Looking back at the log, there had been a large number of requests
    that threw exceptions because they could not get a connection within
    five seconds.
    Most threads were idle, waiting on read, but some were in the state
    shown by the following two stack dumps. Unlike the cache threads above,
    they did not seem to be waiting for a lock to be granted, they seemed
    to be spinning in the connection management code. Apparently trying
    to get a connection. I suspended and resumed the same thread repeatedly
    and it always seemd to be doing the same thing. Single stepping was
    very difficult because the debugger was slow to respond, apparently
    because of other threads also busy spinning.
    Postgres indicated that there where lots of connections open and
    that they were all idle, so there should not have been a shortage
    of connections in the pool. There are two mysteries here: 1) why
    can't this thread get a connection? and 2) Why is it busy spinning?
    Thread-56[1] where
    [1]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
    reStatement (PreparedStatementCache.java:184)
    [2]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
    reStatement (PreparedStatementCache.java:169)
    [3] com.solarmetric.datasource.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:199)
    [4]com.solarmetric.kodo.impl.jdbc.schema.dict.AbstractDictionary.isClosed
    (AbstractDictionary.java:1,912)
    [5]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnectionFromFact
    ory (SQLExecutionManagerImpl.java:186)
    [6] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnection(SQLExecutionManagerImpl.java:147)
    [7]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.newSQLExecutionManag
    er (JDBCStoreManager.java:828)
    [8]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getSQLExecutionManag
    er (JDBCStoreManager.java:714)
    [9]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getDatastoreConnecti
    on (JDBCStoreManager.java:287)
    [10]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.getDatastoreCon
    nection (DataCacheStoreManager.java:465)
    [11] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:591)
    [12] com.solarmetric.kodo.runtime.StateManagerImpl.loadFields(StateManagerImpl.java:2,330)
    [13] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:897)
    [14] com.europeasap.data.City.jdoGetname (null)
    [15] com.europeasap.data.City.getName (City.java:39)
    [16] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:922)
    [17] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
    [18] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
    [19] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
    [20] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
    [21] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [22]
    org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
    [23] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
    [24] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
    [25] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
    [26] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
    [27] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
    [28]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [29] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [30] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [31] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    [32]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [33] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    [34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [35] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [36] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [37] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
    [38] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
    [39]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [40] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
    [41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [42] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
    [43]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [44] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [45] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [46] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
    [47]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [48] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [49] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [50] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
    [51] org.apache.catalina.connector.warp.WarpConnection.run (null)
    [52] java.lang.Thread.run (Thread.java:534)
    Thread-56[1] where
    [1]com.solarmetric.datasource.DataSourceImpl$AbstractPool.findConnection
    (DataSourceImpl.java:826)
    [2] com.solarmetric.datasource.DataSourceImpl$AbstractPool.getConnection(DataSourceImpl.java:605)
    [3] com.solarmetric.datasource.DataSourceImpl.getConnection(DataSourceImpl.java:363)
    [4] com.solarmetric.datasource.DataSourceImpl.getConnection(DataSourceImpl.java:356)
    [5]com.solarmetric.kodo.impl.jdbc.runtime.DataSourceConnector.getConnection
    (DataSourceConnector.java:63)
    [6]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnectionFromFact
    ory (SQLExecutionManagerImpl.java:185)
    [7] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnection(SQLExecutionManagerImpl.java:147)
    [8]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.newSQLExecutionManag
    er (JDBCStoreManager.java:828)
    [9]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getSQLExecutionManag
    er (JDBCStoreManager.java:714)
    [10]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getDatastoreConnecti
    on (JDBCStoreManager.java:287)
    [11]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.getDatastoreCon
    nection (DataCacheStoreManager.java:465)
    [12]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.initialize
    (DataCacheStoreManager.java:519)
    [13] com.solarmetric.kodo.runtime.StateManagerImpl.loadInitialState(StateManagerImpl.java:215)
    [14]com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectByIdFilter
    (PersistenceManagerImpl.java:1,278)
    [15] com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:1,179)
    [16]com.solarmetric.kodo.runtime.datacache.query.CacheAwareQuery$CachedResultLis
    t.get (CacheAwareQuery.java:432)
    [17] java.util.AbstractList$Itr.next (AbstractList.java:421)
    [18] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:919)
    [19] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
    [20] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
    [21] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
    [22] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
    [23] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [24]
    org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
    [25] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
    [26] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
    [27] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
    [28] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
    [29] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
    [30]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [31] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [32] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [33] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    [34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [35] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    [36]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [37] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [38] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [39] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
    [40] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
    [41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [42] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
    [43]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [44] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
    [45]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:641)
    [46] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [47] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [48] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
    [49]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [50] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [51] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [52] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
    [53] org.apache.catalina.connector.warp.WarpConnection.run (null)
    [54] java.lang.Thread.run (Thread.java:534)
    With cache disabled 2.4.3
    This run was an accident. I inadvertently ran the app with the older
    2.4.3 version of Kodo, with the cache disabled. This one got into trouble
    almost immediately. It also seemed to lockup with lots of opentransactions
    in the db. It's also interesting that these two threads also seem to be
    hanging around the same method as in 2.5.3.
    Thread-63[1] where 0x9f9
    [1]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
    reStatement (PreparedStatementCache.java:184)
    [2] com.solarmetric.datasource.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:377)
    [3]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.prepareStatementInter
    nal (SQLExecutionManagerImpl.java:807)
    [4]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executePreparedQueryI
    nternal (SQLExecutionManagerImpl.java:761)
    [5]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQueryInternal
    (SQLExecutionManagerImpl.java:691)
    [6] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQuery(SQLExecutionManagerImpl.java:372)
    [7] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQuery(SQLExecutionManagerImpl.java:356)
    [8] com.solarmetric.kodo.impl.jdbc.ormapping.ClassMapping.loadByPK(ClassMapping.java:950)
    [9] com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.initialize(JDBCStoreManager.java:263)
    [10] com.solarmetric.kodo.runtime.StateManagerImpl.loadInitialState(StateManagerImpl.java:174)
    [11]com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectByIdFilter
    (PersistenceManagerImpl.java:1,023)
    [12] com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:942)
    [13] com.solarmetric.kodo.impl.jdbc.ormapping.OneToOneMapping.load(OneToOneMapping.java:147)
    [14] com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.load(JDBCStoreManager.java:375)
    [15] com.solarmetric.kodo.runtime.StateManagerImpl.loadField(StateManagerImpl.java:2,035)
    [16] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:720)
    [17] com.europeasap.data.CityMarkup.jdoGetcity (null)
    [18] com.europeasap.data.CityMarkup.getCity (CityMarkup.java:30)
    [19] com.europeasap.logic.CostHelper.getCityMarkup (CostHelper.java:81)
    [20] com.europeasap.logic.CostHelper.computeBasePackageCost(CostHelper.java:289)
    [21] com.europeasap.logic.CostHelper.computeFinalPackageCost(CostHelper.java:246)
    [22] com.europeasap.form.CustomerBookingForm.updateDisplayCosts(CustomerBookingForm.java:1,440)
    [23] com.europeasap.form.CustomerBookingForm.updateCustomizeDisplayInfo(CustomerBookingForm.java:1,407)
    [24] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:68)
    [25] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
    [26] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [27]
    org.apache.struts.action.ActionServlet.doPost (ActionServlet.java:510)
    [28] javax.servlet.http.HttpServlet.service (HttpServlet.java:760)
    [29] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
    [30] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
    [31] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
    [32] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
    [33]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
    eNext (StandardPipeline.java:643)
    [34] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
    [35] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
    [36] org.apache.catal

  • PDFMaker Server cannot be loaded

    Acrobat 7.0.1
    Word 2003
    XP SP2
    PDFMaker is unavailable from Word, PowerPoint, Excel, and Publisher. I get the error message "PDFMaker Server cannot be loaded."
    Running the Intaller in repair mode has no effect.
    PDFMaker is NOT listed as disabled (About Word...)
    I have registered the PDFMakerX.dll file with the registry
    There is NOT a PDFMaker.dot file listed in the global templates area of Word.
    What is the workaround?
    All I want is to be able to create tagged PDFs, with TOC links, etc. preserved in the PDF. Much obliged for any help...

    I can confirm Janey's solution in respect to Acrobat 9.0 and Office 2007.
    To restore PDFMaker functionality to Microsoft Office 2007 under Vista, I did the following:
    1/
    With Regedit, I tried to locate the key
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\AcroDist.exe
    2/
    It wasn't present. So I created the key 'AcroDist.exe' with 'EDIT/NEW/KEY'.
    3/
    I clicked on the (Default) entry and changed the value to:
    C:\Program Files\Adobe\Acrobat 9.0\Acrobat\AcroDist.exe
    4/
    With 'EDIT/NEW/String Value' added the value name 'Path' which I then set to:
    C:\Program Files\Adobe\Acrobat 9.0\Acrobat\
    5/
    Regedit was closed and immediately I could use PDFMaker in Office 2007.
    Don't ask me how this important registry entry went missing - no amount of re-installing anything made any difference.  In your case, if your Acrobat location is different, please adjust the above data accordingly.

  • Composites in "Unknown" state after managed soa server restart

    Hi,
    I am using SOA suite 11.1.1.6 version on 64 bit OEL Linux. I am getting following error in managed server output log files:
    Feb 20, 2013 5:59:32 AM IST> <Error> <oracle.integration.platform> <SOA-20003> <Unable to register service.
    oracle.fabric.common.FabricException: Error in getting XML input stream: oramds:/deployed-composites/WirelessEnterprise/svcEnrichSEARequest_rev1.0/svcEnrichSEARequest.wsdl: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/deployed-composites/WirelessEnterprise/svcEnrichSEARequest_rev1.0/svcEnrichSEARequest.wsdl does not exist.
    Though this composites gets loaded, the composites dependent on this composite goes into "Unknown" state. Other composites are properly loaded on server restart.
    Please let us know if anybody has any solution to this problem.

    Hi,
    Thanks for your help.
    we have narrowed down to problem now.
    The actual problem occurs while redeploying same composite on same revision.
    E.g 3 composites A, B and C are deployed on and in "Up" state.
    B And C composites depends on A.
    B composite composite.xml has following entry
    <import namespace="http://xmlns.oracle.com/A"
    location="oramds:/deployed-composites/Enterprise/A_rev1.0/A.wsdl"
    importType="wsdl"/>
    same entry exist for C composite.xml as well.
    Now whenever composite A is redeployed with same revision (1.0 in our case), and managed servers are restarted, B and C composites goes in to "Unknown" state.
    Moving A.wsdl to /apps/wsdl/* in MDS works but it duplicates the A.wsdl file beacause it also exists in MDS @ /deployed-composites/Enterprise/A_rev1.0/A.wsdl
    Please let us know if anyone has any solution.
    Thanks,
    Rupesh.

  • JNDI - rebind required after server restart

    We are using a third party JMS Provider (we have to use this provider implementation as opposed to WLS Provider), which requires a JNDI implementation. We can use the Sun File JNDI implementation, but seeing as WLS is in the environment, binding the JNDI names into WLS seemed the natural thing to do. This works great, except that we have to bind after each server start up (as the WLS JNDI naming is not persistent across server restart).
    I was wondering if anyone had any 'neat' tricks or ideas around this, aside from startup class.

    JMS has a ForeignJMSProvider config option to do this I believe. 9.0 will also support this more generally.

  • Podcast will not load in safari from server, but will load in other Apps

    Podcast will not load in safari from iWeb Page from server, but will load in Firefox and other web apps.
    Here the problem: I have a podcast page made in iWeb. It is setting on a server with GoDaddy. When I try load page with Safari, I get where the Quicktime Playing Bars are a small Quicktime Logo w/? mark and will not download or play. But in Firefox, My iPad and in Windows side it will load and play. Also it play on a friends Macbook Pro in Safari. (Note it will play in my Safari after Firefox has play -it "very strange to me" like Firefox will open some port, but it will not stay open if quit Safari re-booth-it) I have remove the .plist for Quicktime & Safari. And clean out the Cache for Safari. Also have re-set the Firewall setting to Enable Stealth Mode only. Re-load the site to GoDaddy. This is happening to both Mac's in my network and in all users accounts.
    My Network is a A&TT with a 2Wire w/wrieless turn off. going into and Airport Extreme Base Station w/N generation network. One computer is connected with and ethernet cable to the Airport extreme base station the other is a wireless contection.
    I do feel this and network issue and I can't seem to solve-it. How can I get Safari to open the port and download and play-it on my network?

    We've only discover this yesterday. One of our listeners sent me an e-mail about-it. With a little more research this morning. I've discover that everybody I've had check-it is have the same problem only with Safari 4.0.5. So I'm starting to feel that this is an Apple Safari Bug in 4.0.5 for the Mac only. It seems to be OK everywhere else on other devices. I agree in most cases this is not a problem. But when have older people who do not download into iTunes around the world it is to them. We have 250 or more listeners a week. This is a Church podcast.
    For right now we are planing to post a message on the site saying you will not be able to listen to the podcast in Safari, but can in Firefox and ect.

  • Updating policies and Policy Database or Policy Updates without server restart

    Hi again,
    I have implemented the flash access reference implementation server including setting up all example database tables (so we have a central database server).  I have started updating policies for the application "whitelist" features and have run into some questions below.
    The Protecting Content documentation states:
              "If your license server has access to a database for storing policies, you can retrieve the updated policy from the database and call LicenseRequestMessage.setSelectedPolicy()"
    This leads to two questions:
    1)  The reference implementation doesn't store any Policies in the database (it stores them on the filesystem in the Resources Directory).     I can not find any documentation or examples of how to "retrieve the updated policy from the database".     Can somebody please point me to an example or documentation on how to store policies in the database so I can simply LicenseRequestMessage.setSelectedPolicy()?
    2)  As the reference implementation stores policies on the disk I had to use policy update lists when I whitelisted an AIR app.     However, the reference server did not honor the policy update list till I restarted the flash access reference server.      How can I notify the flash access server of the change to the update policy list without restarting the server?
    Thank you,
    -R

    Hello,
    Please see my answers below.
    Answer 1>
    If you want to store the binary representation of the policy in the database, you can invoke Policy.getBytes() and store the returned byte[] as a blob in the database. For details on how to store such binary data in a database, you should consult the documentation for you database. Then, to rebuild a policy that had been stored in a database, you can obtain the byte[] stored in the database, and invoke new Policy(byte[]) with it to reconstruct that policy instance.
    See the following javadocs:
          * Retrieves an encoded byte array representation of the policy.
          * <p>If this <code>Policy</code> was created from an empty constructor, the revision number
          * will be <code>1</code>.  If this was an existing policy and changes were made to the policy,
           * the revision number will be incremented by one. Otherwise, the revision number will not change.</p>
          * <p>The validity of the policy is first checked before encoding into a byte array.  Encoding will fail
          * if {@link #checkValidity()} throws an exception.</p>
           * @return A serialized representation of the policy.
          * @throws PolicyModificationException The <code>Policy</code> cannot be serialized because it was created with a newer version and was modified.
          * @throws PolicyException The <code>Policy</code> is missing required fields or contains conflicting entries.
          * @throws EncodingException Unable to encode the <code>Policy</code> to a byte array.
          public byte[] getBytes() throws PolicyException, EncodingException;
          * This constructor parses an existing policy from the given byte array.
           * @param policy The policy from which to construct this <code>Policy</code>.
          * @throws EncodingException Unable to parse the byte array into a <code>Policy</code>.
          * @throws PolicyException The <code>Policy</code> is missing required fields or contains conflicting entries.
          public Policy( byte[] policy ) throws PolicyException, EncodingException;
    Answer 2>
    In order to implement automatic configuration updates of policy update lists on the flash access server, you will need to tweak the Reference Implementation code, to monitor the file update state and re-load the PolicyUpdateList if it changes. The code snippet below, taken from the com.adobe.flashaccess.refimpl.util.ParamsReader.buildHandlerConfiguration() method, should serve as an example of how the PolicyUpdateList is loaded and then set on the HandlerConfiguration instance for it to take effect for a particular license acquisition request. In the case of the reference implementation, this code (below) is only invoked once, during server initialization. That is why a server restart is necessary for subsequent changes to take effect. However, nothing prevents this code form being invoked elsewhere, and that implementation is entirely upto the customer, to achieve the desired outcome.
          PolicyUpdateList pul = PolicyUpdateListFactory.loadPolicyUpdateList(pulInputStream);
          if (pul != null)
                X509Certificate issuerCert = null;
                if (hsmEnabled)
                      issuerCert = getHSMCertificate(props, hsmUtils, "RevocationList.verifySignature.X509Certificate");
                else
                      String pulSignatureCertFile = props.getProperty("RevocationList.verifySignature.X509Certificate");
                      if (pulSignatureCertFile != null)
                            InputStream pulSignerInputStream = new FileInputStream(resourcesDir + pulSignatureCertFile);
                            issuerCert = CertificateFactory.loadCert(pulSignerInputStream);
                if (issuerCert != null)
                      pul.verifySignature(issuerCert);
                      context.setPolicyUpdateList(pul);
    Regards,
    Safdar

Maybe you are looking for

  • 1st gen Ipod touch update?

    Hi Folks, My son has given me his old Ipod touch 1st Gen.  I have updated this to IOS 3.1 but need IOS 4 and above to run some (most) of the Apps available.  Can this be done and can you point me to a link if available please? Thanks.

  • IPod will not boot up

    Hi, I've been having a problem booting up my iPod, as in getting it to turn on, for about a week now. Everytime I go to use it, the apple logo flashes and I hear the battery starting to turn on; however, it abruptly stops, and the sad iPod face appea

  • How to edit cell in a Data Grid

    Hai     I have pasted the mxml below.    I need how to edit a cell in a datagrid.   1. Run the mxml, enter the values in the textbox and click add button.   2. then click AND button, now u can see the values displayed in the grid. 3. Then again chang

  • Java - InputStream - Weird Characters in the Output!

    Hi, I have a software which outputs lines of numbers one by one. In another java program, I use BufferedReader as follows: BufferedReader input = new BufferedReader(new InputStreamReader(pr.getInputStream(),"US-ASCII")); where pr is the process which

  • "Enter document number error" in BAPI_SALESORDER_CHANGE

    Hi, I am trying to change the sales order thorugh BAPI_SALEORDER_CHANGE as follows, CALL FUNCTION 'BAPI_SALESORDER_CHANGE' EXPORTING salesdocument = <fs_order_alv_aux>-vbeln order_header_in = e_bapisdh1 order_header_inx = e_bapisdh1x simulation = p_c