BEA-002900 Initializing self-tuning thread pool   HANGS

Hi All,
Trying to start a WLS 10.3.5 instance using jrockit and it is hanging on:
Apr 6, 2013 2:08:39 AM CDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
end of trace
I took a thread dump and see this:
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=3810 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xa0c21480[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xa0c21480[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JFR request timer" id=16 idx=0x40 tid=3811 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xa0c20b28[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0xa0c20b28[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
===== END OF THREAD DUMP ===============
Done:
1). Cleared TMP and CACHE directories and still does not solve the issue.
Any ideas are appreciated.
thanks

Hi,
we had same hang issue. Unfortunately we run behind VM.
It was an issue with securerandom generation.
We solved the issue as described here http://stackoverflow.com/a/2564406/2098832
Also deleted the <server>/tmp directory before.
Hope this helps.
Casey

Similar Messages

  • Self tuning Thread Weblogic 9.2

    Hi, I'm using Weblogic 9.2.
    Weblogic 9.2 has a self tuning thread, which adapts itself in order to optimize the throughput.
    The problem is that anytime you restart the server, the thread pool restarts from a low number of threads and takes some time to reach the optimal number of threads again. This is a major issue, since it means that whenever you restart the server, you cannot handle the same amount of traffic as before the restart.
    Is there a way to tell Weblogic to start with a certain number of threads in the thread pool?
    Thanks in advance

    Thanks for sharing this usefule information about self-tuning thread.
    We are using WebLogic 10.2, and also have some issues with self-tuning threads. I suspect that we need to adjust either min or max size for the self-tuning thread tool. I checked our config.xml, there is no information set there. Does this mean we use the default value from WebLogic? If yes, can you tell me what the default values are so I can decide if they are proper or not for our application.
    Thanks,
    David,

  • Self tuning threads are not releasing even though server running fine.

    One of the cluster instance is running fine and serving the requests with good response but there self tuning threads are not releasing .I am not seeing any stuck threads in logs and no errors. why threads are not releasing.How we can troubleshoot this issue. May I know what are the cause of this.

    Arun thanks for your quick response
    Here is the threads details
    Active Execute Threads 192     
    Execute Thread Total Count 222
    ExecuteThreadIdleCount 190     
    QueueLength 0
    PendingUserRequestCount 0
    completed request count 36659285
    HoggingThreadCount     1     
    StandbyThreadCount 30     
    Throughput 7.5     
    More number of threads are waiting for lock
    Here is the sample thread dump.
    "[ACTIVE] ExecuteThread: '218' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@faf757 WAITING
         java.lang.Object.wait(Native Method)
         java.lang.Object.wait(Object.java:474)
         weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:156)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:177)
    "[ACTIVE] ExecuteThread: '217' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@12c34ef WAITING
         java.lang.Object.wait(Native Method)
         java.lang.Object.wait(Object.java:474)
         weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:156)
    some threads are blocked
    ExecuteThread: '9' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@1f39d37 BLOCKED
         weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:93)
         weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
         weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
         weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
    "ExecuteThread: '8' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@1f39d37 BLOCKED
         weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:93)
         weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
         weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
         weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:177)
    "[STANDBY] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1bd18aa WAITING
         java.lang.Object.wait(Native Method)
         java.lang.Object.wait(Object.java:474)
         weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:156)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:177)

  • Why do we need work managers if the self-tuning pool is not limited ?

    Hi all,
    AFAIK, the self-tuning thread pool in WLS 10.3 is not limited in size, meaning it can grow indefinitely.
    On the other hand, work managers only make sense when there's a shortage in resources, and we need to decide what are the priorities to get a thread and CPU time.
    So if, for example, I have a system that's only 20% busy even on the heaviest load, the work managers will never "kick in" actually, since there's "enough for everyone".
    Thanks for sharing your thoughts!

    If you have more than a single application deployed and only have the default work manager, all apps compete equally and are also equally unable to be tuned in their thread usage.
    If you instead have a unique work manager for each app, you can limit each app's consumption of resources - if needed - or alter each work manager's treatment / tolerance of hogging threads. For instance, an app that maintains sockets versus a webapp.
    Having separate workmanagers allows you to tune the apps individually. If needed.

  • Basics of Thread Pool and MDB

    Hi,
    I am not able to connects the dots between Self-Tuning Thread Pool Threads ,number of MDB's and Open connection to Queue Manager (Listeners).
    Following is my setting
    1 Self-Tuning Thread Pool : Default i.e 5
    2) Initial Beans in Free Pool: 100
    3) Max Beans in Free Pool : 200
    What i see
    Pool Current Count :- 100 (this is as expected)
    On start up of server 105 MDB's are created (No problem with this ) (Have put static variable incrementing in constructor)
    When Messages are sent to MQ (50-100) the "Beans In Use Count" under Monitoring never shows more then 16
    and finally the "open MQ Count" on MQ Explorer is always 16.
    Questions.
    1) What do i need to change, for increasing the count of "Beans In Use Count" and "open MQ Count" on MQ Explorer to be more than 16?
    2)If the Self-Tuning thread pool is 5, how come 16 beans are executed at once? or its just that 16 are picked from pool and only 5 are executed at given time?
    NOTE:- I am using weblogic app server to connecto IBM MQ with JMS Module, so creating the customer WorkManager and attaching it to my listener (MDB) is not supported by weblogic, it says the mdb is not under webloic thread pool so this settingt will have no effect
    -thnaks

    Hi ,
    You have to create a custom work manager and then you have to associate this customer work manager to the dispatch policy of the MDB to increase the threads.
    Following links would surely help you as TomB has explained the same issue very well, do have a look at them.
    Re: WorkManager Max thread constraints not applied to MDB
    Also you can also go through this links which would help you get more information:
    Topic: Configuring Enterprise Beans in WebLogic Search for "To map an EJB to a workmanager"
    http://middlewaremagic.com/weblogic/?p=5665
    Topic: WebLogic WorkManager with EJB3 And MaxThread Constraint
    http://middlewaremagic.com/weblogic/?p=5507
    Regards,
    Ravish Mody
    http://middlewaremagic.com/weblogic/
    Come, Join Us and Experience The Magic…

  • Thread pool tuning on J2ee NW04s...

    Hi,
    I have troubles in defining the number of threads in the pool of the server and dispatcher.
    First of all, I know that there are several thread pools depending on the node you are:
      - Server node: System and Application thread pool.
      - Dispatcher: System thread pool.
    I know that application pool is dedicated to client requests that come in and system is for the services, libs and core of the nodes.
    I'd like to know if I set to a server node a certain number of threads for the System and Application pool, what would be the value of the System thread pool of the dispatcher?
    As almost everything is going through the dispatcher before to reach the server node, what would be the number of System threads of the dispatcher if there are two or three server nodes? Should I multiply the number current threads of the dispatcher by the number of server nodes?
    If there are a lot of P4 calls (each call needs System 2 threads, one for the request and one for the response) should I deduce the number of thread of the dispatcher using this formula if there is one server node in the worse case:
    Dispatcher System Threads = Server App Threads + (Server System Threads x 2)
    Thanks in advance,
    Serge.

    Hi Serge,
    One way that you can size the number of threads is by using the monitoring stats in the VA.  VA -> cluster -> dispatcher/server -> services -> monitoring -> Runtime (tab) -> Monitor Tree (tab)
    then Root -> Kernel
    Check the Application Threads Pool and System Threads Pool. 
    As long as you DON'T experience CPU bottlenecking, then you can increase the number of threads, if:
    1) The ActiveThreadsCount is always at the MaximumThreadPoolSize
    2) You are noticing the WaitingTasksCount is always high.
    If you are experiencing CPU bottlenecking then increasing the number of threads is useless.
    Regards
    Anthony

  • BEA Weblogic AS 9.2 close pool connections - messages BEA-001128 in logs

    Hello,
    We have problems with BEA Weblogic AS 9.2 SP3. From time to time servers stop responding and seem to be stuck.
    In logs there are plenty of messages like this:
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729759> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729759> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729759> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729774> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729774> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729774> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729774> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:49 PM CEST> <Info> <JDBC> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1245157729774> <BEA-001128> <Connection for pool "GoD-PRD" closed.>
    ####<Jun 16, 2009 3:08:50 PM CEST> <Info> <Common> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-4A32C18C5F55D8E4A47E> <> <1245157730603> <BEA-000628> <Created "5" resources for pool "GoD-PRD", out of which "5" are available and "0" are unavailable.>
    ####<Jun 16, 2009 3:12:13 PM CEST> <Info> <Common> <SRV-WLS3> <PRD-J> <[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-637BC18C5F55D8E4A47E> <> <1245157933416> <BEA-000628> <Created "5" resources for pool "GoD-PRD", out of which "5" are available and "0" are unavailable.>
    What exactly this message mean? Does it show some invalid behavior in application or it is just standard notice message?
    What should be behavior of app. server when transaction is timed out on database and closed there? We have timeout for one operation on Oracle set on 10 minutes. Maybe there is a connection between the problems?
    Thanks allot!
    Miroslav

    Test is not enabled. Network should be stable all time - part of one switch.
    Here is a configuration for no-TX source:
    <?xml version='1.0' encoding='UTF-8'?>
    <jdbc-data-source xmlns="http://www.bea.com/ns/weblogic/90" xmlns:sec="http://www.bea.com/ns/weblogic/90/security" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wls="http://www.bea.com/ns/weblogic/90/security/wls" xsi:schemaLocation="http://www.bea.com/ns/weblogic/920 http://www.bea.com/ns/weblogic/920.xsd">
    <name>GoD-PRD</name>
    <jdbc-driver-params>
    <url>jdbc:oracle:thin:@db_server:1521:cml</url>
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
    <properties>
    <property>
    <name>user</name>
    <value>our_user</value>
    </property>
    </properties>
    <password-encrypted>xxxxxx</password-encrypted>
    </jdbc-driver-params>
    <jdbc-connection-pool-params>
    <initial-capacity>1</initial-capacity>
    <max-capacity>100</max-capacity>
    <capacity-increment>5</capacity-increment>
    <test-connections-on-reserve>false</test-connections-on-reserve>
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name>
    </jdbc-connection-pool-params>
    <jdbc-data-source-params>
    <jndi-name>idit.jdbc.IDITDS</jndi-name>
    <global-transactions-protocol>OnePhaseCommit</global-transactions-protocol>
    </jdbc-data-source-params>
    </jdbc-data-source>
    Edited by: myroch on 14.7.2009 4:48

  • How to correctly use a fixed size thread pool?

    I am quite new to using concurrency in Java, so please forgive if this is a trivial question.
    I would like to make use of something like pool=Executors.newFixedThreadPool(n) to automatically use a fixed number of threads to process pieces of work. I understand that I can asynchronously run some Runnable by one of the threads in the threadpool using pool.execute(someRunnable).
    My problem is this: I have some fixed amount of N datastructures myDS (which are not reentrant or sharable) that get initialized at program start and which are needed by the runnables to do the work. So, what I really would like to do is that I not only reuse N threads but also N of these datastructures to do the work.
    So, lets say I want to have 10 threads, then I would want to create 10 myDS objects once and for all. Each time some work comes in, I want that work to get processed by the next free thread, using the next free datastructure. What I was wondering is if there is something in the library that lets me do the resusing of threads AND datastructures as simply as just reusing a pool of threads. Ideally, each thread would get associated with one datastructure somehow.
    Currently I use an approach where I create 10 Runnable worker objects, each with its own copy of myDS. Those worker objects get stored in an ArrayBlockingQueue of size 10. Each time some work comes in, I get the next Runner from the queue, pass it the piece of work and submit it to the thread pool.
    The tricky part is how to get the worker object back into the Queue: currently I essentially do queue.put(this) at the very end of each Runnable's run method but I am not sure if that is safe or how to do it safely.
    What are the standard patterns and library classes to use for solving this problem correctly?

    Thank you for that feedback!
    There is one issue that worries me though and I obviously do not understand it enough: as I said I hand back the Runnable to the blocking queue at the end of the Runnable.run method using queue.put(this). This is done via a static method from the main class that creates the threads and runnable objects in a main method. Originally I tried to make that method for putting back the Runnable objects serialized but that inevitably always led to a deadlock or hang condition: the method for putting back the runnable was never actually run. So I ended up doing this without serializing the put action in any way and so far it seems to work ... but is this safe?
    To reiterate: I have a static class that creates a thread pool object and a ArrayBlockingQueue queue object of runnable objects. In a loop I use queue.take() to get the next free runnable object, and pass this runnable to pool.execute. Inside the runnable, in method run, i use staticclass.putBack(this) which in turn does queue.put(therunnableigot). Can I trust that this queue.put operation, which can happen from several threads at the same time works without problem without serializing it explicitly? And why would making the staticclass.putBack method serialized cause a hang? I also tried to serialize using the queue object itself instead of the static class, by doing serialize(queue) { queue.put(therunnable) } but that also caused a hang. I have to admit that I do not understand at all why that hang occurred and if I need the serialization here or not.

  • JMS Thread Pool Size

    Hi,
              I'm using WLS 6.1. The console has a setting for: JMS Thread Pool Size. I wanted to tune the number of threads used by JMS. I thought JMS asynch consumers would use threads in this pool however that doesn't seem to be the case (they all use the default execute threads and queues). Why is this setting available?
              Note the BEA WebLogic JMS Performance Guide talks about tuning this value from version 6.1 up to 8.1 and states "On the server, incoming JMS related requests execute in the JMS execute queue/thread pool."
              Thanks in advance for any responses,
              Mich

    Disregarding what it is for, in my experience, tuning this setting rarely has much effect. For 6.1, the main thread pool related tunables to look at are the EJB thread pools and EJB max-beans... settings, the "default" thread pool, and the internal thread-pool for stand-alone clients -- all of which are mentioned in the performance guide.

  • Outage with BEA-000627 Reached maximum capacity of pool

    Hi , i am having an outage in the application and at the time of outage i am seeing a lot of <BEA-000627> <Reached maximum capacity of pool > messages.
    Now just before i am getting <BEA-000627> <Reached maximum capacity of pool > i have seen messages of :
    <BEA-001128> <Connection for pool "iemphsysOracleCP" closed.>
    there are 45 such connection closing messages .
    the initial capcity for connection is 25 and maximum capacity is 75 , 50 threads are allowed to run.
    just after closing the connection by weblogic , i see only one message of connection created , but after that i see messages of reached maximum capacity.
    The shirking is allowed and shrinking frequency is 900 secs, now the connections are closed when there are connections existing in the pool above the initial capacity and they are not being used, so weblogic closes the connection, in my problem the connections are closed and just after that one connection created and i see messages of reached maximum capacity , if connections are closed then these should be able to be created again and i shoud not see that maximum capacity message.
    The test connections frequency is also 0 and testconns on reserve set to true , also testconnon release is true.
    Remove infected connections is enabled.
    how it can reach maximum capacity while just before that it is closing the connections ????

    gaurav joshi wrote:
    Hi , i am having an outage in the application and at the time of outage i am seeing a lot of <BEA-000627> <Reached maximum capacity of pool > messages.
    Now just before i am getting <BEA-000627> <Reached maximum capacity of pool > i have seen messages of :
    <BEA-001128> <Connection for pool "iemphsysOracleCP" closed.>
    there are 45 such connection closing messages .
    the initial capcity for connection is 25 and maximum capacity is 75 , 50 threads are allowed to run.
    just after closing the connection by weblogic , i see only one message of connection created , but after that i see messages of reached maximum capacity.
    The shirking is allowed and shrinking frequency is 900 secs, now the connections are closed when there are connections existing in the pool above the initial capacity and they are not being used, so weblogic closes the connection, in my problem the connections are closed and just after that one connection created and i see messages of reached maximum capacity , if connections are closed then these should be able to be created again and i shoud not see that maximum capacity message.
    The test connections frequency is also 0 and testconns on reserve set to true , also testconnon release is true.
    Remove infected connections is enabled.
    how it can reach maximum capacity while just before that it is closing the connections ????Hi. Can you tell me the version of WLS you're using, and also
    show me the whole pool definition? There's no need for shrink.
    I would recommend turning it off. I would set initial capacity
    and max capacity to 50. I would turn off test-conn-on-release.
    That is a waste of performance.
    Joe

  • JRun Thread Pool Issue

    I'm running CF 9.0.1 on Ubuntu on an "Medium" Amazon EC2 instance. CF has been crashing intermittently (several times per day). At such times, running top gets me this (or something similar):
    PID
    USER
    PR
    NI
    VIRT
    RES
    SHR
    S
    %CPU
    %MEM
    TIME+COMMAND                                                                                                   
    15855
    wwwrun
    20
    0
    1762m
    730m
    20m
    S
    99.3
    19.4
    13:22.96 coldfusion9
    So, it's obviously consuming most of the server resources. The following error has been showing up in my cfserver.log in the leadup to each crash:
    java.lang.RuntimeException: Request timed out waiting for an available thread to run. You may want to consider increasing the number of active threads in the thread pool.
    If I run /opt/coldfusion9/bin/coldfusion status, I get:
    Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes
    Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec
    0   0   0   0   -1  -1  150   25    0     0      -1352560      0      0
    In the administrator, under Server Settings > Request Tuning, the setting for Maximum number of simultaneous Template requests is 25. So this makes sense so far. I could just increase the thread pool to cover these sort of load spikes. I could make it 200. (Which I did just now as a test.)
    However, there's also this file /opt/coldfusion9/runtime/servers/coldfusion/SERVER-INF/jrun.xml. And some of the settings in there appear to conflict. For example, it reads:
    <service class="jrunx.scheduler.SchedulerService" name="SchedulerService">
      <attribute name="bindToJNDI">true</attribute>
      <attribute name="activeHandlerThreads">25</attribute>
      <attribute name="maxHandlerThreads">1000</attribute>
      <attribute name="minHandlerThreads">20</attribute>
      <attribute name="threadWaitTimeout">180</attribute>
      <attribute name="timeout">600</attribute>
    </service>
    Which a) has fewer active threads (what does this mean?), and b) has a max threads that exceed the simultaneous request limit set in the admin. So, I'm not sure. Are these independent configs that need to be made to match manually? Or is the jrun.xml file supposed to be written by the CF Admin when changes are made there? Hmm. But maybe this is different because presumably the CF Scheduler should only use a subset of all available threads, right...so we'd always have some threads for real live users. We also have this in there:
    <service class="jrun.servlet.http.WebService" name="WebService">
      <attribute name="port">8500</attribute>
      <attribute name="interface">*</attribute>
      <attribute name="deactivated">true</attribute>
      <attribute name="activeHandlerThreads">200</attribute>
      <attribute name="minHandlerThreads">1</attribute>
      <attribute name="maxHandlerThreads">1000</attribute>
      <attribute name="mapCheck">0</attribute>
      <attribute name="threadWaitTimeout">300</attribute>
      <attribute name="backlog">500</attribute>
      <attribute name="timeout">300</attribute>
    </service>
    This appears to have changed when I changed the CF Admin setting...maybe...but it's the activeHandlerThreads that matches my new maximum simulataneous requests setting...rather than the maxHandlerThreads, which again exceeds it. Finally, we have this:
    <service class="jrun.servlet.jrpp.JRunProxyService" name="ProxyService">
      <attribute name="activeHandlerThreads">200</attribute>
      <attribute name="minHandlerThreads">1</attribute>
      <attribute name="maxHandlerThreads">1000</attribute>
      <attribute name="mapCheck">0</attribute>
      <attribute name="threadWaitTimeout">300</attribute>
      <attribute name="backlog">500</attribute>
      <attribute name="deactivated">false</attribute>
      <attribute name="interface">*</attribute>
      <attribute name="port">51800</attribute>
      <attribute name="timeout">300</attribute>
      <attribute name="cacheRealPath">true</attribute>
    </service>
    So, I'm not certain which (if any) of these I should change and what exactly the relationship is between maximum requests and maximum threads. Also, since several of these list the maxHandlerThreads as 1000, I'm wondering if I should just set the maximum simultaneous requests to 1000. There must be some upper limit that depends on available server resources...but I'm not sure what it is and I don't really want to play around with it since it's a production environment.
    I'm not sure if it pertains to this issue at all, but when I run a ps aux | grep coldfusion I get the following:
    wwwrun   15853  0.0  0.0   8704   760 pts/1
    S
    20:22   0:00 /opt/coldfusion9/runtime/bin/coldfusion9 -jar jrun.jar -autorestart -start coldfusion
    wwwrun   15855  5.4 18.2 1678552 701932 pts/1  
    Sl
    20:22   1:38 /opt/coldfusion9/runtime/bin/coldfusion9 -jar jrun.jar -start coldfusion
    There are always these two and never more than these two processes. So there does not appear to be a one-to-one relationship between processes and threads. I recall from an MX 6.1 install I maintained for many years that additional CF processes were visible in the process list. It seemed to me at the time like I had a process for each thread...so either I was wrong or something is quite different in version 9 since it's reporting 25 running requests and only showing these two processes. If a single process can have multiple threads in the background, then I'm given to wonder why I have two processes instead of one...just curious.
    So, anyway, I've been experimenting while composing this post. As noted above I adjusted the maximum simulataneous requests up to 200. I was hoping this would solve my problem, but CF just crashed again (rather it slogged down and requests started timing out...so effectively "crashed"). This time, top looked similar (still consuming more than 99% of the CPU), but CF status looked different:
    Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes
    Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec
    0   0   0   0   -1  -1  0     150   0     0      0      0      0      0
    Obviously, since I'd increased the maximum simultaneous requests, it was allowing more requests to run simultaneously...but it was still maxing out the server resources.
    Further experiments (after restarting CF) showed me that the server became unusably slogged after about 30-35 "Reqs Run'g", with all additional requests headed for an inevitible timeout:
    Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes
    Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec
    0   0   0   0   -1  -1  0     33    0     0      -492   0      0      0
    So, it's clear that increasing the maximum simultaneous requests has not helped. I guess what it comes down to is this: What is it having such a hard time with? Where are these spikes coming from? Bursts of traffic? On what pages? What requests are running at any given time? I guess I simply need more information to continue troubleshooting. If there are long-running requests, or other issues, I'm not seeing it in the logs (although I do have that option checked in the admin). I need to know which requests exactly are those responsible for these spikes. Any help would be much appreciated. Thanks.
    ~Day

    I really appreciate your help. However, I haven't been able to find the JRun Thread settings you describe above.
    Under Request Tuning, I see:
    Server Settings > Request Tuning
    Request Limits
    Maximum number of simultaneous Template requests
      Restricts the number of simultaneously processed requests. Use this setting to increase overall system performance for heavy load applications. Requests beyond the specified limit are queued. On Standard Edition, you must restart ColdFusion to enable this setting. 
    Maximum number of simultaneous Flash Remoting requests
      The number of Flash Remoting requests that can be processed concurrently.
    Maximum number of simultaneous Web Service requests
      The number of Web Service requests that can be processed concurrently.
    Maximum number of simultaneous CFC function requests
      The number of ColdFusion Component methods that can be processed concurrently via HTTP. This does not affect invocation of CFC methods from within CFML, only methods requested via an HTTP request.
    Tag Limit Settings
    Maximum number of simultaneous Report threads
      The maximum number of ColdFusion reports that can be processed concurrently.
    Maximum number of threads available for CFTHREAD
      The maximum number of threads created by CFTHREAD that will be run concurrently. Threads created by CFTHREAD in excess of this are queued.  On Standard Edition, the maximum limit is 10. 
    And under Java and JVM, I see:
    Server Settings > Java and JVM
        Java and JVM settings control the way ColdFusion starts the Java Virtual Machine when it starts.  You can control settings like what classpaths are used and how memory is allocated as well as add custom command line arguments.  Changing these settings requires restarting ColdFusion.  If you enter an incorrect setting, ColdFusion may not restart properly. 
       Backups of the jvm.config file are created when you hit the submit button. You can use this backup to restore from a critical change. 
       Java Virtual Machine Path
      Specifies the location of the Java Virtual Machine.
       Minimum JVM Heap Size (MB)         Maximum JVM Heap Size  (MB)       
       The Memory Size settings determine the amount of memory that the JVM can use for programs and data. 
       ColdFusion Class Path
      Specifies any additional class paths for the JVM, with multiple directories separated by  commas.
       JVM Arguments
      -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib
      Specifies any specific JVM initialization options, separated by spaces.
    I did go take a look at FusionReactor and found it's not free (which would be fine, of course, if it would actually help). It looks like there's a fully functional demo, which is cool...but I've haven't been able to get it to install yet, so we'll see.
    Thanks again!
    ~Day
    (By the way, I've cross-posted this inquiry on StackOverflow. So if you're able to help me arrive at a solution you might want to answer there as well.)

  • Using AsyncEventHandlers as a RTJ buildin thread pool

    Hi David
    1. I would like to use AsyncEventHandler mechanism as a thread pool. Would there be a way in future version to access the AsyncEventHandler's RT Thread pool API ?
    2. I am using the BoundAsyncEventHandler for long duration task (since the BoundAsyncEventHandler is bound to a specific Thread) - Would there be a way in future version to bound a specific RT Thread to this handler ? - for example a RT Thread that was created by my own Thread pool ?
    Thanks

    Gabi wrote:
    1.We need our code to access the AEH's "thread pool" and check the pool's availability; increase/decrease the pool size dynamically; check pool size for enough free threads to run any given number of tasks and so on...There are no API's for this. How a VM supports AEH execution is an implementation detail, and while a "pool" is the logical form of implementation the details can vary significantly so there's no real way to define an API to control it without defining what form it must take. For example in Java RTS the pool threads are initially at the highest system priority and then drop down to the priority of the AEH they execute. In a different design you might have a pool of threads per priority level (or set of priority levels) so even the notion of increasing/decreasing the pool size is not necessarily straight-forward. In Java RTS the pool will grow as needed to service outstanding AEH activations. You can control the initial number of threads - and there are two pools: one of heap-using and one for non-heap AEH.
    2. I was thinking about adding the BAEH class with the next method:
    void setBoundThread(RTThread t) throws IllegalArgumentExceptionand also in the Ctor add the RTThread as a parameter.
    Inside this method, the thread should be check if started and is so should throw an IllegalArgumentException.Sure, but why do you need to do this? What is different about the RTT that you pass in? The thread executing an AEH acquires the characteristics of the AEH.
    An API like this raises some questions such as: when would the thread be started? It also has a problem in that the run() method of the thread has to know how to execute an AEH, which means we'd (the RTSJ expert group) probably have to define a class BoundAEHThread that has a final run() method and you'd then subclass it.
    3. Additional question, what do you think about using lock (CountdownLatch) during the AEH's executions ?
    I've seen somewhere that AEH should only be used for short live tasks - is it true ? (in the JavaDoc it says there is no problem doing so but I want to be sure)If an AEH blocks for any length of time and additional AEH are activated then the pool management overhead may increase (unless you already sized it appropriately). But otherwise there's no restriction on what the code in an AEH can do. But note that none of the java.util.concurrent synchronization tools support priority-inversion avoidance, so you need to be careful if using them (ie only use them across threads of the same priority and don't expect FIFO access).
    David Holmes

  • Log of execute thread pool status

    hi guys
    We are performing some performance analysis on a application and would
    like to log the status of the execute thread pool periodically to a
    log file.
    The performance tests run for a extended period of time and hence
    console is not proving to be effective in monitoring the behaviour
    over a long period of time.
    What other alternatives exist if I want to do this.
    Thanks for your time
    anand

    thanks guys,
    I found an answer in one of rob wollens old post.
    http://dev2dev.bea.com/resourcelibrary/utilitiestools/index.jsp
    Sorry for posting without doing a little research
    ~anand

  • Thread Pool's decreasing application performance

    Hi,
    In my application, I need to make 10,000 threads for network calls at one instant and release the threads after downloading content.
    Content downloading of individual thread consumes less than 1min.
    When I try this using Timer Task and normal Threads, it works perfectly.
    But after introducing ThreadPool (initial :20,000 ; maximum: 50,000) , the response has degraded.
    Is there any limitations or known issues similar to this for thread pool's.
    Thanks!

    rock_win wrote:
    In my application, I need to make 10,000 threads for network calls at one instant and release the threads after downloading content.10000 threads all doing network connects at the same time? You better contact 10000 distinct servers and have tons of network bandwidth.
    You'll hit problems at that level long before the number of threads becomes a problem.
    Content downloading of individual thread consumes less than 1min.
    When I try this using Timer Task and normal Threads, it works perfectly.What are "normal Threads"? How many operations are you running in parallel here?
    But after introducing ThreadPool (initial :20,000 ; maximum: 50,000) , the response has degraded.You want to have 10000 Threads and declare a pool with 20000 initial threads? Why?

  • Combining the fixed and cached thread pools

    Is there a way to 'combine' the behavior of cached and fixed thread pools ? I have a requirement where
    - at startup, I need to execute a fixed number of short-lived tasks at the background
    - after startup, on demand, I need to run one short-lived task at a time
    If I use a fixed thread pool for my startup processing, it creates the fixed number of threads to process the tasks. But subsequently, those many threads are not required since my task submission is going to be one at a time. The remaining (n-1) threads therefore are really sitting idle & useless.
    If I use a cached thread pool, then I cannot constrain the number of threads to run at startup (since it creates one for each task). Though it ends up taking those threads out after they are idle for a fixed period. But I'm worried that it might create many threads and possibly slowing down the startup ?
    Is there a way to create a pool with a fixed number of threads but 'remove' a set of threads when they are idle ?
    TIA

    v_bala wrote:
    Thanks. The SynchronousQueue worked as expected. But I tried a LinkedBlockingQueue with size 1 hoping that would cause the second task submission to cause a new thread creation. It didn't. Instead it queues up the request (maybe the doc in ThreadPoolExecutor says that when it mentions "If corePoolSize or more threads are running, the Executor always prefers queuing a request rather than adding a new thread"). I suppose if I were to try another it would create another thread ? (I was testing with only two tasks - the first one ended up creating a thread for the task and the next got queued since the queue capacity is 1)yes, the TPE will not start adding threads until the queue starts rejecting them. kind of odd in my opinion, but that's how it works.
    Btw, that was an interesting idea to add a task that scale the core pool number down ! Currently it comes back down to just the one thread (my core pool size is one) after the idle timeout but your idea may give me a slightly better response since it will scale down the core pool size quicker....I suppose if there was a task submission before that idle time there maybe a performance hit (but I don't anticipate that in my case )?actually, that won't scale things down any faster. changing the core pool size will not ditch the other threads immediately, they will still stick around until they idle timeout. changing the core pool size allows you to not worry about the queue implementation (the first solution). you can set the initial core pool size to your "max" size on startup, then drop it down for the normal processing, all the while using a linkedblockingqueue of unlimited size.

Maybe you are looking for

  • SRM MDM Installation Problem (.a2a file Unarchive issue)

    Hi All, I am totally new for SRM MDM, and need to install SRM MDM and configure the system. Below are the steps what i have done till up to Already SRM 5.0 Installed Installed Oracle 10g ( with diffrent Oracle SID as I have already one Oracle install

  • To display long header text than 40 characters in ALV_GRID_DISPLAY method??

    Hi all, I have a requirement to display a very large text(more than 40 characters) in header column of ALV grid. Is there any way out. Regards, Anurodh

  • Aspect Ratio on Export

    I'm working on a 720x576 Anamorphic timeline And am exporting my final as a ProRes422 file. When I try opening the exported file in Quicktime, it shows it as 4x3 rather than 16x9 Have I missed something in the settings?

  • Accidentally set up two users on mac pro- i want to delete one user

    When I set up my mac pro I started setting up a user account.  Then I used merge assistant and it created a second account.  I want to merge the two accounts or delete one of them

  • How to print a document on a smaller size ???

    When my document is A4 and I want to print it on a smaller format I put in paper format (shift / apple / P) which is supposed to have eg 20x25 cm. But now I see only a cutout of my document. How can I achieve this, the whole document on the smaller s