Help: CF10 (all hotfixes) on Win2K3 starts to consume up to 100% cpu (coldfusion.exe)

Question

hi
We have a CF9 latest updates running on a Win2K3 server  under Apache 2.2.22 and JRun4 for years. Now, I wanted to upgrade to CF10.
I first updated to Apache 2.2.23 with ssl 1.0.1c
For reassurance, I just disabled all JRUN CF9 services as seen in the Task Manager.
Then I installed CF10. The installer mentioend different  NET Ports. I accepted them.
Install continued without a problem, CF10 was setup and I configured it. I then ran our main ap which also worked fine.
Then, after some few hours while I was checking the health of it, I figured that  COLDFUSION.EXE takes up all CPU time.
Server response  is still there but sluggish.I have not seen any reason why the cpu usage increases.
I have to kill coldfusion.exe to cut cpu load. however, a few minutes after that (about 10-20 minutes) coldfusion.exe again begins to climb up. it takes about 5-10 minutes to hit 100% ...
I have no idea how to deal with that (except going back to CF9)
On my developer CF10 (same config) I have not seen that behavior. Howewer, this server is not exposed to the net, the productive one is ... there might be some deep links pointing somewhere ... however, our server is not a busy server, so it should not face such a cpu load .. as said it ran under CF9 for years without such problems.
any hints how to detect the cause of this? Can I use any CF10 tools to figure out this problem?
any other clues to that?
it's urging ... as usual ...
Thank
Martin

Answer 1

hi carl
thanks for the hints ...
I checked the mod_jk.log and found such lines
[Thu Jan 17 15:02:10 2013] [4908:1128] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1027): Failed opening socket to (127.0.0.1:8012) (errno=61)
[Thu Jan 17 15:02:10 2013] [4908:1128] [error] ajp_send_request::jk_ajp_common.c (1649): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Thu Jan 17 15:02:10 2013] [4908:1128] [info] ajp_service::jk_ajp_common.c (2629): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Thu Jan 17 15:02:11 2013] [4908:1128] [info] jk_open_socket::jk_connect.c (626): connect to 127.0.0.1:8012 failed (errno=61)
[Thu Jan 17 15:02:11 2013] [4908:1128] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1027): Failed opening socket to (127.0.0.1:8012) (errno=61)
[Thu Jan 17 15:02:11 2013] [4908:1128] [error] ajp_send_request::jk_ajp_common.c (1649): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Thu Jan 17 15:02:11 2013] [4908:1128] [info] ajp_service::jk_ajp_common.c (2629): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Thu Jan 17 15:02:11 2013] [4908:1128] [error] ajp_service::jk_ajp_common.c (2648): (cfusion) connecting to tomcat failed.
[Thu Jan 17 15:02:11 2013] [4908:1128] [info] jk_handler::mod_jk.c (2725): Service error=-3 for worker=cfusion
and this is my server.xml (all comments removed):
<?xml version='1.0' encoding='utf-8'?>
<Server port="8007" shutdown="SHUTDOWN">
   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
   <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" />
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="cfusion">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="false">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/>
      </Host>
    </Engine>
  </Service>
</Server>
What drives me crazy is that i never seem to catch the start of process when the CPU begins to climb. I can stare hours with the remote desktop to see the tast manager almost idling (as it should), then I go away and close teh rd session ... later, for curiosity, i get back and see the CPU agin on the ceiling ...
if Windows Perfmon would allow me to add the cf10 counters, i hope i could corelate the time of the cpu öoad climb with a request to my vhosts ... i NEED to find that call that might start all this after a cf10 restart ...
Martin
BTW & FWIW:
I have CF9 Enterprise (JRUN stuff) on the same server, of course all services disabled. by this i mean i did not uninstall it before installing cf10 the first time.  could this be a problem?
As i wrote that, the cpu again begins to climb. I then did a cfstat -s because PerfMon does not work. cfstat says this
Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes
Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec
0   0   0   0   -1  -1  0     0     0     0      0      2889   0      0
and while i atch the cpu climb, i use cfstat over and over and see the AvgDB count increasing constantly ...
Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes
Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec
0   0   0   0   -1  -1  0     0     0     0      0      2889   0      0
0   0   0   0   -1  -1  0     0     0     0      0      2897   0      0
0   0   4   4   -1  -1  0     0     0     0      0      3731   0      0
0   0   0   0   -1  -1  0     0     0     0      0      4255   0      0
0   0   0   0   -1  -1  0     0     0     0      0      4286   0      0
0   0   0   0   -1  -1  0     0     0     0      0      4286   0      0