CF Performance / Timeout Problems

Question

Hi,
we are developing and hosting a site for a client with around ~3 million pages and around 600k visitors per month.  We recently switched to a new server and in the process upgraded to CF10 from CF8. Many things like images, downloads and video streaming is done via external CDNs. The problem we are having right now is that the server/site produces timeouts and gets from time to time slower to a point where it's basically just dead. After a reboot of the virtual machine everything is snappy and great again.  Now this may take a day or a week - there is no logic to it. Also while the site is perfectly snappy cfstat also shows the occasional timeout which we can't explain. The funny thing is that the server itself isn't the problem - on the hardware site it only uses ~10% cpu power and yet coldfusion runs slow.
Just some quick facts:
2 x Intel® Xeon® X5650 Hexa Core (6 x 2,66 GHz, 12 MB L3 Cache)
RAM: 16 GB
4 x 300 GB SAS 15krpm 3,5
CF is running in a virtual machine virtualized via VMware ESXi. It also runs mysql on the same virtual machine.
But now to get to a more specific question. We created a new virtual machine and are running some load tests on it to tweak it and find the problem. It seems that after the update 5 it got a bit better. But still we get unusual timeouts / slow downs every ~2 minutes. So we dumbed down our load tests to one single cfm file that has one query in it and a standard / empty application.cfc.  We also duplicated the code to php and are executing the exact same query from php. The php load tests shows no timeouts / errors whatsoever. The coldfusion loadtest shows timeouts every ~2 minutes or so (not exactly, but damn close to 2 minutes). We have no idea what's causing this. The server is pretty much a vanilla installation (we tried centos and ubuntu). Also there are no memory problems we can see in the cf server monitor. The 2 minute errors are not caused by the garbage collection. I attached two pictures - first the coldfusion loadtest and second the php loadtest.
Does anybody have any ideas what might cause this? We are really running out of ideas here.

Answer 1

Could be Garbage Collection in you jvm?  Also make sure you have the latest updaters for CF10.
I'd suggest downloading the 10 day free trial of http://www.fusion-reactor.com/fr/ and see what it says. It will show memory issues, database issues, and slow pages. I use it everyday to monitor our sites.
p.s. I do not work for them or benefit from the sale of that product...
Cheers!
-Dave