How to determine when FMS is starting to get overloaded?

Question

We have a set of FMS's deployed on Amazon's EC2.  One of the things we want to be able to do is automatically detect when we should start up another FMS instance.  To do that, I've been looking for metrics I could measure on the local FMS box to help me identify "transition" points, e.g., when we should add capacity or remove excess capacity.
I ran some load testing to find out where the capacity limits of a particular box, but ran into a couple of problems:
   * Traditional system metrics (cpu/memory/run queue length) did not do a great job of predicting when we'd hit a wall.  Load was really the only thing that seemed to climb much and it was only at about 4 (on a 4-core box) when things went south.
   * When we *did* hit a wall, it was a pretty sharp cliff.  We seemed to be doing fine at 70+70 streams (~300kbps streams in reflected out) and at 75+75 streams, but when I went to 80+80 streams, BAM!  Things just started unravelling.  With very little in the way of error logs to indicate what might be happening.  But all of the sudden, my counters for simultaneous streams/etc dropped down from 80ish to 20ish (I was still publishing 80 to the server).
I tried bumping up the EC2 instance size (under the theory that we were being bandwidth-capped or stream-capped), but didn't really see much difference.
I see two possibilities:
  * We actually are being bandwidth- or stream- capped and going up to a bigger box didn't help
  * There are a number of other metrics on the server I could look at that would have shown a gradual degradation.
Assuming the latter, does anyone have any suggestions for what metrics I might measure on the FMS to decide if we were starting to get loaded?  For example, I've thought about comparing Stream.time to NetStream.time for streams I'm reflecting out of the server.

Answer 1

Thanks everyone for bringing this up and proceeding with some discussions. I would put up my quick thoughts on this one.
Any load testing , as mentioned would start with CPU and Memory metrics. So is for FMS.
For a live case, CPU usage, for default FMS configuration would be little high. This is because of the aggregate messages and other queues that are maintained. One can disable these (application.xml) to considerably reduce the CPU usage.
Memory starts increasing as more streams are being served, but it will get stabilized, in my experience, for 1200 connections, all playing a 500 kbps stream, i would expect a memory usage of somewhere around 2-3 GB. (i would confirm the numbers if needed for accuracy, later).
One other good thing to look for is Buffer Length on the subscribers. An abnormal increase in its value shows the server is unable to fill the buffer of the client well in time.
Another related option is to look for frequent NetStream.Buffer.Empty and Netstream.Buffer.Full codes, if they are coming up too fast, it means the buffer on the client side is emptied faster than what we want.
Latency is by far, the best identifier. Mark the deviation of the subscribers from the 'actual' live, queues and aggregation of messages will play a part here again.
There are core logs enabled for any system over load (more than 90%) of FMS CPU. Watch out for these logs. Till the point one wont find them, i am sure the FMS is doing good.
Another option to take a look at is fibers. You can either enable and disable them for perf differences.
In the end, there must be some benchmarking each one of us should do, in order to find the just_before_fail_state. We keep doing that internally, with lots of load and expecting it to crash