Implementing FSM's with Threads, soliciting design opinions

Question

I'm currently designing a system that has some unusual requirements and was hoping to solicit the esteemed opinions available at this forum regarding a potential design that may be interpreted as an abuse of Threads.
This system consists of a large collection of Finite State Machines that transition from state to state based on events originating from both within and without the system.
Each event coming into the system is targeted to a particular FSM. It is critical to the system that the events be serviced by the FSM instantaneously. The system should consume and process events as fast as they come in and not get backlogged.
I would like to optimize the number of FSM's that can be hosted by a single instance of the system before backlogging occurs. Ideally the system should be adaptive and detect when it is not servicing events as quickly as possible and begin to shutdown FSM's until it detects that it is operating at peak efficiency.
The design I'm considering entails using an instance of a Thread and a Queue for each FSM instance in the system. There may be as many as 3000 or so FSM's altogether. Additionally I would have a Thread servicing each event source, of which there are only a handful, and it would be this Thread's job to receive and dispatch an event to the appropriate Queue for the particular FSM for which the event is targeted.
I envision the FSM's Thread wait()ing on the monitor on the Queue and having the event dispatcher Thread performing the notify() after enqueing the event. Using this design allows me to have yet another Thread that will periodically peek at the size of the Queues to ensure that they are consistently near empty. If the Queue's begin to get backlogged, then the Monitoring Thread can begin shutting down FSM Threads until the optimum number of active FSM's Threads is achieved.
What I don't like about this design is the potentially large number of Threads being instantiated. I would appreciate and welcome suggestions for a better way of doing this.
Thanks,
-S

Answer 1

Bob, Matfud,
First I'd like to thank both of you for taking the time to reply. I must admit that I originally wanted to approach the problem in a manner similar to your respective suggestions and indeed solicited opinions here because I also was uncomfortable with the waste in having so many threads. However there are some issues which still steer me in that direction.
If I were to use a single Queue up front as suggested then the disparate event sources would be forced to synchronize on every event thus limiting to some extent concurrency. By having separate Queues, the only time synchronization comes into play is when two or more event sources have events targeted to the same FSM at the same time. On a multiprocessor box this allows me to have disparate events dispatched simultaneously rather than sequentially in most cases. I feel that this would add a considerable degree of liveness and concurrency to the system.
With regard to employing a Thread pool, I like the idea as it would give me better control over the number of simultaneously executing Threads as is indeed the case with Session beans. However, I'm not sure that it provides optimal utilization of resources, but rather consistent utilization of resources. Finding the Thread pool dry may not indicate that I'm already using maximum resources but may instead just be a sign that I did not allocate a sufficient number of Threads to the pool.
Nevertheless, my main reason for considering independent Threads per Finite State Machine is also one of concurrency. Consider the following scenario: Two event sources receive two separate events, both targeted for the same FSM. If the event dispatcher were to pluck two Threads from the Thread pool and allow each to attempt to inflict their respective events on the FSM, then I'll need to provide a good deal of synchronization within the FSM as a single FSM cannot service multiple events concurrently, but by definition must service them in sequence thus triggering state transitions. On the other hand, if each FSM is a Thread of its own, then it will wait() until notified that there is an event on the Queue and will pull and process one event at a time until the Queue is drained upon which time it will return to its idle state and wait() again until notified. In this scenario the FSM's processing need not be synchronized since it is pulling the event, and only the momentary access of its Queue requires synchronization.
Matfud, to your point the amount of work per transaction will involve a number of calculations and decision making. Overall considerably more processor effort will reside in the FSM logic as opposed to the event queueing and dispatching mechanism. With regard to interaction between FSM's, there is none.
Again I want to thank you both for your suggestions. I look forward to your responses as I eagerly want someone to convince me to abandon the design I have so far in favor of something more elegant and performant.
bob_boothby, matfud, mattbunch, dubwai, YATArchivist, DrClap, warnerja, sylviae, jschell, kdgregory, somebody, anybody, please talk me out of doing this.
-S