Design for workflow engine...

Question

I'm starting to (re)design a workflow engine and was wondering if there were any design resources/examples/ideas that people might know of that would help.
Essentially the workflow is a simple state machine with user-invoked transitions.
However, it also possesses (multiple) substates and automatic transitions (when certain criteria are met).
The presence of automatic transitions complicates the design slightly but it's still very manageable. The main problem is how to handle the multiple substates of each item without necessarily describing an API for the workflow engine that is bound to the structure of these items (which are basically trees with different node types at each level).
Many thanks in advance!

Answer 1

I use a message passing approach with all troublesome
you mentioned. Hopefully delays are not so significant
but sometimes annoying. Could you elaborate more about
MOAT approach, what is complicity?Let's assume that you're going to maintain all information in a database, with tables holding jobs, tasks, and relationships between tasks. Going to the example of a task completion, you might have to do the following: (1) determine whether dependent tasks can become ready, (2) modify parent status if this is the last child, (3) modify job status if this task is the last in the job, (4) notify external listeners that the task is done.
In the MOAT approach, this all happens within one transaction: you update the task's row, find the rows of all dependents, potentially update them, find the parent, potentially update it, find the job, potentially update it, and then notify the listeners or write some transaction log that will later be read by them. For a typical workflow, you'll probably read a dozen rows from all 3 tables (jobs, tasks, relationships), and update a half dozen. But it's a recursive operation, so you might read a hundred rows or more, and update fifty of them.
The benefit of this approach is that state remains consistent to the user. After a client completes a task, it can immediately work on dependent tasks. The downside is that completing one task takes a non-deterministic and potentially long time.
In the message-passing approach, the completing task simply publishes an event indicating its completion. Execution time from the client perspective is relatively constant.
The central event loop then passes this event onto handlers that perform the related actions (updating dependents &c). As long as you control simultaneous updates to the job (from multiple threads), the server-side code is pretty straightforward.
It's on the client side that things get interesting. Let's say that the client tries to anticipate actions: it knows that when it completes one task, the dependent should be worked on, so it tries to explicitly grab the dependent (rather than use a generic operation like getNextTask()). In this situation, the server may not be done updating the dependent tasks, so they're not ready when the client tries to grab them.