Architecture advice for Effect and AEGP communication

Question

Happy monday!
In my current plugin project I need to use an "helper" AEGP for tasks my main effect plugin cannot accomplish (delayed tasks via AEGP idle hook, project modifications that will crash if performed inside the Effect logic, etc.)
I'm trying to figure out an elegant and robust architecture design for this cross-communication. 
Heres the way I see it:
Effect passes info along to AEGP via custom suite data.
AEGP reads the data at idle, and performs some required task.
The success/fail of this work and the resulting products are communicated back to the Effect via AEGP_EffectCallGeneric.
My question is basically surrounding the data/info handoff.  What kind of objects should I be passing back and forth?  Is it considered poor design to simply pass along a pointer to the sequence or custom arb data to the AEGP via the custom suite?  And vice versa when the GP sends EffectCallGeneric to the effect and passes a pointer, it could just be the same pointer that the effect originally sent the AEGP using the custom suite. 
Aside from undo/redo, is there an advantage to using arb parameter data over sequence data for this task?
It seems to me that AEGP plugins are loaded once and only once in the session, so if multiple instances of my effect plugin are used in the project, and they are all calling the AEGP there is the potential for a race condition if the custom suite data is used to pass along information.  Is this an accurate assessment of AEGPs? A simple example of this would be that the custom data suite has a variable that stores which effect plugin is calling the AEGP.  If another instance of the effect starts talking to the AEGP, it would overwrite the suite var telling the AEGP which effect instance is calling it, thereby making the AEGP perform the work of one effect instance but sending the results to different instance.
Even if the AEGP is extremely simple and just performs small atomic operations each time the custom suite data tells it to, how do I prevent n+1 Effect instances from colliding when using the AEGP worker?  Am I missing some key part of the Effect->AEGP communication that prevents this race condition situation?
This is the first time I've tried my hand at writing inter-process communication where I have the option to exchange actual in-memory objects. I'm hoping someone with more experience with this sort of problem can give some pointers(hah) or at least a few cautionary words about designs to avoid.
Thanks, sorry for such an open ended question, but this is the place to talk to the pros
-Andy

Answer 1

yo andy! what's up?
i see you were up to no good... naughty.
you are correct in your assumptions.
AEGPs are indeed loaded only once per session, but if you're rendering with
the "multiple frame simultaneously" option, then the AEGP is loaded
separately for each AE instance.
(so would your plug-in. each AE instance calls it's own global setup)
now, plug-ins are called on the main thread. which should mean that only
one plug-in is is called at a time, which should mean that your AEGP may
only get one call at a time. i'm not absolutely sure that the base
assumption is correct here. it's possible that multiple effects are called
at the same time, so re-entrancy of your AEGP suite is possible.
how do we know? we can either test, or ask zac lam.
re-entrancy is a non issue when reading data (all can read at once. it
doesn't matter). but when writing data... that's a problem.
to prevent such issues you need to implement some mutex (mutually
exclusive) mechanism.
it only allows one caller access at a time, while the other caller stall
until their turn.
boost library offers such tools.
as for what data you should transfer.
you can either pass data by value, or by reference.
when a function gets it's data like so:
function(int val);
it get's a local copy of the data. no worries about scope here.
if it gets the data like so:
function(int *val);
or function(int &val);
then we're talking reference. in this case, the val variable is only valid
while the object to which it refers is valid.
that depends on stability of that memory chunk. what makes a chunk stable?
1. if the data is created as a local scope variable it will invalidate when
the code block finishes. make sure don't access that data at other times.
2. AE handles of a plug-in instance are only valid during calls to that
plug-in.
let's look at the two directions of communications you have.
1. effect calls AEGP.
in this case, the call order is as follows:
AE calls effect.
effect is now executing. at this point, sequence/global data are valid, and
anything locally allocated in the call handling function.
effect calls AEGP.
AEGP is now executing.
you can now pass anything you like from the effect to the AEGP.
if the AEGP returns data, it should be data that remains valid after the
AEGP finishes executing.
AEGP finishes it's thing and exits.
effect is now executing.
effect finishes the call, exits, and AE is executing.
at this point, seq/global data handles are no longer valid. (at least you
can't rely on that)
also, anything allocated in the effect's calling function as a local
variable, is no longer valid.
so now your AEGP gets called on idle time.
it should only rely on memory that has kept since the effect was executing.
what memory is kept?
anything in global scope of the AEGP, or pointers to memory that was not
deallocated or moved.
seq data handle may not have been deallocated, but you can't tell if it
moved or not. so you can't rely on it.
if the seq data has another pointer in it, which point to a non movable
piece of data, then you can rely on that pointer.
seq/global handles are locked an unlocked by AE. memory handles are managed
by the plug-in (so you know if they can be relied on or not).
2. AEGP calls effect.
AE is calling the AEGP on idle time.
AEGP is executing.
AEGP calls effect with data.
effect is now executing.
effect uses data gotten from AEGP.
effect optionally returns data to AEGP.
effect exits.
AEGP is now back executing.
AEGP uses data returned from effect. that data can't be references to local
variables in the effect as they are no longer valid.
AEGP exits, AE is now executing.
so in both directions you need to see that at the time the data is used,
that is is valid.
as for sequence data vs arb param.
yes, there is a difference besides undo/redo.
AE creates (multiple) copies of the data stored in arb params before each
call to the plug-in and also after.
if you're storing very large amounts of data in the arb param, then the
copy operations will start dragging your performance down.
it's usually a non-issue, but... it's a point to think about.