Is this possible to access arbitrary frame and to modify the mask?..

Question

Sorry for a bit messy discussion title. Please, let me explain.
I have some experience in implementing video processing algorithms, but as our own software and never under AE. Now there is an idea to implement something for AE and first of all, I'm curious, if this is possible at all.
1) When processing the n-th frame, I need the full access to the data of (n-k)-th and (n+k)-th. Is this possible to do in general case? Or the only variant is to implement frames buffering during the first pass?
2) Our algorithm uses the mask as one of the inputs. But in contrary to the usual case, mask does not define an area of interest. In any case the full frame is processed, but the processing differs for different regions, depending on mask. In our software, it works like this: user defines a mask (and some parameters). Then effect is slightly modifying (let say – refining) the mask during the render and renders final images. After this user can modify the mask somehow again if he/she is not satisfied with what algorithm has done. Can I produce the same workflow in AE? The only way is using a drawbot (this means CS 5.5 and higher)? And what if I want masks to be raster images?
The second question in the framework of this one is how to do two video inputs for an effect (first – the video to be processed and second – the mask (grayscale sequence in general case))?
The possible workflow is that user provides trivial mask – lets say, the black solid. But even during the first pass effect should produce the mask, according to its sight to the video – so, it should somehow substitute the black solid with the sequence on black'n'white images. Is this all possible in AE?
3) Is this possible (should be, I guess) to implement multipass algorithms? How the preview works then, if I need, for example, to pass all the video length 3-4 times, to produce the final render?
I can expect that all this is solvable in AE, but may be experienced users could be so kind to give me some basic recommendation on how to do this. Looking through the PDF, which accompanies the SDK has not allowed to come to a distinct answer for the question: is this possible or not. So, I rely on your consultation.

Answer 1

hi Theofil Mulony, welcome to the forum!
in short, yes, what you want to do is possible, but some parts may be tricky.
and now the long version:
1) When processing the n-th frame, I need the full access to the data of (n-k)-th and (n+k)-th. Is this possible to do in general case? Or the only variant is to implement frames buffering during the first pass?
you can access the pixel data of any layer at times other than the currently processed time. this concept of getting that data is referred to as "layer checkout" in the AE SDK, and "wide time" is the name of getting pixel data at times other than the current.
the following thread discusses the difference between two methods of getting such data:
http://forums.adobe.com/message/4033930#4033930
you can also look at the "checkout" sample project to see how both concepts are implemented.
2) Our algorithm uses the mask as one of the inputs. But in contrary to the usual case, mask does not define an area of interest. In any case the full frame is processed, but the processing differs for different regions, depending on mask. In our software, it works like this: user defines a mask (and some parameters). Then effect is slightly modifying (let say – refining) the mask during the render and renders final images. After this user can modify the mask somehow again if he/she is not satisfied with what algorithm has done. Can I produce the same workflow in AE?
masks in AE generate a matte by default. the user can turn that mask off, so that the shape remains, but the mask does not... mask. (you can also set the mask mode to "none" via API commands).
you can then read the "off" mask attributes and either render using an API command (see "pathmaster" sample to see how), or write your own algorithm if the API constraints on that operation are a problem for you. (and there are a few constraints on rendering masks in the API)
now. when you say "modifying the mask" do you mean you modify it's generated matte, or do you mean change it's defining vector?
both are possible, but completely different.
if you turned the mask off, and rendered it's alpha yourself, then you can modify that alpha as you wish.
if you want to use AE's generated matte of the mask and change the mask's outline shape, then you need to look into "streams" and the "mask outline suite".
the user can modify a mask's shape as he/she pleases, unless you lock that mask or otherwise sabotage the user's operation.
The only way is using a drawbot (this means CS 5.5 and higher)?
drawbot is the API for drawing over AE's interface in CS5 and up.
it's only a drawing tool. previous versions used OS drawing tools. that's all the difference.
what i think you're asking is weather you need to create the mask drawing/handling capabilities or not.
the answer is: you can, but you don't have to.
unless you need control over the mask drawing that AE's existing masks don't have, i would have used AE's masks and not create a brand new mechanism (that the user need to learn as well).
if you insist at creating a custom UI for drawing masks, the "CCU" sample project would be a good place to start the research.
(just don't come crying to me when you get a splitting head ache)
And what if I want masks to be raster images?
The second question in the framework of this one is how to do two video inputs for an effect (first – the video to be processed and second – the mask (grayscale sequence in general case))?
The possible workflow is that user provides trivial mask – lets say, the black solid. But even during the first pass effect should produce the mask, according to its sight to the video – so, it should somehow substitute the black solid with the sequence on black'n'white images. Is this all possible in AE?
by default, "effect" plugins (as opposed to AEGPs) get only one input - the image of the layer, post masks and previous effects (but before transformations).
you can put as many "layer param"s in your effect and let the user select any number of other layers, which you can the "checkout" and use as second inputs.
you can also checkout other layers without the user selecting them manually.
however, checked out layer results are pre-masks and effects. so if the user draws a mask over a black solid, you will not be able to read the "post mask" image unless the user pre-composes the layer with the mask inside the pre-comp. it's a common practice for users to do so, as many effects rely on a second input, and that's the only way to make the input pos-masks and effects. (so it's no biggie from the user's point of view)
in the same manner, you can read any layer's image and use it as a mask.
you get the image buffer, so it's up to you to decide whether to use the alpha channel of that image or the rgb (or any other method you see fit).
3) Is this possible (should be, I guess) to implement multipass algorithms? How the preview works then, if I need, for example, to pass all the video length 3-4 times, to produce the final render?
AE doesn't do multi pass rendering.
it actually can render arbitrary frames, in no particular order.
but as you already saw, you can access the image data from any time - of any time, so you can emulate passes.
if you need to fly through the whole sequence for every frame, that would of course be very time consuming and wasteful.
in such a case, you can try to do these passes only once, when effect is implemented, and cache the results.
look up "sequence data", to see how to manage memory specific to a single instance fo you effect.
but i must say that if you have to do full length scans of the footage, a couple of times over, and cache EVERYTHING, then you're in a bit of a problem in AE.
it's just not set up for such rendering logic.
not impossible, but highly inefficient in the AE rendering pipeline.
there is a lot more to al of the above, but i hope this gave you a clearer picture of how things are in AE.