Continuous rasterization with contained comps, 3D layers and track mattes

Question

I have an project comprised only of AI objects contained in comps, used with 3D and I have a problem similar to many I have seen on this forum, though I have tried every solution supplied to no avail.
At the lowest level comp are some AI layers that are animated in the comp. Call it comp A.
Two copies of comp A are in comp B, each track matted by another AI layer.
Five copies of comp B are in comp C.
Comp C is placed several times into 3D space in the main comp where a camera moves past.
All the vector layers are 3D and set to CR
All the comp layers are 3D and NOT set to CT
Without the proper settings for CR/CT, the track mattes slip all over the place (but things rasterize well).
WITH the above settings, nothing rasterizes properly (but the mattes work). The comps appear rendered at the original size of comp A, which is heavily pixelated (or blurred with quality turned up) as the camera moves past.
I have tried precomposing the track-mattes and their objects contained in comp B into their own comp (also following above settings) and got no change.
Of course, in retrospect, I could have scaled up the objects in comps A, B and C but at this point the project is in final stages and I'm wary of all the adjustments it would take to go this route (it also seems like a very wrong way to go about working with vector graphics).
Can anyone help, or is this just the way AE is? (I am fairly experienced, but < 1 year.)
Thank you for this forum.

Answer 1

Here's what you are probably missing. Let's set up something very simple. You have a Main Comp and a PreComp. The Main comp contains the PreComp and a Camera. The PreComp only contains single layer - TestShape that is a 20 X 20 pixel red solid and it has a circular mask applied so it looks like a dot. When you add a mask to a solid you have essentially turned that layer into a vector layer.
Let's go through a couple of things in the PreComp. Fist, scale TestShape 600% and look at the edges. They are a mess. Turn on Continuously Rasterize and they are perfect.
Look at the Main comp and scale up the PreComp layer 600% and the edges look terrible. Turn on Collapse Transformations and the edge is sharp.
Notice that I used two different terms here for the same button. Collapse Transformations only applies to layers that are compositions. It means that the items inside that composition are treated as if they were in the main comp. In other words when Collapse Transformations is applied we are scaling the TestShape layer 600% and then scaling that again 600% effectively making the layer 3600% bigger. Did you follow that? It's important.
Now let's go back to the PreComp and add a second 20 X 20 blue solid with a circular mask. Call the Blue Solid TestShape 2 and make it 3D and turn on CR. Add a camera to the comp. Now adjust the z value of the TestShape 2 layer so that the layer is about the same size as the 2D TestLayer. If the TestShape layer is scaled to 600% then the Z position of TestLayer 2 will be about 80% of the Zoom value of the camera. Now move the camera's X position just a little to the right so you have a Blue circle overlapping a Red circle in your comp. Turn CR on and off for the TestShape 2 layer to see how that works. Moving the TestShape 2 layer closer to the camera is exactly like scaling the layer.
Now return to your Main Comp. With CT turned on you should see only the blue circle. This is because CT is ignoring the camera in the nested comp and using only the Active Camera in the MainComp. Turn off CT and you'll see what you see in the Pre-comp but the edges will be soft. The only way to get the same view in both comps is to copy the camera in the PreComp and paste it in the Nested Comp.
Here's one more interesting thing that happens. If you make the PreComp a 3D layer it will multiply the position value of any 3D layers in the PreComp making camera moves an interesting but not impossible thing to deal with. In this example if you were to make the PreComp 3D it would look like the blue layer disappeared, but just move the camera back and it will show up again.
Now let's add a track matte by creating a new rectangular solid that is 1000 pixels wide and 200 pixels tall and add an elliptical mask. Scale this up 600% use it as a track matte and turn on and off CR. The results should be predictable. If you make a track matte layer 3D moving the Z position around will not effect what is matted out in the nested comp. If you had 2 3D layers in the nested comp you could not move the track matte between the two 3D layers and only mask one of them.
I hope this helps. It's a good exercise to help you understand how CT and CR work. Start throwing in effects and you'll learn more.