Why do my nested compositions have worse video quality each nested layer I add?

Question

Using AE Mac v.13.2.0.49 with h.264 mpeg-4 footage, 4 compositions deep. Preserve Nested Resolution is on for compositions using original footage.
Footage in original comp - this comp has "preserve nested resolution":
once nested:
twice nested:
thrice nested:
At no point is the footage scaled up. All footage is at or below 100% scale. Can't read it at all by the final comp. What the heck am I doing wrong?

Answer 1

So I took a looked at your composition structure and your original screenshots and I have a few suggestions for your workflow and a few unavoidable facts about turning screen captures into videos.
First, the fonts on most computer screens and the lines in most web designs and app UI's are way too thin to allow any scaling of screen capture movies without terrible degrading. It looks like the fonts on your captured page are only one pixel wide so you cannot scale, especially scale down the footage without seriously degrading the quality. The other problem is that computer displays are seldom standard video resolutions. I have a separate monitor for doing screen captures. It's 1920 X 1080. You should also have the refresh rate set to 60 Hertz (NTSC) or 50 Hertz (PAL) depending on your location. I also turn on accessibility options and enlarge the fonts or scale up the browser window by pressing Ctrl and + on the mac to make the fonts and lines on the browser bigger.
On a Mac I suggest that you take a look at ScreenFlow instead of Camtasia. If you can improve your screen capture and do your initial editing of the screen capture in Screenflow or Camtasia, and render a suitable DI you'll have much better luck in post. Adding overlays, click callouts and other elements directly in your screen capture program is a lot faster than doing it in AE.
Last point. Looking at your comps it looks like you are just cutting up the nested comps. A lot of them are cuts to the same comp. I see the comps but I don't see position or scale values. I'm not sure what is going on in any of those nested comps and I'm not sure why they are not footage layers. This is what I'm getting at, you need to make sure that the screen capture footage and the nested comp is NOT scaled or repositioned. If you must reposition and your layer and both are an even number of pixels high and wide, then make sure that the position values are whole numbers. To make things easier the size of the comp should match the size of the video. Your selected footage size is 1424 X 1260 is not a standard size for that format. H.264/MPEG-4 AVC - Wikipedia It may work, but you would be better off using a standard sized monitor to do your screen captures. Whatever you do, do not scale up the screen capture to fill a standard HD composition. If the cut up footage needs to be a comp for some reason then collapse transformations should be on for all nested comps. Again, it would be very helpful to know what you are doing in your nested comps.
Last point, and I"m not sure whats going on in your composition so take this as a suggestion rather than a criticism, but it looks like your project is a lot of cuts made to the screen capture video and a bunch of audio effects like mouse clicks (which you should be able to capture), and an animated cursor. If that's what you are doing I would never approach this project entirely in After Effects. I'd put the screen capture in Premiere Pro, cut it, add my sound, animate my cursor if possible, and be done with the project. I can cut video in Premiere Pro about 100 times faster than I can cut in After Effects and if things need to be changed per client request, it's way easier. If there is something going on with the cursor that you can't do in Premiere Pro, then render your edit in premiere pro or import the sequence and add the cursor in AE.
To answer why your footage looks worse and worse I can only guess that you are shifting position or that the footage or the comp is scaled. If it were me I'd invest in ScreenFlow, do 90% of my cutting and compositing in Screenflow, then polish up the edit and add the other elements in Premiere Pro, if complex composites are required, do those in AE and bring them into your NLE to edit.