Looks like a pulldown issue, duplicated frames are baked into transcoded files...

Question

Have a bit of a messy project atm...this started with a client call on my way back from a festival where I ran large format video screens as a VJ & camera switcher/editor (Sasquatch in NW USA) so I was tired from 5 solid days of work.  I dropped by their studio to examine an example clip…tired and thinking they had a good test case pulled up & ready to look at.
Well, the first clip we analyzed ONLY had shutter speed issues.  Checking the source file/.mov it was apparent it was shot at a shutter speed closer to 1/1000 than 1/60, adding a bit of pixel motion blur in AE solved that...  In retrospect his export from FCP7 had the pulldown problems in the EXPORTED version of the clip, but we hadn't checked that (we watched the source clip play back and he sent the reference clip with me so I could match the edit).  So the issues that have been in almost all subsequent clips were not present in the initial analyzed file unfortunately!   When I got back to my workspace & the files were ingested it became obvious that there's a bad pulldown at work in most of the transcoded footage.  The client had mentioned it looking 'choppy' and rather than just being motion blur it was a case of duplicate frames.  Bad pulldown removal... 
MOST of the footage as I have it is ProRes 4:2:2 LT 29.97p  with every 3rd frame duplicated (1 out of 3 duplicated rather than 1 out of 5).   The ProRes format didn't come from a digital recorder in this case, it was transcoded from the source media using a software tool to organize & collect the source media and compressor to transcode for editing.  Since these 2 episodes were left in the editing pile for months, the source files are obviously *no longer around*….
In the past I've dealt with reverse pulldown issues from 24p & 24pA (where the frames are either interlaced or duplicated to achieve the pulldown) but these all give a 5 frame cadence.  Also when done correctly they don't destroy good frames and leave frame dupes.  What we have here seems more like improperly handled PsF (via HDMI or SD output from 1080p cameras) or pulldown incorrectly applied to 29.97 over 60i/59.94i, or...?   In any case these are destined for 60i broadcast, so simply interlacing them again to create the 3rd frame is out of the question, and frame blending isn't much better than the dupe frames.
Removal of every 3rd frame which (since that = every 2nd frame) obviously gives a visual "jump" over the now removed dupe frame, and leaves footage has to be converted from 20p to 29.97 which looks horrible regardless of how its done (conformed or interpolated).    Removal of every 3rd frame from 29.97 is actually not too hard, as 2/3rds of the frames are still 'good' it's easy to
apply a conform to the file headers bringing the rate down to 19.98p, bringing that into AE and creating a comp that's set to 'preserve frame rate when nested' under advanced, then putting that into a 29.97 comp..., that works relatively well but the missing frames STILL need to be replaced somehow...
To make matters worse, some clips have a cadence that doesn't stay consistent (the repeated frames will 'jump' from every 3rd periodically), OR worse yet there are sometimes 4-6 duplicated frames in a row (???), and then other sequences have only occasional frames repeated (meaning they need to be identified manually by stepping frame by frame).  Some of this is potentially due to rounding errors in the pulldown, but there's got to be something more amiss as is evident by the other inconsistencies...very odd.
In any case what do to about the 3rd frames that need to be 'repaired' or 'created'?  This is the crux of my issue, and software interpolation alone isn't going to cut it...  Since a simple pass on the entire clip(s) causes vast sections get time stretched (not just every 3rd frame synthesized but rather the whole thing interpolated), I tried going in and setting timestretch keyframes manually (so I could manually stretch the area where the 3rd frame needs to be 'created).  As the software applies motion techniques that obviously assumes the distance between frames in both directions is equal...well it just replaces 1 visual problem (dupe frames) with another (visual distortions).   This has been the case with ALL the software tried so far...
I've tried AE's native higher quality mode (rather than just frame blending, which will look awful.  Twixtor (demo) in AE, Smoke (autodesk), DaVinci Resolve (Light version but I have access to the GPU accellerated version as well) &  Apple's Motion.  All do a horrible job (FootageFIxing directory examples in dropbox) of handling this automatically, which I expected.  However I had originally intended to simply 'replace' every duplicated frame using the interpolated footage, this resulted in WORSE matching to surrounding frames due to the poor interpolation not matching up well on many frames than frame blending, not better (portions of the frame 'jump' out of alignment with both the previous & following frames, after analysis it acts like a spine with a midpoint that is not parallel to a line drawn between the start & end points).
So dropping 29.97 to 20p then synthesizing every 3rd frame (until cadence jumps) is where I'm at.  Once done synthesized frames have to be checked and a huge % of them (when there's motion on a person in the scene or camera pans etc) have elements…or whole frames…that are incorrectly synthesized.  So pieces of surrounding frames have to be roto'd & composited into selective areas to 'patch' the portions that get 'squashed', 'stretched' or 'rippled' into position (VERY VERY noticeable).
Lastly, due to a variety of shutter speeds used the motion blur across frames is also an issue, so once a shot has been correctly re-synthesized for the missing frames it must be evaluated for whether motion blur is required, which brings us back to the rendering process that was assumed at the start of this (in other words final output of each clip may need motion blur in post before delivery).
At this point I'm left with doing the synthesis of the 3rd frame and 'repairing' the areas of that which distort too heavily, or duplicate (some of the camera pans are done to track high speed motion, birds flying through frame etc)...
Anyone have any interesting techniques I could try, beyond telling the studio I'm working with that they're hosed (that discussion is coming but I'm going to show best efforts and the timeframe required to achieve that across all shots/edits)....

Answer 1

Oh and the easier way to removing the dupe frames was just to drop it to 19.984, set (nested) comp advanced settings to 'preserve framerate when nested' and watch for the occasional rounding error when back in a 29.97 container composition.  It would be nice to be able to specify 29.976...
In any case the missing frame needs to be replaced, not removed.  The visual jump i that gap actually looks to be closer to 2 frames when motion of objects is estimated by hand.
Also (as I think I mentioned) other clips seem to have even further oddities.
The reality is that I can actually create smoother motion by doing a lot of hand massaging of the footage (scaling previous frames in simple pans, hand-rotoscoping/masking/compositing in sections of frames around missing ones to synthesize motion in onscreen talent & background objects, things tracked across the screen by the camera and so on.
But that's a heck of a lot of work and probably beyond what's achievable at present I suspect...