Mocha & Keylight: problems with a greenscreen on the iPad

Question

Hey guys,
at the moment I am doing my first greenscreen project with an iPad (and a TV). But I have a few problems with the turbulent footage.
In the first picture you can see the tracked path on the iPad with Mocha. Everything moves really fine in the final result (see second picture). But there is a lot green left in the person's face and around the screen that I need to get rid of.
So my first question: How do I get rid of those green diffuse edges around the screen and the green in the face?
I tried several keying effects including keylight etc. – nothing worked. Either the green from the plants around dissappeared or it looked really edged. Is the only possibilty using moving masks? Or is there a more precise method?
Second question: How can I isolate the hand from the greenscreen while interacting "over" the iPad? I have a few sequences where the hand of the person is interacting on the (green)screen. So I need to get the screen behind the hand although everything's moving :-/ Is that possible?
Thanks a lot for your support!

Answer 1

Green isn't wrong, but you'll get a better result using tracking and roto using a dark gray screen unless the objects passing in front of the screen are very complex. The dark gray screen allos you to pick up reflections and add them to the composite easily. It just looks more real. Mocha can generate the roto as well as the track. If you have a lot of movement then you have the same problems that you have with a bunch of movement on a green screen set. Edge contamination, motion blur, details being lost. With lots of detail and lots of movement you need a very good camera with a very good lens and a lossless or nearly lossless original. Shooting with a professional cine camera that shoots raw or 10 bit uncompressed is the only way to capture that kind of fine detail and pull it out in a key. Consumer and prosumer cameras just dont give you the color information or the resolution necessary to really do a great job with fine details. Acceptable for most work but not great.
I guess my point is that if you shoot on a green screen you have to plan for color suppression and spill control on set. Plan your back lighting, and accept the fact that you'll have to do a bunch of color correcting and color suppression that you don't have to do if you shoot for roto. This gets into art direction, costuming, and set design. It's all about planning and the quality of the photography. If you are going to suppress green then you better not have anything in the scene that is important and green. Look at what happened to the green in the flower arrangement and the green in the plants in the window in my example below. A good set designer would have worked around that or let you know that you needed to do some rough roto work on the piece to restore those greens in the shot.
I've done a ton of screen replacements for phones and ipads. We carefully plan the shoot so that there's just a bit of simple roto work. If we do need to use green screen (or blue or orange depending on the art direction of the scene) we keep the exposure levels down and the reflected light down. The biggest problem you have in your scene is that the green from the TV and the green from the tablet are so bright they are acting as light sources for the set. This is where an experienced cinematographer or vfx supervisor would have saved you a bunch of time in post.
When you're stuck with footage with a lot of spill you have to isolate that spill and try and remove as much color as you can.
Try this with keylight. I'm using your mocha.png. I set keylight to view the corrected source then set foreground color correction to on and set color suppression to green, then I set edge color correction to on and set color suppression to green. There is no green left in the image. Your green edges should be gone completely. As you can see, with these settings, the green from both screen is completely gone.
To restore the green in the flower arrangement and the window just duplicate your keyed layer, do a rough roto around the flowers and the plants and reduce the color correction and color suppression to bring back the color there.
This shows you that you'll need another layer to fix the color problems. Most of my work that involves keying results in a comp with 5 or 6 layers just to control the color, most of them copies of the main plate (green screen footage) two or three copies of the replacement footage (things that fill the green screen) a layer for light wrap, a layer for lighting effects, one or two layers for reflections if there are shiny things that are being keyed. and an adjustment layer for final color correction. Sometime two or three adjustment layers for color correction.
Great compositing is seldom if ever two layers and one instance of Keylight.