Gamma problems

Question

Hi there . . . . .
When I export stuff from AE in most formats (H.264, MPEG4 etc) the render is pale, washed out and much lighter than my AE project.
I know there is a gamma issue with H.264 and quicktime, but MPEG4 (in a Quicktime wrapper) now also seems to suffer the same problem, whereas it didn't pre-Snow Leopard ?
Anyone got any clues as to the root of the mismatch between AE and Quicktime player ?
P.S. If I render out as ProRes the problem disappears ?
Cheers
Lee.

Answer 1

Thanks for putting in hours of time, this whole area is a pain. I need to point out that while I understand the technical issues, the guy who knows the practical issues is out due to a family emergency so I will be a poor balm.
Where to start?
After Effects uses RGB internally to represent color. Using the export menu has Quicktime drive the whole export process, which includes having quicktime call back into AE to render an RGB frame for quicktime to go compress. When using the render queue, AE is master, it renders a frame and then pushes it to quicktime. When we do the pushing, we get the opportunity to set some setting that sometimes help with gamma and color shifts. I'll call out what we do in the render queue path that we can't do in the Export menu path due to the underlying architecture.
The problem with gamma and color shift almost exclusively occurs with Codecs that run in YCrCb. This includes all ProRes (I'm lumping in ProRes 4444 even though it can do RGBA) and even PhotoJPEG if you aren't at highest quality (I think above 75%). These codecs are often defined to be in either the Rec601 or Rec709 color space (http://en.wikipedia.org/wiki/Rec._709). My guess is PhotoJPEG is sRGB, but that is a wild guess. Quicktime makes certain assumptions about the RGB color buffers it gets. These assumptions are different in Snow Leopard, Leopard, and Windows.
Since the codec is expecting color to be represented a certain way, the assumptions and associated color conversions when going from RGB to YCrCb is where things go awry.
By default Quicktime assumes RGB buffers are in a monitor color space. I believe this means sRGB (http://en.wikipedia.org/wiki/SRGB) with a 1.8 gamma on Leopard (technically not sRGB, but what can you do), sRGB with 2.2 on Leopard, and sRGB with 2.2 on Windows. Since Rec709 and sRGB have the same color primaries, any incorrect assumptions here only appear as brightness/gamma changes.
The idealized black box export path for the Export Menu is:
AE->RGB (untagged)---Color Conversion in Quicktime--->YCrCb--->compressor.
In CS4 (maybe CS3?) we added the ability to tag the RGB buffer with the gamma. AE also will request RGB frames from Quicktime with the GAMA tag applied when importing. This is controlled with the "Legacy Quicktime Gamma" preference. I believe the export half of this will only affect the Render Queue and not the Export menu.
The intent behind using the gama tag was to make the RGB to YCrCb conversion more predictable. Alas, the tag ends up in the resultant movie and can influence other consumers of the file (like FCP). Further confounding things. This is why we added the Legacy Quicktime Gamma checkbox, we could not make every single test workflow work. The Legacy workflow does not add the tag.
Lastly, not only does each product on the market talk to Quicktime differently (i.e. some use RGB, some use YCrCb buffers directly, some obey the gama tag for color conversions, some don't), but some codecs themselves implement color conversions. Also, some 3rd party codecs install their own color conversion routines that compete with the Apple defined ones and will cause different RGB to YCrCb conversions even when NOT using their codec. Also, Quicktime makes different guesses based on which OS you are running.
Since a couple of ProRes codecs were shipped after CS4, they don't work correctly out of the box. They need some custom settings (http://blogs.adobe.com/toddkopriva/2009/12/prores-4444-colors-and-gamma-s.html).
Also, pixel dimension will also affect the color conversions as some codec use non-HD stuff as Rec601, not Rec709 which leads to another source of shifts.
In the end the RGB to YCrCb color conversions are influenced by:
1) Your export path (Export Menu, vs Render Queue)
2) The operating system you are running
3) Which codecs are installed on the system (even if you aren't using them)
4) The setting of the Legacy Quicktime Gamma Setting
5) The particular destination application.
6) The pixel dimension of the footage.
I also feel the need to point out that while ProRes may be in Rec709 (which nominally has a gamma of 2.2), Final Cut Pro will correctly convert YCrCb Rec709 to Monitor RGB (Gamma of 1.8 on Pre Snow Leopard) for display. By default After Effects does no display color correction. When importing a file that starts in YCrCb Rec709 we convert it to RGB Rec709 (Gamma 2.2) if AE does the decoding. This means something like a DV file will look different in AE and FCP on a mac with 1.8 display gamma. If you use an external preview device, I do not believe you will see any difference, but things will look different on the mac monitor.
I'm hungry enough now that I'm just going to make more errors than I already have.
I hope this helps someone.
--chris