Hilights Clipping Problem with Premiere CS3 Media Encoder

Question

I've just discovered a pretty serious problem with the way Premiere CS3 processes video to the Media Encoder.
Many of you are familiar with the term "super whites", the ability of modern cameras to capture and record levels above 100 IRE. With HDV and newer technology, recording to 109 IRE is quite common.
I was aware for some time that importing this footage into Premiere results in a hard truncation of whites above 100 IRE. However, I discovered last month that the Fast Color Corrector has a Output Levels adjustment that can rescale all the levels to fit down in the 0-100 IRE range. I verified this using the scopes in Premiere and by observing recovered details in my footage. I was even able to export frames to TIFF files and show the recovered highlights in the exported TIFFs. The difference is substantial and not subtle--it's the difference between a sky completely blown to white and one where you can see the sky is blue with clouds in it.
Last night, I discovered that the Unsharp Mask filter doesn't handle footage above 100 IRE, either. Even if I use Levels to set 0-235 RGB range, somehow, Unsharp Mask is processing the un-level-adjusted signal, regardless of stacking order of the filters.
I rendered two Blu-ray discs, one with Unsharp and the other without. To my astonishment, BOTH discs lacked any detail in the highlights--the whites were clipped at 100 IRE!
Now I turned attention to the Media Encoder. Pausing on a frame where I can see the difference between 100 IRE and 109 IRE and shades in between, I fire up the Media Encoder and note that even in ME's previews screens, the whites are already clipped. There is more highlight detail in the program monitor, whereas it's all gone in the Media Encoder's display.
It is obvious that highlight information, even scaled back to 0-235, is being treated as if it were 0-255 and clipped at 235 before passing to the encoder. This is a real disappointment, because there is another half f-stop of detail up there that I was trying to pull into the useable range of 100 IRE for the Blu-ray disc. Since it is very difficult to shoot footage and limit the peak whites in-camera to no more than 100 IRE, nearly all of us are losing valuable picture information.
Even if you can recover the highlights on the timeline through a Levels filter, it gets thrown away again on the way to the Media Encoder. Short of exporting a few hundred thousand frames as .TGA files and then loading those back in and rendering out to whatever motion CODEC, I am at a loss as to how to get Premiere to leave the whites intact.
I haven't seen this issue discussed in any detail here. Has anyone found a way around it, besides reshooting all your footage of one-time events to sub 100 IRE levels?

Answer 1

I'd rather the camera added that extra lattitide within the appropriate 0 to 100 IRE range.
That's a lot like saying you want 10 gallons of water in an 9-gallon jug.
As it is, it looks like such adjustments will need to be done on a clip by clip basis.
Or, you can do it the "Dan" way in AviSynth. I wrote a script function called SoftLimiter: It applies photoshop-style curves on the Y + U + V channels to bring the maximums within legal range… using a sort-of "soft-knee" approach to reducing extreme highlights and/or oversaturated areas.
It causes a tiny bit of posterization in the highlights, just like all 8-bit per channel color adjustments do… but it is hardly noticeable on a computer screen and even less on TVs.
SoftLimiter for AviSynth 2.5
Create a folder C:\SoftLimiterCurves\ and place the .amp curves files in that folder.
Place the other files in your AviSynth 2.5 / plugins folder.
The example below assumes DV avi files and requires a DV codec (such as Cecocida DV, MainConcept, Canopus) that can read / write in YUV formats.
Create a script like this:
AviSource("dv.avi", pixel_type="YUY2")
SoftLimiter()
Open the script in VirtualDub, use Video / Fast Recompress, Select Cedocida DV codec. File / Save as .avi
Before:
After:
And, Jim… you can place your dv2film() command at the end of this script if that's what you're trying to accomplish.