Source levels 16-256... how to normalize

fp615 wrote on 1/8/2014, 12:48 PM
I know it is a FAQ but I ask again hoping some nice guy can answer...

All 3 cameras I have produce avchd pal files with levels from 16 to 256. On my computer monitor black can be "blacker". I remember there is a setting to have the on-screen preview automatically converto to 0-256... but my levels are already 256, doesn't I risk to lose details on screen ?

Monitoring RGB parade I can use Brightness and contrast to bring levels 0 to 256 trying not to lose too much details.

I have to produce PAL dvd and my first bluray. I may need to produce an mp4 rendering but not for vimeo nor for youtube, just private playback.

How can I improve my workflow ? Which settings should I use for rendering and bring lev4els back into the standard values ?

Or should I do until today, ignoring the problem (videos are for personal use). ?

Comments

Marco. wrote on 1/8/2014, 1:24 PM
If your source level is 16-255 while your deepest blacks are at 16, reference white is at 235 and only regions which are brighter than reference white are between 235 and 255 then you are ready to go for most cases including DVD and Blu-ray delivery.

.
farss wrote on 1/8/2014, 1:55 PM
1) set your monitor to correctly display 16-235.
If you're using Vegas's Secondary Display Device then there's a checkbox to do this.

If you're using the internal preview monitor add the Levels FX using the Studio RGB to Computer RGB preset to the monitor. There's an FX icon at the top of the monitor pane for this. If you go down this path remember to disable the FX before rendering as it will affect the final render.

2) Correct your video so you don't lose highlights.
I use the Sony colour Curves for this purpose as they allow me to adjust everything in one FX but mostly just to roll off the highlights. You can apply this FX either to the video bus, track header or individual events as best suits your project.

Bob.
VidMus wrote on 1/8/2014, 2:23 PM
Use Sony levels and make the 'Output end:' setting to be 0.920 which will lower the high side down to 235. You will then have 16-235 for your video.

Save that setting for future use.

Calibrate your external monitor for this level.
musicvid10 wrote on 1/8/2014, 4:17 PM
Your levels in the Vegas histogram should be 16-235 for render to dvd. You have control over what highlights you want to include under the 235 ceiling by the methods already described above.
Marco. wrote on 1/8/2014, 4:24 PM
If he sets the peak levels instead of the reference white to 235 this could cause the reference white to be only 91 %/RGB 217 and reference white will be greyed down.

If Levels FX with an Output End setting of 0.920 is used and the DVD or blu-ray disc will be watched via a hardware device on a correctly calibrated LCD TV which is connected via HDMI exactly this will happen.
farss wrote on 1/8/2014, 9:55 PM
Marco said:

[I]"If he sets the peak levels instead of the reference white to 235 this could cause the reference white to be only 91 %/RGB 217 and reference white will be greyed down.

If Levels FX with an Output End setting of 0.920 is used and the DVD or blu-ray disc will be watched via a hardware device on a correctly calibrated LCD TV which is connected via HDMI exactly this will happen. "[/I]

Indeed but does this matter and if it does why?

We can obviously see brighter than white and our eye/brain is quite capable of compensating for what we see on a TV.

Bob.
DiDequ wrote on 1/9/2014, 1:47 AM
fp615,
Vegas pro 12 has a bug that you probably noticed :If 3D is enabled in Nvidiacontrol panel



As soon as you launch VP12, your monitor swiches from your setting (probably cinema) to a 3D mode.

Check with the buttons of your monitor.
Vegas pro switches from 2D to 3D very fast, but is bad for color management as Icc profile is disabled.

in this 3D mode, I can only change brightness and contrast settings of my monitor.

I wrote to Sony in september, the answer was that they would correct this in a future realease ... still waiting.
I tried MANY Nvidia drivers - this is a sony problem - no other 3D softwares change the monitor mode upon startup.

I found a trick to bypass this bug : just disable the 3D checkbox in the Nvidia control panel before starting vegas pro.
Do not worry, Vegas pro will display your film upon demand - moving from the cinema monitor mode to the 3d monitor mode.
Closing the Vegas Pro 3D window switches your monitor back to cinema mode, with Icc support (using your profile AND your calibration.)

Vegas pro just need 5 seconds to switch from 2D to 3D. (very fast if 3D NVidia checkbox is checked)

Same with your 3D TV : the 2D cinema mode is different from the 3DTV cinema mode. Did you calibrate both ?
If no, do some 3D tv calibration : you will re-discover your 3D bluerays with more details.

In the optimum working conditions described above, we can chek our black and white levels in 2D AND in 3D.

Didier.
Marco. wrote on 1/9/2014, 3:35 AM
It matters in cases you'd aim your reference white to be at level 235.
Lou van Wijhe wrote on 1/9/2014, 5:36 AM
I prefer the following workflow:

My Canon HF M41 records video in the range 14-255. I put all camera footage on a dedicated track and during editing convert it to 0-255 using the levels FX for that track only. In this way I can match it with graphics, titles, etc. (which already are 0-255 to start with) on other, dedicated tracks. For rendering to DVD or Blu-ray I do a Computer to Studio RGB conversion (0-255 to 16-235) for the entire project and everything comes out perfect. For display on a PC (Vimeo, Youtube) I uncheck the Computer to Studio RGB conversion.

Using this workflow, you can leave your monitor setting at 0-255.

Lou

P.S.
To convert your camera footage from 16-255 to 0-255 use the Sony Levels FX on the camera video track and set the import start at 0,063.
Warper wrote on 1/9/2014, 5:59 AM
Why noone mentioned color correction yet? It should fix your levels as side effect.

P.S. If you render through Sony AVC or Mainconcept AVC you need studio levels even if you render for youtube.
fp615 wrote on 1/9/2014, 6:28 AM
Thank you everybody for your answers. I will have to read all of them again.
I have 1 notebook (home) and 1 workstation with 2 monitors (office) and I will need to check settings for all of them. One of the monitor is a 10bit capable, reference-grade, HP monitor, capable of showing different color spaces and I need to check which setting it is using. I also need to calibrate the second monitor...

thank you again
Lou van Wijhe wrote on 1/9/2014, 8:54 AM
Hi Warper,

Why noone mentioned color correction yet? It should fix your levels as side effect.
I don't understand. What does color correction have to do with levels correction?

P.S. If you render through Sony AVC or Mainconcept AVC you need studio levels even if you render for youtube.
You are right. These renderers are blind to anything below 16 and over 235. My wrong. But I never checked if other render utilities also expect input in this 16-235 range.

Lou
Marco. wrote on 1/9/2014, 9:00 AM
The encoders work according to the dedicated specs and for 8 bit video these only preserve data value 0 and 255. The rest of values (1-254) is meant to be video data. Actually both Sony AVC as well as MainConcept AVC (as well as any codec implemented in Sony Vegas Pro) allows for video data between 0 and 255 for 8 bit video.

This is the common mistake of mixing up reference levels with peak levels.
farss wrote on 1/9/2014, 1:32 PM
Marco said:
[I]"It matters in cases you'd aim your reference white to be at level 235."[/I]

Indeed but why should we do that and what will go wrong if we don't?

My answer to that question is; outside of a traditionally lit television studio reference white should not be at 235 because anything brighter will be clipped.

Bob.
Marco. wrote on 1/9/2014, 2:05 PM
We should do that because almost any video standard expects reference white in 8 bit video to be level 235.

It's just a wide spread misunderstanding that levels above 235 will be clipped. This was true in analog times and this is true for certain pc environments where the software players or the grafic card might be adjusted to clip levels beyond.
It is not true for any digital based hardware device and it is not true for any video calibrated LCD display. Digital video environment is build and meant to carry video from 0-255, or more strictly from 1-254. And the specs define black level at 16, white level at 235 and brighter whites up to 254/255. If you think you don't need brighter whites than that of a sheet of paper you certainly will be fine. But there are many brighter elements shot as video and we can be lucky our digital video systems nowadays are capable to transport them without any problems.

While writing this I'm also watching (and listening) to the BBC World broadcast. And the levels they send reach from 1 to 254 with reference white set to 235 and reference black at 16. The sheets of paper in front of the speaker almost exactly match 235 while levels of other light sources, reflections and even part of their logo reach up to 255. And our TVs are capable of clearly reproduce these differences. Great!

Let me – once again – point to one of Poynton's last blog post where Poynton says:

"In acquisition, post/DI, recording, mastering, transmission, and consumer display there is no longer any need to limit the coding of 8-bit video signals to the range 16 to 235 (or 10-bit signals to the range 64 to 940); apart from avoiding all-zeros and all-ones codes across HD-SDI interfaces, there are no “legal” requirements to limit range."

Poynton and other experts fight for many years to overcome these still alive misunderstandings and they always emphasize the very need to distinguish between reference white and peak levels. Hopefully there will be some success some day ;)
farss wrote on 1/9/2014, 3:02 PM
I support Poynton's struggle but that's all it is.

Our national broadcaster still uses a legaliser or if you get unlucky they'll send your content back to you with a terse note telling you to keep your levels legal.

What happens to the millions of hours of analogue content already converted to digital?

My own monitor and Vegas is setup so that 16 displays as absolute black and 235 displays as peak white. Anything above or below that will be clipped.
Every DVD player I've checked will clip anything outside legal on its analogue outputs.

What Poynton is arguing for is going to be digital anarchy in the real world.

Bob.
Marco. wrote on 1/9/2014, 4:18 PM
"What happens to the millions of hours of analogue content already converted to digital?"

No harm. There are no peak levels above 235 in there and reference white is the brightest levels. All is fine with old analogue source in digital age. Its (reference) white is the same white as it is in our freshly produced digital video because both share the same definition for reference white. Now we just have some extra cents left to spend to the brigher lights. In analogue video there is no or only very little difference between reference and peak levels. In digital video there is a big difference.

"My own monitor and Vegas is setup so that 16 displays as absolute black and 235 displays as peak white. Anything above or below that will be clipped."

Then your digital hd equipment is not calibrated and adjusted to support rec. 709 and other common specs as for LCD displays but for your own flavour instead, which is fine for personal usage and for a person who exactly knows what he does (and I have no doubt at all you do know).
Digital video peak level usually isn't meant to be level 235. Correct gamma, gamut, color space rely on correctly set blacks and whites. Of course for someone's own private use everything what looks fine is fine. So I sometimes set my video black to RGB zero and then adust my display same way just for my personal use or for a very certain intended use (sometimes I just want to get any damned bit used out of that 8 bit dynamic range). But I don't tell the public to do same way because digital color won't work if anybody makes his own rules on it.

"What Poynton is arguing for is going to be digital anarchy in the real world."

This is your opinion. Mine is that he just makes clear what digital video means and how it could and should be used to work in the correct way. We usually shoot digital, edit digital, preview digital, export digital, deliver digital and watch on LCD or similar displays. What is sticking to analogue rules good for? You can maintain what it serves but you shouldn't tranfer analogue rules into the digital age. Digital video just doesn't work this way.

So vice versa I call it "a tightrope walk between ignorance and anarchy" what happens to players like Flashplayer and to the so called "video adjustments" offered for many grafic cards: "What, digital video is meant to carry signals above 235? Video codecs are designed to do so? Broadcasters began to do so? Video cameras work this way? – We don't care!!!"
What still happens in this area doesn't make too much sense but it is likely to change because everything around changes …

My posts tend to be ironic because user fp615 indicated he uses software players and notebooks but not video equipment to watch the videos. Though some notebooks and software players are designed to correctly mimic video systems even by default, he has a good chance it will not work this way but instead on the notebooks levels of 16-235 might be spread to 0-255. Some are at least adjustable to correctly use 16-255 levels, some can only be adjusted for the use of either 16-235 or 0-255. Some (grafic devices) can't be adjusted at all and if they spread 16-235 video to 0-255 you are left to accordingly adjust your Vegas output this way but then you'd maybe need a second output for regular TV usage.

.
Lou van Wijhe wrote on 1/9/2014, 5:00 PM
Do the following test:

Include a graphics file containing the full 0-255 levels range. Render it in Vegas and you'll find that everything under 16 and over 235 is clipped. That's what the Sony and Mainconcept renderers do (others may be different) and that's what should be taken into account. So, if you want to encompass the full dynamic range, you should offer footage converted to 16-235 to the renderer.

Lou
Marco. wrote on 1/9/2014, 5:08 PM
If you saw it clips you proofed the wrong way or you didn't use the correct color space at a certain level. Maybe you watched the exported result via an external player or via a grafic device which works different to the Vegas internal preview. Re-import the exported media into Vegas Pro and check the waveform of original source and exported source. They will match (except of small compression losses).

These codecs in Vegas Pro definitely pass the range of 0-255 untouched, no different mapping done. I regularly demonstrate this to a broader audience with various kind of test charts including pluge signals etc.
If it would clip in an 8 bit signal it would have been lost irreversible after export. This is not the case and a codec working according to rec. 709 must not do that. If a video codec works this way it would cause quite a catastrophe when doing multi render generations. Render ten times and your video is gone …

.
fp615 wrote on 1/9/2014, 5:40 PM
> My posts tend to be ironic because user fp615 indicated he uses software players and notebooks but not video equipment to watch the videos.

Discussion becames very interesting but I'm wondering if I'd better ignore video levels as usual :-))))

Actually I edit with a PC or a notebook, and to be honest I have 3 different video players that accept mp4 files but this time I should make copies of the theater show to several people, on DVD... probably the 99.99% of the people will watch the DVD on a LCD TV... some on a pc... I also wanted to try to burn my first bluray... but everything is messing up now... :-)
farss wrote on 1/9/2014, 8:36 PM
[I]"No harm. There are no peak levels above 235 in there and reference white is the brightest levels. All is fine with old analogue source in digital age. Its (reference) white is the same white as it is in our freshly produced digital video because both share the same definition for reference white. Now we just have some extra cents left to spend to the brigher lights. In analogue video there is no or only very little difference between reference and peak levels. In digital video there is a big difference."[/I]

That depends on how it was transferred / telecined. Old school telecine transfers were done off low contrast prints because the gear back then could handle the dynamic range. Modern transfers no doubt attempted to cope by shifting reference white down to leave enough headroom to handle specular highlights.

Whilst there's no harm done something will happen, older transfers will not match this new "standard". The black from old 4:3 film will not match the black bars on the side of a 16:9 HDTV, ugly.

[I]"Then your digital hd equipment is not calibrated and adjusted to support rec. 709 and other common specs as for LCD displays but for your own flavour instead, which is fine for personal usage and for a person who exactly knows what he does (and I have no doubt at all you do know).
Digital video peak level usually isn't meant to be level 235. Correct gamma, gamut, color space rely on correctly set blacks and whites. Of course for someone's own private use everything what looks fine is fine. So I sometimes set my video black to RGB zero and then adust my display same way just for my personal use or for a very certain intended use (sometimes I just want to get any damned bit used out of that 8 bit dynamic range). But I don't tell the public to do same way because digital color won't work if anybody makes his own rules on it."[/I]

No, my monitor itself is not calibrated to Rec 709, it's calibrated to sRGB so it does display 0-255. Prior to sending any video to it the 16-235 levels are converted to 0-255 so it does display video correctly. I can check this against the IRIG test patterns generated by my EX1. All this works as anticipated, black from my camera is displayed as black.

[I]" This is your opinion. Mine is that he just makes clear what digital video means and how it could and should be used to work in the correct way. We usually shoot digital, edit digital, preview digital, export digital, deliver digital and watch on LCD or similar displays. What is sticking to analogue rules good for? You can maintain what it serves but you shouldn't tranfer analogue rules into the digital age. Digital video just doesn't work this way."[/I]

I didn't decide how analogue video was to be transferred into the digital age, that's a long ago done deal. I'd agree that in hindsight it wasn't done in a logical fashion at all but we're kind of stuck with it. Given the push towards using 10 rather than 8 bits why mess around with the standards set for 8 bits.

As for myself given that my camera using either one of the Cinegamma settings or even with its default Autoknee (aka Dynamic Contrast Control) will push reference white down to accommodate highlights I make certain by keeping everything under 235 in post that none of what the camera has recorded will be lost. Yes, that means my reference white is slightly below where it should be but so what, the eye easily adjusts to this when watching the content. On the other hand I've got a good chance that the details in fabrics such as white satin will not be lost regardless of things beyond my control.

Bob.
VidMus wrote on 1/9/2014, 8:50 PM
Using the "dualgray.png" provided by musicvid10, I rendered to mpg for DVDA, mpg Sony AVC and MC, and ALL levels from 235 to 255 and levels below 16 were clipped.

I get this result with Vimeo: https://vimeo.com/83816676

I did not test YouTube. I am assuming that I will get the same results there considering the previous results I had with video uploads.

On my monitor connected via HDMI to my video card and also on my main monitor the video levels are clipped at 235 and 16.

If I put the video on my USB drive and then play it on my Roku on my big screen the levels will also be clipped there as well.

I tried it with HandBrake and got the same results with Sony AVI uncompressed and the Avid DNxHD codec for intermediate videos.

All are rendering to levels for 16 to 235 so any video with less than 16 or greater than 235 with default settings will be clipped using the default project settings.

If I set my project settings to 32 bit floating point (full range) and view transform to = off, render to mp4 with Sony I will then get greater than 235 and less than 16.

On the default settings, no changes in monitor setting will display beyond 16 to 235. The ONLY way I can get beyond 16 to 235 is to change my project settings.

Now, in certain parts of the world that may be different, but here in the USA my results ARE the way it is with the default settings.

Note: considering how much longer it takes to render with 32 bit, I will leave it alone for now. Even though the darker areas of the videos would probably look a whole lot better if I do.

The number one problem is because of the 'old' CRT analog days we are still stuck in the 16 to 235 realm. Shame as it is, I need my videos to display properly for those who view them. If I make changes to 0 to 255 or 1 to 254 (whatever) then they will not be able to view them as I intended.

I have tried to make my videos to the wider range only to see poor results on my computer as well as my TV be it mp4 through the player or DVD.

Until the industry changes its way to fully support the greater range, I am stuck with the limited 16 to 235.

So one should stick to 16 to 235 for now.

Now that is the best I know how to explain this.


musicvid10 wrote on 1/9/2014, 9:28 PM
Well said, VidMus.
VidMus wrote on 1/9/2014, 9:51 PM
Good News!

I uploaded the video I rendered using Sony mp4 5,000 kbps and 32 bit full and the whatever it was turned off to Vimeo and it shows 0 to 255 just fine. So I guess if one wants to have videos with a range greater than 16 to 235 then one would need to go this route.

https://vimeo.com/83821244

Note: Vimeo had a recommendation for this video but I was in too big of a hurry and did not see it in time to read it. Most likely suggesting it be between 16 and 235.

I got a lot of time so my next test will be to DVD and see if that works and is not re-rendered by DVDA.

Another thought on DVD, if someone out there still has an analog TV with CRT they might have problems with this type of video. I know quite a number of people who still do. I still have my Samsung 27" CRT HD television and it would break a person's back trying to carry it!

I hope this helps and thanks, musicvid10