Most people know the basics of color theory. By combining the brightness of several primary colors, you can recreate any color visible to a person. Many people know that individual colors are simply the wavelengths of the electromagnetic spectrum. But what many do not realize is how difficult the situation becomes when we strive to accurately record and reproduce color.
There are many systems involved in converting an RGB triplet value to a specific wavelength of light. This transformation must be standardized so that all software, all video decoders, video cards and monitors (even made by different manufacturers in different decades) can produce the same results from the same input data. To meet this challenge, color standards have been developed. However, displays and other technologies have evolved over time. Television went digital, compression began, and we ditched CRTs in favor of LCD and OLED. The new equipment was capable of displaying more colors at higher brightness, but the signals it received were still adapted to the capabilities of older displays.
The solution to this problem was to create a new "color space". New HD content can be produced in a new color space and new HD displays can display it. And if the old color space is converted to a new one before being displayed, the old content will not lose compatibility.
This is a working solution, but it complicates the video creation process. At every step in the process of capturing, recording, editing, encoding, decoding, compositing, and displaying, you need to consider the color space you are using. And if at least one stage of the process uses old equipment, or there is a software bug due to which the color space is not taken into account, then the result may be an incorrect image. The video can still be watched, but the colors may appear too dark or pale.
Today, the two most popular color spaces are BT.601 (also called smpte170m; I'll use both of these names in this article), which has become the standard for SD content, and BT.709, which has become the standard for HD content. There is also BT.2020, which is becoming more popular thanks to its HDR and UHD content. It is worth noting that the HD / SD division is a bit erroneous here. There are no technical limitations, this is just a traditional approach. HD content can be encoded in BT.601 and SD content in BT.709. If you take a 1080p video file and reduce it to 480p, the color space will not change automatically. Changing the color space is an additional step that is done as part of the process.
What happens if a process is not executed correctly? Let's do an experiment.
The ffmpeg program is a real miracle of our time. It is a very popular digital video tool, and the entire industry owes a lot to its developers. In my experiment, I will use this tool to create and modify video files.
First, I'll create a simple test file using ffmpeg:
ffmpeg -f rawvideo -s 320x240 -pix_fmt yuv420p -t 1 -i /dev/zero -an -vcodec libx264 -profile:v baseline -crf 18 -preset:v placebo -color_range pc -y 601.mp4
A short explanation of this command:
ffmpeg
—
-f rawvideo
— ffmpeg, , .
-s 320x240
— . , . .
-pix_fmt yuv420p
— . Yuv420p — . , yuv(0,0,0) . RGB, , .
-t 1
— 1
-i /dev/zero
— . /dev/zero — , mac. , .
-an
— , .
-vcodec libx264
— libx264.
-profile:v baseline
— h.264. h.264, .
-preset:v placebo
— libx264, . , . , .
-color_range pc
— 0 255. 16-235. - , . ,pc
,tv
.
-crf 18
- the constant rate factor option tells libx264 to create a high quality video file and use as many bits as necessary to ensure quality18
. The lower the number, the higher the quality. 18 is a very high quality.
-y
- gives ffmpeg permission to overwrite the file if it exists.
601.mp4
- the name of the resulting file.
This command creates a 601.mp4 file that is 1 second long, which can be opened and played. After running this command, we can verify that ffmpeg has not distorted the pixel values by running the following command and examining the output:
ffmpeg -i 601.mp4 -f rawvideo - | xxd
00000000: 0000 0000 0000 0000 0000 0000 0000 0000
00000010: 0000 0000 0000 0000 0000 0000 0000 0000
00000020: 0000 0000 0000 0000 0000 0000 0000 0000
00000030: 0000 0000 0000 0000 0000 0000 0000 0000
00000040: 0000 0000 0000 0000 0000 0000 0000 0000
00000050: 0000 0000 0000 0000 0000 0000 0000 0000
...
...
This hexadecimal data indicates that all pixel values after decoding are zero.
When rendering a video in Safari, we get a screenshot like this:
The question arises: what is this color space? I named the file 601.mp4, but nowhere in the command did I specify a color space, so how did Safari know which shade of green to render? How does the browser know that yuv (0,0,0) should be equal to rgb (0,135,0)? Obviously, there is an algorithm for calculating these values. In fact, this is a simple matrix multiplication. (Note: some pixel formats, including yuv420p, require a pre- and post-processing step to convert, but in this demo we will omit such subtleties.) Each color space has its own matrix. Since we didn't specify the color space matrix when encoding the video, Safari just makes an assumption. We can iterate over all the matrices, multiply all the RGB values by the inverse matrices and see what they correspond to,but let's try a more visual approach and see if we can figure out what Safari is doing.
To do this, I'll insert metadata into the file so that Safari knows which color space is being used and doesn't have to guess.
ffmpeg -f rawvideo -s 320x240 -pix_fmt yuv420p -t 1 -i /dev/zero -an -vcodec libx264 -profile:v baseline -crf 18 -preset:v placebo -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range pc -y 601vui.mp4
The ffmpeg command is pretty much the same, but I added the following:
-color_trc smpte170m
-colorspace smpte170m
-color_primaries smpte170m
This is the metadata for the color space that the file will be encoded into. I will not explain the differences between these options, because this will require another article. For now, we just set all of them the color space we need. smpte170m is the same as BT.601.
Specifying a color space does not affect how the file is encoded, the pixel values are still encoded as yuv (0,0,0). To verify this, we can run the command on the new file
ffmpeg -i zero.mp4 -f rawvideo - | xxd
. The color space flags are not ignored, but simply written in a few bits within the "video usability information" (VUI) section of the video stream header. The decoder will now search for the VUI and use it to load the desired matrix.
And here's the result:
Both with and without VUI, videos are rendered with the same color. Let's try the BT.709 file:
ffmpeg -i 601vui.mp4 -an -vcodec libx264 -profile:v baseline -crf 18 -preset:v placebo -vf "colorspace=range=pc:all=bt709" -y 709.mp4
New options:
-i 601vui.mp4
— 601vui.mp4
-vf "colorspace=all=BT.709"
— ffmpeg, . yuv rgb, . «all» — color_primaries, colorspace color_trc.
Here we take 601vui.mp4 video and use color space filter to convert to BT.709. The color space filter can read the color space of the incoming data from the vui file 601vui.mp4, so we just need to specify the color space that we want to receive on the output. By executing the
command for this file
ffmpeg -i 709.mp4 -f rawvideo - | xxd
, we get after the transformation of the color space the pixel values yuv (93,67,68). However, when rendering the file, it shouldlook the same. It's worth noting that the final results may not be identical because we continue to use 24 bits to encode each pixel, and BT.709 has a slightly larger range of colors. Therefore, some colors in BT.709 do not map exactly to BT.601 and vice versa.
By looking at the result, you can clearly see that something is wrong. The new file is rendered with rgb values of 0.157.0 - much brighter than the input file.
Let's take a close look at the file properties using the ffprobe application:
ffprobe 601vui.mp4:
Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuvj420p(pc, smpte170m), 320x240, 9 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
AND
ffprobe 709.mp4:
Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuvj420p(pc), 320x240, 5 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Most of the information here is not important to us, but we will notice that 601vui.mp4 is in the “yuvj420p (pc, smpte170m)” pixel format. This is how we understand that the file has the correct VUI. But 709.mp4 only contains "yuvj420p (pc)". It looks like the color space metadata was not included in the output file. Even though the color space filter was able to read the original color space and we specified the new space explicitly, ffmpeg did not write the correct vui to the final file.
This is bad ... Ffmpeg is the most popular video conversion tool. And he is missing out on color information. This is probably the reason why many videos do not contain color space metadata, and therefore many videos are rendered with mismatched colors. In ffmpeg's defense, this is a tricky problem. To initialize the encoder, you need to know the color space in advance. In addition, the color space can be changed with a video filter. This is a tricky problem to solve in code, but it's still sad that this behavior is standard.
You can work around it by manually adding color space metadata:
ffmpeg -i 601vui.mp4 -an -vcodec libx264 -profile:v baseline -crf 18 -preset:v placebo -vf "colorspace=range=pc:all=bt709" -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range pc -y 709vui.mp4
As a result, the color value in 709vui.mp4 will be rgb (0.132.0). The brightness of the green channel is slightly less than in 601vui.mp4, but since the color space conversion occurs with loss, and the result suits me, we will call it success.
From this we can conclude that when no color space is specified in the file, Safari thinks it is BT.601. And on the Safari side, that's a very good assumption. But as said above, BT.601 is the SD video standard and BT.709 is the HD video standard. Let's check out HD videos with and without VUI and see how Safari renders them. I used the same ffmpeg commands, only changed the resolution to 1920x1080.
In both SD and HD, the color is rendered the same. Safari ignores resolution when making an assumption about color space. Apple has been in the media and publishing space for a long time, so I expected this company's product to deliver decent results. But even if everything is so clever in Safari, it is interesting how the situation is in other browsers.
Chrome:
Chrome renders video 601 slightly darker than Safari, but 709 looks the same. I suspect the difference is due to speed optimizations in floating point computations, but this is irrelevant for our test.
I know from experience that when hardware acceleration is disabled in the settings, Chrome renders differently:
By examining this result, we can see that the values in 601 are rendered very similar to the previous ones, but the 709 files are rendered as if they did not have a VUI. From this we can conclude that Chrome, with hardware acceleration disabled, simply ignores the VUI and renders all files as if they are in 601 format. This means that all 709 files will not play correctly.
Finally, let's explore Firefox:
There is a lot to make out here. Since 709.mp4 and 709vui.mp4 look the same, you can conclude that if VUI is not available, Firefox will assume BT.709 format. Rendering 601vui.mp4 correctly means that for BT.601 content the VUI section is taken into account. However, when a BT.601 file without VUI is rendered as 709, it becomes very dark. Obviously, it is impossible to render a picture correctly without all the necessary information, but the method chosen by Firefox distorts the color more than the one chosen by the Safari and Chrome browsers.
As we can see, the video color rendering situation is still the Wild West. Unfortunately, we have only covered part of the problem. We haven't studied Windows yet. Let's do that.
Microsoft Edge:
It looks like Edge (at least on my computer) just ignores VUI and renders everything as 601.
Chrome (with hardware acceleration enabled):
The situation is not very different from the Mac. If VUI is present, it is handled correctly, but if it is not present, it is assumed to be BT.601 for SD content and BT.709 for HD content. This is the only browser in which I have seen this, but there is a certain logic to it. Since the rendering is done differently than on the Mac, I suspect that the problem is in the OS or, more likely, in something at the level of video card drivers, and this choice was not made by the Chrome development team.
Firefox behaves the same as on Mac.
For Linux, iOS, Android, Roku, Fire TV, smart TVs, game consoles, etc., I'll leave this as an exercise for the reader.
What have we learned? Most important: alwaysinclude color space metadata in your videos. If you are using ffmpeg and do not set color flags, then you are not working properly. Second, while ffmpeg is a terrific program, its popularity, ease of use, and poorly chosen defaults have been a disservice. You should never assume that the software is smart enough to figure it out on its own. The project managers of Ffmpeg, Google, Mozilla, Microsoft (and probably Nvidia and AMD) need to get together and work together to find one way. I understand that there is no good solution here, but bad and predictable is better than bad and random. Personally, I recommend always assuming BT.601 format if the VUI section is missing. This creates the least amount of distortion. Can be selected to align this FOMS standard , or even the AOM , as these organizations are pretty well represented.
Last but not least, if you have a video with no color information and need to convert or render it, then good luck!
Advertising
VDSina offers inexpensive servers with daily pay. The Internet channel for each server is 500 Megabits, protection against DDoS attacks is included in the tariff, the ability to install Windows, Linux or the OS in general from your image, and also a very convenient proprietary server control panel . It's high time to try;)
Join our chat on Telegram .
