Signal and colour theory

From light to three channels

Colourimetry begins with a measurement problem. Two different spectra of light can produce the same sensation of colour, because the eye reduces the incoming spectrum to a small number of signals inside three types of cone cell. The science of colour therefore replaces the continuous spectrum with three tristimulus values, conventionally written X, Y and Z. These three values are sufficient to describe any colour that a standard observer can distinguish, and they are defined by internationally agreed colour-matching functions rather than by any particular display.

Display practice uses a related but more convenient triple: R, G and B, the intensities of three primary lights that, when added, reproduce the target colour. Real displays cannot reproduce every colour that the eye can see. The set of colours a given display can produce is its gamut, and the gamut is bounded by the triangle whose corners are its three primaries. A saturated deep green or a vivid cyan may fall outside that triangle, and no combination of the primaries will reach it. This limit is one of the reasons different standards choose different primary sets and different white points.

Luminance and chrominance

Storing a picture as three equal colour channels is wasteful, because the eye resolves brightness far more finely than it resolves colour. The remedy, adopted by every consumer video standard, is to transform the three colour channels into a luminance channel and two chrominance channels. For the common set of primaries, the transformation is a fixed weighted sum. Luminance is written Y, and the two colour-difference channels are written Cb and Cr, forming the YCbCr family of representations.

The separation is not merely a convenience. It makes the subsequent step, chroma subsampling, meaningful. Because luminance is carried at full resolution and chrominance at reduced resolution, an encoder can discard most of the colour detail while preserving the structure of the image.

Transfer functions and the gamma curve

A second transformation sits between the physical light and the stored numbers. Image sensors respond to light in an approximately linear manner, but the eye does not, and neither do most displays. If the linear signal were stored directly, dark tones would be represented by very few code values and would band badly. A transfer function, often loosely called gamma, is applied to redistribute the code values so that they are spent more evenly across the range the eye can perceive. The function is applied at capture and inverted at display, and the two operations must agree, or the picture will look either washed out or over-dark.

Modern practice distinguishes two related notions. Optical transfer describes how the code relates to light, while electrical transfer describes the voltage or digital value itself. Broadcast standards specify both, together with the reference white point and the surround conditions under which the picture is meant to be judged. The choice of transfer function affects not only tone but also the perceived saturation of colours, which is why a mismatch between the capture curve and the display curve produces both a tonal and a chromatic error.

The sampling theorem

Sampling theory states that a signal whose highest frequency is f must be sampled at more than twice f if it is to be reconstructed without error. The threshold, twice the top frequency, is the Nyquist rate. In a picture, the horizontal sampling limit fixes the finest vertical stripe pattern the picture can carry; the vertical sampling limit fixes the finest horizontal stripe pattern. In time, the frame rate fixes the fastest motion the signal can describe.

When the rule is broken, the excess detail does not disappear quietly. It reappears as a lower-frequency pattern that was never in the scene — the moiré visible on fine cloth, the false colour fringes on a striped wall, the wheel that turns backwards. Every capture and conversion stage therefore places a low-pass limit ahead of its sampler, so that frequencies above the Nyquist rate never reach the quantiser. This single precaution, more than any later processing, decides whether a digital video signal is faithful.