Foundation 9 min 1 of 5

Scene-linear and sRGB

The number in the file is a perceptual code, and arithmetic on it is not arithmetic on light.

What this repairs

Believing the numbers in a PNG are proportional to light, so every blur, resize, mean and fog composite is done in a space where addition is not addition.

Two different quantities share the name "pixel value".

Scene-linear radiance is proportional to the number of photons: double the light and you double the number. It is the space every physical process lives in — a lens sums it, a sensor integrates it, fog mixes into it — and the space in which addition means addition.

Display-encoded sRGB is a perceptual code. Eight bits of linear radiance would band visibly in the shadows, where the eye is most sensitive to ratio, so the standard spends its codes unevenly, roughly as $L^{1/2.2}$ . Every PNG, every JPEG, and every image tensor that came from either is in this space unless someone converted it.

The conversion is not a pure power law — it has a linear segment near black, because a power law has infinite slope at zero:

$L = \begin{cases} c/12.92 & c \le 0.04045, \\[2pt] \left(\dfrac{c + 0.055}{1.055}\right)^{2.4} & \text{otherwise,}\end{cases}$

with $c$ the code value in $[0, 1]$ and $L$ the linear radiance. The srgbToLinear helper in the education geometry module is this function, transliterated from the same common/color.py the training data uses, so a figure and the dataset cannot disagree about what a number means.

What averaging two code values actually computes

$\tfrac{1}{2}(0 + 255) = 127.5 \;\longrightarrow\; L = 0.214$

Average two codes — black and white — the way a resize, a blur or a mean pooling layer would, and decode the answer to see what light it claims.
$\tfrac{1}{2}\big(L(0) + L(255)\big) = \tfrac{1}{2}(0 + 1) = 0.5$

Now average the light instead. Half the photons is the physically correct answer for "half of each", and it is a different number.
$L = 0.5 \;\longrightarrow\; c = 1.055 \cdot 0.5^{1/2.4} - 0.055 = 0.735 \;\longrightarrow\; 187.5$

Encoding the correct answer gives code 188, not 128. The two disagree by 60 code values on the most ordinary operation in image processing.
$\log_2\!\left(\frac{0.5}{0.214}\right) = 1.22 \text{ stops}$

Expressed the way a photographer would: averaging in the encoded space is 1.2 stops too dark. Every edge in a downsized image is darkened by some fraction of that, which is why naively resized images look muddy.

This is why the fog pipeline has a render input space setting at all. When it is set to sRGB the pipeline converts to linear before the fog render, because $I = J t + L_s(1 - t)$ is a mixture of radiances and mixing encoded codes with those weights is not a mixture of anything. Fog composited in the encoded space comes out too light in the shadows and too contrasty in the midtones — it reads as a grey wash laid over the picture, which is exactly the look people mean when they say augmented fog looks fake.

Then the sensor stage converts again, because the camera it models is looking at scene radiance, not at a display. So a sample can make the round trip twice before anything trains on it, and each 8-bit leg of that trip quantises.

Pitfall The round trip is lossy at the dark end

Decoding an 8-bit image to linear and re-encoding is not the identity you might hope for, and the reason is that the perceptual spacing is not there for decoration. A linear value of $0.02$ — an ordinary shadow — is code $39$ when encoded and byte $5$ when stored linearly. Decode into another 8-bit buffer and you have thrown away seven eighths of the shadow's precision before doing any work in it, and the re-encode posterises what is left. A figure or a preprocessing step that needs linear should decode into a float buffer and stay there until the last encode, which is what the geometry module's byte-to-linear helper is for.

The corollary is the reason the encoding exists in the first place: those same few codes cover a range the eye can distinguish, so the perceptual spacing is buying you the shadow precision that a linear 8-bit image would not have.

Check yourself

You composite a fog veil at 50 % opacity onto an 8-bit sRGB image and it looks thin in the shadows and heavy in the midtones. What is wrong?

Show answer

The blend is being done on codes, not on light. A $50\%$ blend toward the airlight is a statement about radiance; performed on the encoded values it applies a much larger fraction of the veil to the midtones — where the curve is shallow — and a much smaller one to the shadows, where it is steep. The fix is one decode and one encode around the composite, and the visible effect is that the fog stops looking like a layer and starts looking like distance.