The Camera Pipeline
A pixel value is not a measurement of light. It is the output of four lossy stages that each had an opinion.
Projective geometry says where a scene point lands. It says nothing about the number stored there — and that number has been through four stages, each lossy, most of them nonlinear, and none of them recorded in the file: a lens, a sensor, an image signal processor, and whatever compressed it on the way out.
This matters here for a specific reason. Everything downstream — a fog composite, a blur, an average, a loss — is arithmetic, and arithmetic assumes the numbers mean something additive. For most of this pipeline they do not. This topic goes through the four stages in order, says what each one destroys, and names the ones that quietly break the correspondence between a pixel and its label.
- 01 Scene-linear and sRGB Foundation 1 interactive 9′ The number in the file is a perceptual code, and arithmetic on it is not arithmetic on light. Repairs Believing the numbers in a PNG are proportional to light, so every blur, resize, mean and fog composite is done in a space where addition is not addition.
- 02 Exposure is a decision, not a measurement Core 1 interactive 11′ A darker picture is a gain the camera chose, from a weighted average of the frame you never saw. Repairs Reading a darker image as a darker world, when the camera chose a gain from a metering rule that was never recorded alongside the image.
- 03 Photons, electrons, and why σ is not a constant Core 2 interactive 12′ The noise is a line through variance and mean, and ISO changes its slope by shrinking the well. Repairs Modelling image noise as additive Gaussian with one σ, so both the denoiser and the augmentation are calibrated to a distribution the sensor does not produce.
- 04 Everything that is not linear any more Core 1 interactive 11′ Tone, gamma, local contrast and sharpening run before anything you train sees the image. Repairs Treating the ISP as a colour-space conversion, so tone mapping, local contrast and sharpening read as cosmetic rather than as an irreversible nonlinearity applied before the network sees anything.
- 05 The lens moved your labels Core 1 interactive 10′ Distortion, crop, resize and compression act on the image and not on the depth map or the mask. Repairs Assuming geometric artefacts are a nuisance on the image only, when distortion, crop and resize move pixels without moving the depth map, the mask or the auxiliary targets.