Core 10 min 5 of 5

The lens moved your labels

Distortion, crop, resize and compression act on the image and not on the depth map or the mask.

What this repairs

Assuming geometric artefacts are a nuisance on the image only, when distortion, crop and resize move pixels without moving the depth map, the mask or the auxiliary targets.

Assumes

Intrinsics: from metres to pixels · Projective Geometry & the Camera Model
Lens distortion · Projective Geometry & the Camera Model

The optics stage applies the effects that belong to the glass: radial distortion, chromatic aberration, blur, bloom, veiling glare, vignetting. Two structural facts about it are worth more than the list.

They are centred on the optical centre, not the image centre. Every one of these effects is computed from a radius built out of $K$ — $r = \lVert((u - c_x)/f_x,\; (v - c_y)/f_y)\rVert$ — so when the principal point is off-centre, as it always is by a few pixels, the vignette and the distortion are off-centre with it. That is the physically correct choice, and it is also why a figure that draws the vignette around the image centre and one that draws it around $(c_x, c_y)$ disagree in a way that looks like a bug in both.

They are a look, not a derivation. The vignette is $1 - \text{strength}\cdot\min\big((r/\rho)^2,\, 1\big)$ for a radius parameter $\rho$ — a mask with two dials on it. The physical falloff of an ideal lens is $\cos^4$ of the field angle, which has no dials at all. These are different functions, and the simulation is honest about modelling the appearance rather than the optics — but it means the vignette strength is not a number you can carry to a real lens.

Pitfall The geometric stages break the pixel-to-label correspondence

This is the one to remember from the whole topic. Lens distortion, crop and resize are applied to the RGB image only. The depth map, the sky mask and the auxiliary $\beta$ and airlight targets are written from the render stage, before the camera ran, and are never resampled to match.

So the delivered pair is misaligned. The offset is zero at the optical centre by construction and grows radially, which means it is invisible in exactly the place you would check it and worst in the corners, where the labels are least redundant. A model trained on the pair learns the average of a correspondence that shifts across the frame, and every metric computed against those targets inherits the same shift.

How far is that? At the $k_1 = 0.015$ the camera preset samples, on a lens with a $45°$ half-diagonal field of view, the figure below measures a worst-case corner drift of $1.1\%$ of the frame width and a frame-wide mean of about a fifth of that. The percentage is the number worth carrying, because the drift in pixels is $f_x k_1 r^3$ and $f_x$ scales with the delivered resolution: the same lens and the same coefficient move the corner by $2.7$ px at $256$ px wide, $7$ px at $640$ , and $21$ px at $1920$ . Note also that it depends on the field of view through the normalised radius $r$ — the narrower $640 imes480$ example in the derivation below reaches only $0.6\%$ — so a single pixel count is never the answer to this question.

The fix is not subtle: apply the same geometric transform to every raster in the sample, or record the transform and undo it before evaluation. What must not happen is assuming the two share a pixel grid because they share a resolution.

How far the labels move

$x_d = (1 + k_1 r^2 + k_2 r^4)\,x_n, \qquad r^2 = x_n^2 + y_n^2$

Radial distortion in normalised coordinates, as always applied before K. The displacement is what is left when the ideal position is subtracted.
$\Delta x_n = x_d - x_n \approx k_1 r^2 x_n \;\Longrightarrow\; \lVert\Delta\mathbf{x}_n\rVert \approx k_1 r^{3}$

Because the displacement is radial, its magnitude is the radial factor times the radius itself — so it grows as the cube of the radius, not linearly. That cubic growth is the fingerprint.
$\Delta u \approx f_x\, k_1 r^2 x_n, \qquad \Delta v \approx f_y\, k_1 r^2 y_n$

In pixels, each component is scaled by its own focal length — which is also why the drift is not radially symmetric in pixel units unless the pixels are square.
$k_1 = 0.015,\; f_x = 500,\; 640\times480 \;\Longrightarrow\; \lVert\Delta\mathbf{u}\rVert \approx 4\ \text{px at the corner}$

A coefficient of the size the camera preset samples, on a modest frame, already moves the corner by four pixels — while the centre moves by nothing at all. Nobody notices a four-pixel label error that is zero where they looked.

In practice Transport: bits, blocks and chroma

The last stage crops or resizes, quantises to 8 bits and compresses. Three effects that matter downstream:

Chroma subsampling. At 4:2:0 the colour channels are stored at half resolution in both axes, so they carry a quarter of the samples the luminance does. A model that keys on the colour of a small distant object is reading mostly interpolated data; one that keys on its brightness is not.
Block structure. JPEG quantises 8×8 DCT blocks, so the compression noise is correlated within a block and discontinuous across block boundaries — a completely different distribution from the sensor noise upstream, and one that a denoiser trained on Gaussian noise handles badly.
Effective bit depth. Bits are spent over the whole range, so a scene that occupies only the top $20\%$ of the range has $\log_2(1/0.2) \approx 2.3$ fewer effective bits. Fog does exactly that to a frame, which is why the contrast budget in the next topic has a quantisation floor in it.

Check yourself

Your segmentation masks are within half a pixel of the image in the centre and about three pixels out at the corners. Which stage is responsible?

Show answer

Radial distortion applied to the RGB only. The signature is the growth law: a crop offsets everything by a constant, a resize scales the offset linearly with radius, and only the radial term grows as $r^3$ — negligible in the middle, several pixels at the edge. Confirm it by undistorting the image with the recorded $k_1, k_2$ and re-checking the alignment; if the corners snap into place, you have found it, and the fix is to distort the masks with the same coefficients rather than to undistort every image at training time.