Advanced 10 min 3 of 3

Bundle adjustment

The reprojection error is the residual, and everything else is bookkeeping about which Jacobian block goes where.

What this repairs

Regarding bundle adjustment as a mysterious refinement step rather than as the same least-squares problem with an obvious residual.

Assumes

Factor graphs and sparsity
Triangulation and its uncertainty · Two-View Geometry

Bundle adjustment minimises, over all camera poses and all 3D points,

$\sum_{i}\sum_{j \in \mathcal{V}(i)} \rho\!\left( \left\lVert \pi\!\left(K,\, {}^{C_j}T_{W},\, \mathbf{X}_i \right) - \mathbf{u}_{ij} \right\rVert^2_{\Sigma_{ij}} \right),$

where $\pi$ is the projection from the first topic, $\rho$ is the robust kernel from the estimation topic, and $\mathcal{V}(i)$ is the set of cameras that saw point $i$ . There is nothing else in it. The name refers to the bundles of rays from each point, adjusted until they agree.

Minimising reprojection error — in pixels, where the noise actually is — rather than 3D distance is the correct choice and the one that makes it work at all. It weights each observation by the geometry of its own ray, which is exactly what the midpoint triangulation of the two-view topic failed to do.

Pitfall Gauge freedom: the solution is a family, not a point

Reprojection error is invariant to a global rigid transform of everything — move all cameras and all points together and every image is unchanged. For monocular reconstruction it is also invariant to a global scale. So the problem has seven unobservable directions ( $6$ rigid $+ 1$ scale), the information matrix is exactly rank-deficient by seven, and a naive solve fails or returns an arbitrary drift along them.

The fixes are all forms of choosing a gauge: fix the first camera, fix the first camera plus one distance, or add a weak prior in the null-space directions. The consequence for reporting is the part usually missed — an absolute covariance over poses is meaningless without stating the gauge, while relative covariances between poses are gauge-invariant and are what should be compared.

Check yourself

Why does fixing one camera and one baseline length remove all gauge freedom in a monocular reconstruction, while fixing one camera alone does not?

Show answer

Fixing one camera pins the six rigid degrees of freedom — the reconstruction can no longer be translated or rotated. It cannot, however, prevent everything from being scaled about that camera, which leaves the images unchanged. Fixing one additional distance — a baseline, or one point's depth — pins the seventh. Together they reduce the seven-dimensional family of equivalent solutions to a single point, and the information matrix becomes full rank.