Core 12 min 2 of 4

An autoencoder that is not the point

A frozen, separately trained codec. Encode then decode with no diffusion at all, and whatever is lost is lost for every sample the model will ever make.

What this repairs

Treating the VAE as part of the generative model, when it is a fixed, frozen, separately-trained codec that bounds the output before diffusion begins.

Assumes

The 48× that makes it possible

The autoencoder is trained first, on images, with an objective that has nothing to do with diffusion — a reconstruction term, a perceptual term, a patch discriminator, and a KL regulariser with a deliberately tiny weight. Then it is frozen, and the diffusion model is trained against its output.

In the model this arc is built on, that freezing is not incidental — it is enforced in one line and explained in the next. The whole codec has its gradients disabled at construction; a flag can unfreeze the decoder alone, and the comment that guards the encoder gives the reason: the encoder must stay frozen so that the latent distribution the UNet consumes is stable. Train the encoder and you are moving the target distribution out from under a model that is simultaneously learning it. The shipped configuration leaves even the decoder frozen.

So the correct mental model is a codec, in the sense that JPEG is a codec: a fixed, lossy, previously-agreed representation that everything downstream is defined against.

The ceiling, stated so it can be measured

$z \sim p_\theta(z), \qquad \hat{x} = D(z)$

Every output the system can ever produce has this form: a latent from the diffusion model, pushed through the decoder. The decoder is the last thing that touches the result and it is not part of the sampling.
$\mathcal{R} = \{\, D(z) : z \in \mathbb{R}^{4 \times 64 \times 64} \,\}$

The reachable set is therefore the image of the decoder, and it does not depend on the diffusion model at all. A better sampler moves you around inside this set; it cannot leave it.
$x \in \mathcal{R} \;\Longleftarrow\; D(E(x)) = x$

And the cheap probe for whether a particular image is even in range is the round trip: encode it, decode it, compare. No diffusion, no noise, no sampler — one forward pass each way.
$\big\lVert x - D(E(x)) \big\rVert \;\le\; \sup_{\text{sampler}} \big\lVert x - \hat{x} \big\rVert$

The round-trip error is a floor under the error of every sample the model will ever produce of that image. This is the sentence worth carrying out of the topic: the ceiling is measurable in two lines and it is fixed before diffusion starts.

Insight Run the round trip before you debug the sampler

The experiment is: encode a batch, decode it immediately, and look at the difference — amplified, because it is small enough to be invisible at unit gain.

It costs two forward passes and it changes what a debugging session is about. If your dehazer's output is slightly soft, or thin structures are missing, or fine text has turned into text-shaped texture, the round trip tells you within a minute whether the diffusion model is failing to produce that detail or whether the decoder cannot represent it. Those two diagnoses have nothing in common and they lead to opposite work.

The failures a latent codec actually has are consistent enough to recognise: small faces, printed text, dense periodic texture, and single-pixel-wide structures. They are the things a perceptual-plus-adversarial objective is happiest to approximate, because approximating them convincingly is exactly what that objective rewards.

Convention The KL weight is small on purpose, and it has a consequence

It is called a VAE and its KL term is weighted low enough that it barely regularises. The reason is deliberate: a strongly-regularised latent would be close to $\mathcal{N}(0, I)$ and correspondingly lossy, and the point of this autoencoder is fidelity rather than a well-behaved prior — the prior is the diffusion model's job, and it is a far better prior than a unit Gaussian.

So what ships is nearly a plain autoencoder with a light smoothness nudge. Two things follow. Sampling a latent from $\mathcal{N}(0, I)$ and decoding it does not give you an image, which surprises people who take the name literally. And the latent's scale is whatever training happened to land on rather than anything normalised — which is the loose end the last concept of this topic ties off.

In practice Depth goes through the same codec, and that is a compromise

The joint model in this codebase predicts a depth map through the same VAE that was trained on photographs. A single-channel depth field is tiled to three identical channels, encoded as though it were an image, diffused as a latent, and decoded back to three channels which are then averaged into one.

Why it is defensible. A tiled depth map is a legal achromatic image, and it is an unusually easy one: smooth over most of its extent, with the structure concentrated at edges that coincide with edges in the RGB image the codec was trained on. Reusing the codec means no second autoencoder to train, no second latent space for the UNet to reconcile, and a depth latent that is already spatially registered with the image latent — which is what makes the shared encoder in the next topic possible at all. The final channel-average is not cosmetic either: it discards whatever chroma the codec invented, which is error by construction, since the input had none.

What it costs, measured. Run that round trip on three driving scenes and the naive reading of the answer is backwards. Over whole frames the codec is kinder to depth than to the photographic content it was trained on: a round-trip RMSE of $0.029$ against $0.071$ for RGB, in the same $[-1, 1]$ units. Depth comes back 2.4 times more accurately, through a codec that has never seen a depth map.

That is an artefact of averaging rather than a reprieve. Depth is smooth over most of a frame, so its mean error is dominated by pixels that carry no information at all. Split the same measurement on whether a pixel sits at a depth discontinuity and it inverts: depth's error is $0.104$ at the discontinuities against $0.0064$ everywhere else — a factor of sixteen — where RGB's own split is $0.159$ against $0.058$ , a factor of under three. Read back in metres, the median absolute error is $0.28$ m and the 95th percentile is $9.4$ m, thirty-four times larger.

So the cost is not a worse average. It is an error distribution that puts nearly all of its mass on exactly the pixels a depth map exists to locate: silhouettes, obstacle boundaries, the near edge of anything. Softened thin structures and ringing at depth discontinuities are the codec's signature rather than the model's, and they sit precisely where an obstacle boundary sits — which is why a per-frame metric will not merely miss this, it will report the opposite.

Treat both numbers as floors. Everything measured here is rendered driving imagery, and renders lack what a codec trained on photographs finds hardest — sensor noise, lens softness, demosaic texture — so camera data should cost more than this rather than less.

State it the honest way: this is a compromise that works, not a design. The argument for it is engineering economy and spatial registration; the argument against it is that a codec trained on the statistics of photographs is being applied to a quantity with none of them, and what that costs has to be stated as a distribution rather than as an average. The figure below is that measurement, split the way it has to be split.

Check yourself

Your latent-diffusion dehazer produces output that is slightly soft no matter how many sampler steps you run. Where do you look first, and what would rule the codec in or out?

Show answer

At the codec, before touching anything about diffusion — because "no matter how many steps" is the signature of a ceiling rather than of an under-converged integration.

The test is the round trip: take the ground-truth clear image, encode it, decode it, and compare that to the model's output. If the round trip is just as soft, the diffusion model is already producing everything the decoder can express and the softness is the ceiling; more steps, a better sampler and more training are all working on the wrong stage. If the round trip is sharp and the model's output is not, the problem really is upstream and the sampler is worth investigating.

Do this before reading a metric. A per-pixel score cannot tell the two cases apart, because they produce the same number.