Foundation 10 min 1 of 4

The 48× that makes it possible

Not a compression trick for speed. Moving the process into a code changes what is being modelled and what can no longer be represented.

What this repairs

Believing latent diffusion is a compression trick for speed, when it changes what the model is modelling and what it can no longer represent.

Assumes

Destroying an image on purpose · Diffusion

Take the previous topic's process literally at $512 \times 512$ and count what it asks for. Every sampler step is one forward pass of a UNet whose blocks include self-attention, and self-attention over an image is attention over its pixels as tokens. Twenty-five steps means twenty-five of those.

The arithmetic is short enough to do in full, and it is the topic's entire thesis.

The two numbers, and why they are not the same number

$512 \times 512 \times 3 = 786\,432 \quad\text{values per image}$

The pixel-space tensor the process would have to carry, noise, and predict at every step. Nothing subtle here — it is the number of scalars in a 512-pixel RGB image.
$64 \times 64 \times 4 = 16\,384 \quad\text{values per latent}$

The latent SD produces from it: an 8× spatial downsample in each direction, and 4 channels rather than 3. The extra channel means the reduction is not the naive 64× the spatial stride alone suggests.
$\frac{786\,432}{16\,384} = 48$

A 48× reduction in everything that scales with tensor size — memory, every convolution, every elementwise operation, and the noise draw itself. This is the number the topic is named after, and of the two savings it is the smaller one.
$N_{\text{pix}} = 262\,144 \;\longrightarrow\; N_{\text{lat}} = 4\,096 \quad \text{tokens}$

Attention does not see channels as tokens, it sees spatial positions. So the token count falls by the spatial factor alone — 64×, not 48×.
$\frac{N_{\text{pix}}^{2}}{N_{\text{lat}}^{2}} = \frac{6.87 \times 10^{10}}{1.68 \times 10^{7}} = 4\,096$

And attention is quadratic in token count, so the attention blocks get four thousand times cheaper while the convolutions get forty-eight times cheaper. The two savings differ by a factor of eighty-five, which is why what changes is the shape of the network and not merely its price.

Insight The saving is not uniform, and that is the interesting part

If latents were only about speed, the $48\times$ would be the story and a smaller image would do just as well. The reason they are architecturally load-bearing is that the two savings are unequal.

In pixel space, an attention layer over $262\,144$ tokens is not expensive — it is impossible at the resolutions the rest of the network wants, so a pixel-space model has to confine attention to its deepest, smallest feature maps or drop it entirely. In latent space, attention over $4\,096$ tokens is ordinary, so it can be afforded at every resolution of the UNet.

That is a change in what the model can express, not in how fast it runs. The long-range structure that self-attention provides is available throughout the network rather than only at the bottleneck, and the argument for latents is therefore about capability first and cost second.

Here is the part that "compression trick" gets wrong.

After the move, the diffusion model is no longer a model of images. It is a model of the autoencoder's code distribution. Its job is to be right about which $64 \times 64 \times 4$ tensors look like encodings of real photographs, and the map from those tensors to pixels is somebody else's problem — fixed, frozen and already trained.

Two consequences follow immediately, and both are structural rather than practical. First, anything the decoder cannot produce is unreachable: it is not merely hard to sample, it is not in the model's range at all, and no sampler, step count, guidance scheme or amount of further training changes that. Second, the prior that diffusion is learning has been reparameterised — a distance in latent space is not a distance in pixel space, so the noise schedule, which was designed with pixel statistics in mind, is now being applied to something else. The last concept in this topic is about the one line of code that patches over the difference.

Check yourself

If an 8× downsample buys 48×, would a 16× downsample buy 4× more again? What is the argument against?

Show answer

The arithmetic says yes: $32 \times 32 \times 4 = 4\,096$ values, another $4\times$ in tensor size and another $16\times$ in attention cost.

The argument against is not about compute at all. Every factor of two in spatial stride asks the decoder to invent more, and what it invents is fixed for every sample the model will ever produce — so the trade is compute against a reconstruction ceiling you cannot raise later.

The right way to settle it is therefore not to reason about it. Encode and decode a batch of real images at each candidate stride, with no diffusion involved at all, and look at what comes back. That measurement is the whole subject of the next concept, and it is a two-line experiment that people consistently skip in favour of arguing.