Latent Diffusion
Diffusion in pixel space is unaffordable at 512×512. The fix is not a better sampler — it is to run the whole process somewhere smaller, and to be honest about what that costs.
Everything in the previous topic is defined over "an image", and none of it cares what the numbers are. That is fortunate, because at the process is unaffordable — twenty-five passes of an attention-bearing network over a quarter of a million tokens, per sample.
Latent diffusion moves the entire process into a learned code: a frozen autoencoder maps the image to first, diffusion happens there, and a decoder maps back at the end. The saving is large and easy to compute. The cost is a ceiling that is fixed before diffusion starts and that no sampler, no step count and no amount of training can lift — which makes it the most important number in this topic and the one least often measured.
- 01 The 48× that makes it possible Foundation 1 interactive 10′ Not a compression trick for speed. Moving the process into a code changes what is being modelled and what can no longer be represented. Repairs Believing latent diffusion is a compression trick for speed, when it changes what the model is modelling and what it can no longer represent.
- 02 An autoencoder that is not the point Core 1 interactive 12′ A frozen, separately trained codec. Encode then decode with no diffusion at all, and whatever is lost is lost for every sample the model will ever make. Repairs Treating the VAE as part of the generative model, when it is a fixed, frozen, separately-trained codec that bounds the output before diffusion begins.
- 03 What a 4-channel latent is Core 1 interactive 10′ Enough spatial correspondence to be useful, and enough channel opacity to be misleading if you read it as a small image. Repairs Reading latent channels as colour channels, and expecting latent-space edits to behave like pixel-space edits.
- 04 The 0.18215 nobody explains Core 9′ A constant that exists so the noise schedule sees the variance it was designed for — and a division whose absence produces an image rather than an error. Repairs Copying the scaling factor from a reference implementation without knowing it exists to put latents at unit variance, which the noise schedule assumes.