Core 12 min 2 of 4

Learning to undo one step

The network never returns an image. It supplies the one missing term of a posterior that a sampler then integrates.

What this repairs

Believing the network denoises an image in one shot, when it estimates one term of a posterior that a sampler then integrates.

Assumes

Destroying an image on purpose

Reversing the forward process means sampling from $q(x_{t-1} \mid x_t)$ , and that distribution is hopeless. It depends on the distribution of all natural images: to know what the slightly-less-noisy version of a noisy patch looked like, you have to know what patches exist in the world.

But there is a conditional that is tractable. If you also knew $x_0$ — the clean image — then $q(x_{t-1} \mid x_t, x_0)$ is Gaussian with a mean and a variance you can write down in closed form, because both of the densities it is built from are Gaussians you already have.

We do not have $x_0$ . So we estimate it. That sentence is the entire derivation of the training objective, and it is the one that gets skipped: the network is not inverting the forward process, it is supplying the single unavailable term of a posterior whose other parts are known exactly.

The posterior that is available

$q(x_{t-1} \mid x_t, x_0) = \frac{q(x_t \mid x_{t-1}, x_0)\; q(x_{t-1} \mid x_0)}{q(x_t \mid x_0)}$

Bayes, conditioned on x₀ throughout. Nothing has been assumed yet; this is a rearrangement of the definition of conditional probability.
$q(x_t \mid x_{t-1}, x_0) = q(x_t \mid x_{t-1})$

The forward process is Markov by construction, so conditioning on x₀ as well tells the step nothing new. This is why the numerator is made only of things the previous concept already wrote down.
$\propto \exp\!\left(-\frac{(x_t - \sqrt{\alpha_t}\,x_{t-1})^2}{2\beta_t} - \frac{(x_{t-1} - \sqrt{\bar{\alpha}_{t-1}}\,x_0)^2}{2(1-\bar{\alpha}_{t-1})}\right)$

Both surviving factors are Gaussian in x_{t−1} — one from the single step, one from the closed form — and the denominator does not contain x_{t−1} at all, so it is a normalising constant.
$\tilde{\mu}_t(x_t, x_0) = \frac{\sqrt{\bar{\alpha}_{t-1}}\,\beta_t}{1-\bar{\alpha}_t}\,x_0 \;+\; \frac{\sqrt{\alpha_t}\,(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\,x_t$

A product of two Gaussians in the same variable is a Gaussian; completing the square in x_{t−1} gives its mean. Note the shape of the answer — it is a weighted blend of the clean image and the sample you already hold, not a correction applied to one of them.
$\tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\,\beta_t$

The variance falls out of the same completion of the square, and it depends on the schedule alone — not on the image, not on the network, and not on anything that has to be learned.
$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\!\left(\tilde{\mu}_t\big(x_t, \hat{x}_0(x_t, t)\big),\ \tilde{\beta}_t I\right)$

The whole model, in one line: take the exact posterior and substitute an estimate for the one term you cannot observe. Everything else on this page is known in closed form before training starts.

Insight One step barely moves

Read the two coefficients of $\tilde{\mu}_t$ as the blend they are. At $t = 999$ on the textbook curve, the posterior mean is

$\tilde{\mu}_{999} = 0.000828\;\hat{x}_0 \;+\; 0.993925\;x_t.$

Eight hundredths of one percent of the estimated image. A network that returned a perfect $\hat{x}_0$ at that timestep would move the sample almost nowhere. The share of $\hat{x}_0$ rises monotonically as $t$ falls, and it rises slowly: $1.24\%$ at $t = 100$ , $5.02\%$ at $t = 20$ , and it only reaches half at $t = 1$ . On the shipped rescaled curve the two middle numbers are $1.24\%$ and $5.04\%$ — the same story, since the rescale barely moves the middle of the chain and this note is about the middle of the chain.

So "the network denoises the image" is wrong twice over. It does not produce an image — it produces one term of a posterior — and applying it once changes almost nothing. The picture appears in the last handful of steps of a long integration, and everything before that is the sampler slowly agreeing with itself.

From a KL divergence to a plain squared error

$\mathcal{L}_{t-1} = D_{\mathrm{KL}}\!\left(q(x_{t-1} \mid x_t, x_0)\ \|\ p_\theta(x_{t-1} \mid x_t)\right)$

The variational bound decomposes into one of these per timestep. This is where the objective actually comes from, and it is a divergence between distributions rather than an error on an image.
$= \frac{1}{2\tilde{\beta}_t}\left\lVert \tilde{\mu}_t - \mu_\theta \right\rVert^2 + \text{const}$

Both sides are Gaussians with one and the same fixed variance, because we chose p_θ that way. The KL between two equal-variance Gaussians is just the squared distance between their means — so a divergence becomes an L2, and it is the variance choice that did it.
$= \frac{\bar{\alpha}_{t-1}\,\beta_t^{2}}{2\tilde{\beta}_t\,(1-\bar{\alpha}_t)^{2}}\left\lVert x_0 - \hat{x}_0 \right\rVert^2$

Substitute the posterior mean for both. The two means differ only in their x₀ slot, so everything else cancels and the coefficient in front of x₀ comes out squared.
$= \frac{\beta_t^{2}}{2\tilde{\beta}_t\,\alpha_t\,(1-\bar{\alpha}_t)}\left\lVert \varepsilon - \hat{\varepsilon} \right\rVert^2$

Reparameterise x₀ through the closed form, x₀ = (x_t − √(1−ᾱ)ε)/√ᾱ. Predicting the noise and predicting the image are the same task in different coordinates, and this line is where that first shows up.
$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t}\left[\left\lVert \varepsilon - \hat{\varepsilon} \right\rVert^2\right]$

And then the weight is thrown away. This is a choice, not an identity: the unweighted loss is no longer the variational bound, it up-weights the high-noise timesteps relative to it, and it trains better. Which weighting you are implicitly choosing is the subject of the next concept.

In practice Predicting ε is predicting the score

The forward marginal is the Gaussian $q(x_t \mid x_0) = \mathcal{N}(\sqrt{\bar{\alpha}_t}x_0, (1-\bar{\alpha}_t)I)$ , so its gradient with respect to $x_t$ is available in closed form:

$\nabla_{x_t} \log q(x_t \mid x_0) = -\frac{x_t - \sqrt{\bar{\alpha}_t}\,x_0}{1-\bar{\alpha}_t} = -\frac{\varepsilon}{\sqrt{1-\bar{\alpha}_t}}.$

A network trained to predict $\varepsilon$ is therefore, up to a known negative scale factor, estimating the score of the noised data distribution. That is the bridge to score matching and Langevin dynamics, and it is why the diffusion and score-based literatures are the same literature with two vocabularies.

The sign is the part people drop. The score points toward higher density and $\varepsilon$ points away from the clean image, so they are opposed; a sampler written from a score-matching paper and a scheduler written from a DDPM paper will disagree by a minus sign that produces a confidently wrong image rather than an error.

Check yourself

Your network emits a perfect ε̂ at t = 999 — the exact noise that was added. Why is the image not finished?

Show answer

Because a perfect $\hat{\varepsilon}$ at $t = 999$ still only tells the sampler where to move by $0.08\%$ of the distance to $\hat{x}_0$ . The posterior $q(x_{998} \mid x_{999}, x_0)$ is a distribution over slightly-less-noisy images, not over clean ones, and $\tilde{\mu}$ is dominated by the $x_t$ term you already had.

There is a sharper version of the same answer. The network's output is only used to choose a distribution; the sampler then draws from it, adding $\sqrt{\tilde{\beta}_t}$ of fresh noise back in. Even a perfect network run at every timestep does not produce a deterministic image under DDPM — it produces a draw from the data distribution, which is the entire point of a generative model and the thing an $L2$ regressor cannot do.