Foundation 10 min 1 of 4

Destroying an image on purpose

A fixed schedule with a closed form: no network, no simulation, and any timestep in one multiply-add.

What this repairs

Thinking the model learns to add noise, or that the forward process is something the network does rather than a fixed schedule with a closed form.

The forward process is a decision, not a computation. Before any training runs, someone picks a thousand numbers $\beta_1 \dots \beta_T$ and declares that

$q(x_t \mid x_{t-1}) = \mathcal{N}\!\left(\sqrt{1-\beta_t}\,x_{t-1},\ \beta_t I\right).$

That is the whole forward process. There is no network in it, nothing is fitted to data, and it is identical for every image the model will ever see. The two coefficients are chosen so that their squares sum to one, since $(1-\beta_t) + \beta_t = 1$ . That makes the step variance-preserving: feed it unit-variance data and it returns unit-variance data, forever.

Written that way it looks like something you would have to simulate: a thousand small Gaussian steps, applied in order. You do not, and the reason is the only piece of algebra in this concept.

A thousand steps collapse into one

$x_t = \sqrt{1-\beta_t}\;x_{t-1} + \sqrt{\beta_t}\;z_t, \qquad z_t \sim \mathcal{N}(0, I)$

The definition above, written as a reparameterisation instead of a density. Each step scales what it was given down slightly and adds a fresh, independent draw.
$x_t = \sqrt{\alpha_t \alpha_{t-1}}\;x_{t-2} + \sqrt{\alpha_t \beta_{t-1}}\;z_{t-1} + \sqrt{\beta_t}\;z_t, \qquad \alpha_t = 1 - \beta_t$

Substitute the previous step into this one. Nothing has been approximated yet — this is one line of algebra, and the two noise draws are still separate terms.
$\sqrt{\alpha_t \beta_{t-1}}\;z_{t-1} + \sqrt{\beta_t}\;z_t \;=\; \sqrt{\alpha_t \beta_{t-1} + \beta_t}\;\bar{z}, \qquad \bar{z} \sim \mathcal{N}(0, I)$

This is the step that does the work, and it is the only one. Two independent zero-mean Gaussians sum to one Gaussian whose variance is the sum of theirs, so two draws become one draw and the chain stops growing.
$x_t = \sqrt{\bar{\alpha}_t}\;x_0 + \sqrt{1-\bar{\alpha}_t}\;\varepsilon, \qquad \bar{\alpha}_t = \prod_{s \le t}(1 - \beta_s)$

Induct on the previous line all the way down to x₀ and the telescoping product is what survives. Any timestep, from the clean image, in one multiply-add — no loop and no intermediate states.
$\left(\sqrt{\bar{\alpha}_t}\right)^{2} + \left(\sqrt{1-\bar{\alpha}_t}\right)^{2} = 1 \quad \text{for every } t$

Variance preservation survived the collapse, which is not automatic and is worth checking. The two coefficients are a cosine and a sine of one angle sweeping from 0 to π/2, and that identity is the whole subject of the third concept in this topic.

Insight Half the schedule is nowhere near half the noise

$\bar{\alpha}$ is a cumulative product, so it collapses far faster than $\beta$ makes it look. On the schedule SD2.1 actually uses, $\beta$ crawls from $0.00085$ to $0.012$ — a factor of fourteen across the whole chain, which reads as gentle. Meanwhile, on the textbook curve — the plain cumulative product, which is the one every line of this concept is written against:

$t$	$\bar{\alpha}_t$	$\sqrt{\bar{\alpha}_t}$	$\mathrm{SNR} = \bar{\alpha}/(1-\bar{\alpha})$
0	0.99915	0.9996	1175
199	0.75524	0.8690	3.09
354	0.49982	0.7070	0.999
499	0.27767	0.5269	0.384
999	0.00466	0.0683	0.0047

The signal-to-noise ratio passes $1$ at $t = 354$ , not at $500$ . By the literal midpoint of the chain only $27.8\%$ of the variance is signal. A reader who pictures the schedule as a linear ramp and the halfway point as half-destroyed is wrong by a third of the chain, and every intuition built on that picture — about which timesteps matter, about where the model spends its capacity — is off by the same amount.

The model in this codebase trains on a rescaled version of this curve, where the same crossing lands at $t = 337$ and the midpoint has $24.2\%$ of the variance left as signal. The note below says which curve every number in this topic is on; the argument here is identical on either, because both are cumulative products and it is the product that does this.

Convention Two curves, and which one a number is on

There are two $\bar{\alpha}$ curves in play throughout this topic, and every number below belongs to exactly one of them.

The textbook curve is $\prod_{s \le t}(1-\beta_s)$ — the cumulative product above, the one every derivation in the literature and on this page is written against. The shipped curve is that one after rescale_betas_zero_snr, which both schedulers in sd2_joint.py are constructed with. The fourth concept derives the rescale and says what defect it fixes; what matters here is only that it is a different curve, so it has different landmarks:

landmark	textbook	shipped
$\bar{\alpha}$ crosses $\tfrac{1}{2}$	$t = 354$	$t = 337$
$\bar{\alpha}$ at $t = 999$	$0.00466$	exactly $0$
$\sqrt{\bar{\alpha}}$ at $t = 999$	$0.0683$	exactly $0$
$\bar{\alpha}$ at $t = 499$	$0.27767$	$0.24236$

Ten to twenty timesteps apart, everywhere in the middle of the chain. Neither column is wrong and neither is a correction of the other — the derivations really are written against the textbook curve, and the weights really were trained on the shipped one.

The rule for this topic: a number is on the textbook curve unless it says otherwise. Where the shipped curve moves a landmark far enough to matter, both appear. Where it changes the answer qualitatively — and at $t = 999$ it does, because $0.00466$ and exactly zero are different kinds of number — the concept that turns on it says so at length.

Convention The schedule interpolates √β, not β

SD does not use a linear $\beta$ ramp. Its schedule is $\beta = \mathrm{linspace}(\sqrt{\beta_0}, \sqrt{\beta_1}, T)^2$ — square the ramp, do not ramp the square. Squaring pushes $\beta$ down near the start and leaves the end alone, so more of the chain is spent near clean data. The diffusers branch that implements it carries the comment "this schedule is very specific to the latent diffusion model", and it is right to.

Quantifying it takes care, because there are two comparisons available and they give different answers:

Same endpoints, different shape. Hold $\beta \in [0.00085, 0.012]$ and swap only the interpolation. At $t = 499$ a plain linear ramp leaves $\bar{\alpha} = 0.1618$ against $0.2777$ — a ratio of 1.72×, and the one a figure with a single pair of endpoint sliders and a shape toggle shows.
Different endpoints as well. The DDPM paper's own schedule is linear and $\beta \in [10^{-4}, 0.02]$ , which leaves $0.0786$ — a ratio of $3.53\times$ .

Both numbers are correct and they are answers to different questions. Quote the one whose comparison you are actually making, and say which endpoints are held fixed; a claim of $3.5\times$ attributed to the schedule shape is two changes reported as one.

The closed form is what makes training affordable, and the mechanism is worth being explicit about because it is the reason the training loop looks the way it does.

Without it, producing a training example at $t = 700$ would mean running seven hundred sequential steps, and a batch would be as expensive as its deepest member. With it, a training step draws $t$ uniformly at random, draws one $\varepsilon$ , and jumps straight there. The loss is therefore an expectation over timesteps,

$\mathcal{L} = \mathbb{E}_{x_0,\,\varepsilon,\,t}\Big[\,\big\lVert \text{target}(x_0, \varepsilon, t) - f_\theta(x_t, t) \big\rVert^2\,\Big],$

and one network is trained to handle all thousand noise levels at once, each one sampled rather than simulated. Training a diffusion model is a sampling problem over $t$ ; only inference is sequential.

Check yourself

You need a training example at t = 499. Do you have to run 499 noising steps to get one, and if you take the shortcut, is the result the same?

Show answer

No, and yes — in the sense that matters.

One multiply-add gets you there, and on the textbook curve the coefficients are the ones in the table above, $0.5269$ and $0.8499$ :

$x_{499} = \sqrt{\bar{\alpha}_{499}}\,x_0 + \sqrt{1-\bar{\alpha}_{499}}\,\varepsilon = 0.5269\,x_0 + 0.8499\,\varepsilon.$

On the shipped curve the same line reads $0.4923\,x_0 + 0.8704\,\varepsilon$ , which is the same argument with the schedule the model was actually trained on. Either way it is one multiply-add.

It is not the same sample the sequential route would have produced from the same seed, because a different number of Gaussians were drawn — five hundred against one. It is an exact sample from the same distribution $q(x_{499} \mid x_0)$ , which is the only property the loss depends on. The two routes differ by which draw you got, not by what you are sampling from, and no amount of running the long way round makes the training signal better.