Conditional Diffusion
A generator samples a plausible image. A conditional model samples the one that matches your input. The gap between them, in this architecture, is four extra channels.
Everything so far samples from a distribution over images. Useful work needs samples from a distribution over images given something — a hazy photograph, and the clear scene and depth field behind it.
There are three mechanisms in common use for getting a condition into a diffusion model, and they differ in what they assume about the correspondence between the condition and the output. When the condition is a photograph of the same scene on the same pixel grid, that assumption picks the mechanism for you, and what remains is a set of small, specific, easily-botched decisions about pretrained weights. This topic works through those, and then asks the question that decides whether any of it was worth building: what does sampling give you that a regressor does not?
- 01 Cross-attention, concat, ControlNet Core 11′ Three mechanisms, three assumptions about where the condition lives. Spatial alignment picks one of them for you. Repairs Assuming conditioning means text prompts and cross-attention, when spatially-aligned conditioning wants a fundamentally different mechanism.
- 02 Eight channels, not four Advanced 2 interactive 12′ Widening the first convolution is one line. What its new weights start at is a real decision with three defensible answers and one right question. Repairs Thinking a pretrained UNet can simply "take another input", without asking what the new conv_in weights are initialised to.
- 03 One encoder, two decoders Advanced 1 interactive 11′ Not two models. One pretrained encoder whose skips are consumed twice — and in the default mode it runs twice, which is where the cost is. Repairs Treating multi-task output as multiple models, when it is one shared encoder whose skips are consumed twice.
- 04 Why not just regress? Advanced 12′ A regressor returns the mean of the plausible answers. A sampler returns one of them — and for an ill-posed inverse problem those differ in a way you can name. Repairs Not being able to say what a diffusion dehazer gives you that an L2 regressor does not — which is the question that decides whether any of this was worth it.