Core 11 min 1 of 4

Cross-attention, concat, ControlNet

Three mechanisms, three assumptions about where the condition lives. Spatial alignment picks one of them for you.

What this repairs

Assuming conditioning means text prompts and cross-attention, when spatially-aligned conditioning wants a fundamentally different mechanism.

Assumes

Learning to undo one step · Diffusion
The 48× that makes it possible · Latent Diffusion

Conditioning is not one technique. The question a conditioning mechanism answers is: where does the condition enter the network, and what does it assume about the correspondence between the condition and the output?

The second half is the one that decides the design. If the condition is a text prompt, there is no correspondence at all — the word "sunset" does not belong to any region of the image, and a mechanism that tied it to one would be wrong. If the condition is a photograph of the same scene through the same lens, the correspondence is exact and known in advance, and a mechanism that has to learn it is spending capacity to rediscover something the data guarantees.

Insight Three mechanisms, compared by what they assume

Mechanism	Where it enters	Assumes	Costs
Cross-attention	every attention block, as a sequence of tokens attended to	no spatial correspondence — any token may affect any position	an attention over token-by-position pairs, at every block
Input concatenation	the first convolution, as extra channels	exact, pixel-for-pixel alignment	one wider first conv, and retraining it
Adapter / ControlNet	added into the decoder from a parallel trainable encoder	approximate spatial alignment	a second encoder's parameters and compute

Read the third column as the design rule. Cross-attention is general — it makes no spatial assumption, which is why it can carry text — and generality is expensive both in compute and in the data needed to learn a correspondence from scratch. Concatenation is specific: it hands the network the condition already registered to the output, so the alignment costs nothing to learn because it was never learned. The adapter sits between: it buys spatial conditioning without touching the base model's weights, which is the right trade when you cannot retrain the base model and the wrong one when you can.

For dehazing, the choice is not close. The hazy image and the clear image are the same scene, the same camera, the same instant, on the same pixel grid — and after encoding, as the previous topic established, on the same latent cell grid. Pixel $(u, v)$ in the condition and pixel $(u, v)$ in the target are looking at the same surface.

Concatenation puts those two at the same position in the same tensor, and every convolution in the network sees them together from the first layer onward. A cross-attention route would have to learn that position $(u, v)$ in the condition is relevant to position $(u, v)$ in the output — a fact that is true by construction, that no amount of learning can make more true, and that would cost an attention over quadratically many pairs to represent.

The same argument picks concatenation for any spatially-registered condition: segmentation maps, edge maps, normals, a previous video frame. The moment the condition stops being registered — a reference image of a different scene, a style exemplar, a text prompt — the argument inverts, and cross-attention or an adapter is right.

In practice This model keeps the text pathway and feeds it zeros

SD2.1's UNet has cross-attention in every block, and the joint model inherits all of it. It is not used. At both training and inference the model constructs encoder_hidden_states as an explicit tensor of zeros with a sequence length of one, and the comment in the training path calls it what it is — zero text embeddings, unconditional. The CLIP text encoder is never loaded at all: only the vae and unet subfolders of the checkpoint are read.

There is a matching option to freeze the cross-attention projections outright, and the docstring gives the reason plainly: those projections receive only zero-valued inputs during joint training, so they carry no informative gradient and would otherwise drift away from their pretrained values under weight decay and optimiser noise.

So a substantial fraction of this network is inference-time dead weight, kept for exactly one reason: the weights around it are pretrained, and removing the attention blocks would change what "pretrained" means. That is a real and common situation when adapting a foundation model, and it is worth seeing once — the architecture you inherit is rarely the architecture you would draw.

Convention Classifier-free guidance, named once

The standard trick for conditional diffusion is classifier-free guidance: train with the condition randomly dropped some fraction of the time so the network learns both a conditional and an unconditional predictor, then at inference extrapolate away from the unconditional one,

$\hat{\varepsilon}_w = \hat{\varepsilon}_\varnothing + w\left(\hat{\varepsilon}_c - \hat{\varepsilon}_\varnothing\right), \qquad w > 1.$

It is the reason text-to-image samples follow their prompts as closely as they do, and the guidance scale is the knob most people have actually turned.

It is not used here, and this topic is not about it. There is no guidance scale in sd2_joint.py, no unconditional branch, and no condition dropout during training — the conditioning latent is present in every forward pass. Every claim in this topic about what conditioning buys is a claim about unguided concat conditioning, and results about CFG-driven models do not transfer to it. It is named here so that it is not silently assumed, and then left alone.

Check yourself

You want to condition a pretrained diffusion model on a semantic segmentation map at the same resolution as the output. Which mechanism, and what does that commit you to?

Show answer

Concatenation, on the alignment argument: the segmentation map is registered to the output pixel-for-pixel, so the correspondence is free and any mechanism that learns it is paying for nothing.

What it commits you to is the subject of the next concept. The pretrained first convolution takes four channels and you now need eight, so it has to be replaced and its new weights initialised — which means you cannot do this without retraining, and you have a real decision to make about what the new half starts at.

If you cannot retrain the base model — no compute, or you need the original behaviour preserved exactly — that constraint, not the alignment argument, is what pushes you to an adapter instead. Notice that this is a decision about your situation rather than about the condition.