Core 10 min 3 of 4

What a 4-channel latent is

Enough spatial correspondence to be useful, and enough channel opacity to be misleading if you read it as a small image.

What this repairs

Reading latent channels as colour channels, and expecting latent-space edits to behave like pixel-space edits.

Assumes

An autoencoder that is not the point

A latent is $[4, 64, 64]$ from an image that was $[3, 512, 512]$ . The spatial stride is $8$ in each direction, and it is not a hyperparameter someone picked — it is the count of downsampling stages in the encoder, $2^{4-1} = 8$ for the four-level codec SD uses. That is worth knowing because it means the stride and the depth of the codec are the same fact, and neither moves without retraining.

The temptation is to read the result as a small image with a spare channel. The spatial half of that reading is roughly right and useful. The channel half is wrong in a way that survives inspection, because a latent channel rendered with a colourmap looks exactly like a meaningful map.

Pitfall Channel 0 is not red

The four channels are whatever the autoencoder's training happened to find convenient. They are not a colour space, they are not ordered by importance, they carry no agreed semantics, and they are not aligned between checkpoints — a different VAE with the same architecture will use a different basis, and the same latent tensor decoded by the two produces different images.

You can verify this without any tooling. Encode a pure red image and a pure blue one and compare their latents: the difference is spread across all four channels rather than concentrated in one, because the encoder is a stack of convolutions with no reason to preserve an input basis it was never asked to preserve.

Then look at what the channels actually do behave like: three of them tend to carry something luminance-and-structure shaped and one something chroma-opponent shaped, but tend to is the strongest honest claim, it varies by checkpoint, and building anything on it is building on an accident of training.

Where a latent cell actually reaches

$z = E(x), \qquad z \in \mathbb{R}^{4 \times 64 \times 64}, \quad x \in \mathbb{R}^{3 \times 512 \times 512}$

The map, with its shapes. The encoder is fully convolutional — no flattening, no positional embedding, no global pooling — which is the property everything below rests on.
$E(T_{8k}\,x) = T_{k}\,E(x)$

A fully convolutional encoder is translation-equivariant at its stride: shift the image by a multiple of 8 pixels and the latent shifts by that many cells. This is why a latent has a spatial layout at all and why overlaying it on the image is meaningful.
$(i, j) \;\longmapsto\; \left[8i,\, 8i+8\right) \times \left[8j,\, 8j+8\right)$

The nominal correspondence: one latent cell per 8×8 pixel block. This is the map every latent-space mask is built from, and about position it is correct.
$r_{\text{eff}} \gg 8 \quad \text{pixels}$

About extent it is wrong. The encoder is many convolutions deep, so each cell’s receptive field is far wider than its stride — a cell summarises its 8×8 block in the context of a neighbourhood many times that size, and the decoder’s reach is symmetric.
$\Delta z_{ij} \neq 0 \;\Longrightarrow\; \Delta D(z) \text{ supported well outside } 8 \times 8$

So a single-cell edit does not produce a single-block change. This is the reason latent inpainting bleeds past its mask, why latent masks need feathering, and why "edit the 2×2 cells under this 16×16 region" is a statement about position rather than about extent.

Insight Right about layout, wrong about everything else

That pair of facts is the useful summary. Position transfers: a mask, a bounding box, a spatially-aligned condition all map into latent space by dividing by eight, and they land in the right place. Value does not transfer: a brightness offset, a channel swap, a colour edit, a per-channel normalisation are all operations on a basis that has no agreed meaning.

The second half is exactly why concat conditioning works in the next topic. A hazy image and the clear image behind it share a pixel grid, so their latents share a cell grid, so stacking them on the channel axis puts corresponding content at corresponding positions. That correspondence is a geometric property, and it is the one property of the latent space you are allowed to rely on.

Check yourself

You mask a 16×16 pixel region, edit the corresponding 2×2 latent cells, and decode. The change in the output is clearly wider than 16×16. Is the mask arithmetic wrong?

Show answer

No — the arithmetic is right and the expectation is wrong. $16/8 = 2$ , and those two cells are the ones whose nominal blocks cover the region.

What spilled is the decoder's receptive field. Each latent cell participates in many convolutions on the way back up, so the pixels it influences extend well past its own $8 \times 8$ block; the influence falls off with distance rather than stopping at a boundary. Nothing in the encoder or decoder ever promised a hard block boundary — the $8\times$ figure is a stride, and a stride is about sampling rate, not about support.

The practical fix is the one every latent-space editing pipeline arrives at: feather the latent mask, composite in pixel space after decoding, or both. And the practical warning is that the spill is largest exactly where it is most visible, at high-contrast edges.