How neural style transfer works: VGG features, Gram matrices, and receptive fields.

What this post is about

Neural style transfer is the technique behind "photo of my house, painted like Van Gogh": given a content image (the house) and a style image (the painting), produce a new image that keeps the house's layout but the painting's look — brushstrokes, palette, texture. The 2015 paper by Gatys and colleagues ("A Neural Algorithm of Artistic Style") was the first to do it well, and its idea is still the cleanest introduction to a deep idea: what a trained network responds to can be used as a definition of image content — and, with one trick, of image style too.

I reimplemented the method from scratch (repo here) because reading "we match Gram matrices of feature maps" is very different from knowing why that means anything. This post builds the whole thing in order:

1. the method's strange inversion: the image is the parameter, the network is frozen;
2. the two losses — content and style — and the one object (the Gram matrix) that separates them;
3. a proof + measurement that the Gram matrix cannot encode position, and why that's the point;
4. the arithmetic that explains which network layers each loss uses.

Prerequisites: matrix multiplication and "a neural network is layers of learned filters." Everything else is defined as it appears.

The project and the history in one minute

Before Gatys, texture synthesis and style transfer were hand-crafted statistics (wavelets, patch matching). Gatys' group noticed the same year that a trained CNN's filter responses could generate textures if you matched the right statistics of those responses — and style, visually, is mostly texture. The style-transfer paper is that observation plus one more term for content. My reimplementation follows the paper directly: VGG as the frozen feature extractor, per-layer style losses, one content loss, L-BFGS updating the pixels. Everything in this post is that sentence, unpacked.

The vocabulary you'll need

• VGG: a classic ImageNet classification CNN — 13 small (3×3) convolutions stacked in 5 blocks with pooling between. We never use its classifications; we use its intermediate responses.
• Feature map: at any layer, the network holds a 3-D stack of C channels over an H × W grid — one 2-D map per filter, each map saying "where this filter fired." Flattened, that's a matrix F of shape C × (H·W): one column per pixel position, holding that position's C filter responses.
• Gram matrix: the matrix G = F Fᵀ (normalized), of shape C × C. Entry (i, j) sums, over all positions, (channel i's response) × (channel j's response). It counts co-occurrence of filters — which filters fire together — and nothing about where.
• Receptive field: the region of the input image that influences one neuron's value. Deeper layers (after more convolutions and pooling) have larger receptive fields.

The inversion: optimize the image, not the network

Normal deep learning: fix the data, train the weights. Here: fix the weights, train an image. Start from random noise. Run VGG on it. Compare VGG's responses to those of the content photo (one comparison) and the style painting (another comparison). Backpropagate the combined difference — not into the weights, but into the pixel values. Repeat a few hundred times. The noise anneals into a picture that VGG "reads" as having the photo's content and the painting's style.

So the entire method is: two comparisons, one optimizer. And the two comparisons differ in exactly one respect — whether they remember positions.

The two losses

Content — compare feature maps element-wise, position by position:

L_content = ½ ‖F^ℓ(x) − F^ℓ(p)‖²

where x is the image being synthesized and p the content photo. The activation at location (i, j) should match the photo's activation at (i, j). Positions matter — that's the content.

Style — compare Gram matrices instead:

L_style = Σ_ℓ w_ℓ ‖G^ℓ(x) − G^ℓ(s)‖²,   G = F Fᵀ / HW

where s is the style image. Now nothing says where any filter should fire — only which filters fire together, and how strongly. The total loss, α·L_content + β·L_style, is minimized over pixels with L-BFGS (a quasi-Newton optimizer suited to smooth low-dimensional problems — Gatys' choice), plus a tiny total-variation term to suppress per-pixel noise.

The proof: G is a sum over positions

Why does the Gram matrix capture "style"? Here's the algebra. Unroll the matrix product as a sum over the HW positions, with f_k the C-dimensional column of filter responses at position k:

G = F Fᵀ / HW = (1 / HW) · Σ_k=1…HW f_k f_kᵀ

It's a sum. Addition commutes — so any rearrangement of the spatial positions leaves G exactly unchanged. The Gram matrix is incapable of encoding where anything is. It records co-occurrence — "the vertical-edge filter and the warm-color filter fire together" — and position never enters.

That's not a bug; it's the design. Consider what "style" means visually: brushstrokes, weave, grain — texture. Texture is exactly the property that survives rearrangement: shuffle the patches of a brick wall and it's still a brick wall. Content — a face, a house — dies under the same shuffle. A representation blind to position is a texture detector by construction.

Assertions are cheap, so I ran it. Two synthetic feature tensors — one edge-structured (Gabor-like filter responses at fixed positions), one stationary texture (smooth random responses, the same everywhere in distribution) — then one shuffle of every single position:

The top row is the proof made physical: visibly different images, identical Gram to 18 decimal places. The bottom row is the interpretation: a genuine texture doesn't merely keep its Gram under shuffling — different samples of it share the same Gram. That's what lets the optimizer reproduce a painting's feel without reproducing any of its strokes' positions. (Formally: G is the sample covariance of filter responses, and a stationary random texture is characterized by exactly such second-order statistics — Gatys' earlier texture-synthesis paper built on this.)

The gradients (why this trains so smoothly)

Both losses are quadratic in the features, so their gradients are closed-form — worth writing down because their shape explains the method's behavior. Content, by inspection:

∂L_content/∂F = F(x) − F(p)

A per-position correction map: "make position (i, j) respond more like the photo does."

Style (differentiate the trace form, constants folded into the layer weights w_ℓ):

∂L_style/∂F = 2 (G − A) F

with A the painting's Gram. Read it: the residual (G − A) — which co-occurrences are wrong — gets projected back onto the features, and the same instruction lands at every position. The style gradient is spatially uniform because the style objective is spatially blind. One loss whispers "put things in the right places," the other hums "make everything feel the same, everywhere." The output image is their negotiated settlement.

Why style layers are shallow and content layers are deep

Every tutorial says: use early layers for style, deeper layers for content. The reason is arithmetic, and you can compute it yourself. Each 3×3 convolution grows a neuron's receptive field by 2 pixels times the accumulated stride; each 2×2 max-pool doubles the stride. Run the recurrence RF_ℓ = RF_ℓ−1 + (k−1)·s_cum across VGG16's 13 convolutions and 5 pools:

Now the layer assignment stops being folklore. A neuron whose window is 3–24 px can only detect local texture — strokes, grain, weave — and a Gram matrix over such neurons holds texture statistics: the right currency for style. A neuron with a 76–196 px window responds to arrangements of parts — where the eyes are relative to the chin — exactly the positional information the content loss needs and the Gram matrix would destroy. Gatys' layer sets (relu1_1…relu5_1 for style, relu4_2 for content) sit precisely where this computed table says they should. The architecture hands you a texture dial and an object dial; the paper just labeled them.

What I take from it

1. A representation is a choice of what to ignore. The Gram matrix isn't a trick — it's the statement "for style, position is noise," implemented in one line of linear algebra. Most feature design is this kind of statement, usually left implicit.
2. Optimization can replace training. No parameters were learned in the making of any stylized image. When the objective is differentiable and the model is fixed, synthesis-by-optimization is often cheaper and more controllable than fine-tuning anything.
3. Check invariants numerically. The permutation experiment is five lines of NumPy. It confirmed the invariance to 10⁻¹⁸ — i.e., it holds in floating-point arithmetic, not just on the whiteboard. Cheap checks like that are how derivations earn trust.

Reproduce

git clone https://github.com/anubhavagr/neural-style-transfer
cd neural-style-transfer

# drop a photo into data/content/ and a painting into data/style/, then:
python main.py     # Gatys-style optimization; outputs land in results/

The permutation experiment, if you want to repeat it yourself: build any feature tensor F, compute G = F @ F.T / F.shape[1], shuffle F's position axis with rng.permutation, recompute, and take np.abs(...).max(). If your Gram moves by more than machine epsilon, your normalization is off — and now you know how to check.

Next: on-device RAG over a whole disk → Repo on GitHub Sibling repo: no_more_BWs