Convolutional Kernels: How CNNs Learn Local Patterns

Convolutional Kernels: How CNNs Learn Local Patterns#

Twitter Handle LinkedIn Profile GitHub Profile Tag Tag

Take a \(224 \times 224\) RGB image and hand it to a dense layer. Flattened, that is \(150{,}528\) inputs; a single \(1{,}000\)-unit hidden layer costs roughly 150 million weights, and not one of them is told that pixel \((i, j)\) has anything to do with pixel \((i, j+1)\). Move the cat one pixel to the right and the entire input vector changes, so the entire representation changes with it. Why do convolutional layers — the workhorse of computer vision — sidestep both the parameter explosion and the shift problem in one move?

This three-page chapter answers that. It is written for someone who has trained a multilayer perceptron but never opened a convolutional network, and it isolates the single idea that makes vision tractable: a small, learned filter that slides across the image. The formal sliding-window operator lives in the concept chapter; the from-scratch NumPy implementation lives in the implementation chapter. By the end of the series you will be able to name the three structural priors a convolutional layer bakes in — locality, parameter sharing, and translation equivariance — explain why each cuts the dense layer’s bill, and read a feature map without panic.

Prerequisites

You need comfort with matrix multiplication and the forward pass of a one-hidden-layer MLP — we restate every equation we use, but we do not teach linear layers from scratch. The notation follows the deep-learning notation page. The discussion of why bad parameter scales harm training connects forward to the numerical-stability and initialization series, but nothing here depends on having read it.

Why a dense layer is the wrong tool for an image#

A dense (fully connected) layer connects every input element to every output unit through its own weight. For tabular data that is exactly what you want. For an image it fails on two independent counts.

Parameter explosion. Spatial resolution is expensive. A megapixel image has on the order of \(10^6\) pixels per channel, so even a modest first hidden layer carries upwards of a billion weights before it has learned anything. Most of that capacity is spent rediscovering the same local edge at every position in the frame.

Loss of spatial structure. Flattening a 2D grid into a vector throws away the only fact that makes vision tractable — that nearby pixels are correlated and distant ones mostly are not. After the flatten, the layer has no way to know that input \(42\) and input \(43\) were once neighbours. It also has no invariance to position: shift the input by one pixel and the network sees an unrelated vector, so a cat at column \(10\) and a cat at column \(11\) require two largely independent sets of weights to recognise.

A convolutional layer replaces the giant weight matrix with a small kernel (also called a filter) that slides across the image, and in doing so it bakes in three priors that address every problem above at once.

What a convolutional layer does instead#

At each spatial position, the layer places its kernel over a small neighbourhood of the input, takes a dot product, and writes the result to one pixel of the feature map. It then steps the kernel across by one position and repeats, using the same kernel weights everywhere. Three structural properties fall out of that single design choice.

Property

Dense (fully connected) layer

Convolutional layer

Parameters per output unit

one weight per input pixel: \(H \cdot W \cdot C\)

one shared \(K_h \times K_w \times C_{in}\) kernel, reused at every position

Spatial structure

flattened away before the layer sees it

preserved — each output sees one \(K_h \times K_w\) neighbourhood

Shift the input by one pixel

a completely different input vector

the output shifts by one pixel — the same features, relocated

Locality says each output unit looks only at a small neighbourhood of the input, of size \(K_h \times K_w\). The layer never connects a single output to a pixel on the far side of the image, which matches the empirical fact that visual patterns are local: an edge is made of a few adjacent pixels, not the whole frame.

Parameter sharing says the same kernel slides across every position. A \(3 \times 3\) kernel has nine weights regardless of image size, and a vertical edge detector learned at the top-left corner is applied, for free, at the bottom-right corner too. Sharing is what collapses the billion-weight first layer into something with a few thousand parameters.

Translation equivariance is the mathematical payoff of sharing: if a feature fires at position \((i, j)\) in the image, then shifting the image by one pixel makes the same feature fire at \((i+1, j)\). Shifts propagate through the layer instead of scrambling it.

Definition 78 (Translation equivariance)

Let \(f\) be an image-to-image operator and let \(T_\Delta\) denote a spatial shift by the displacement \(\Delta\). The operator \(f\) is translation equivariant when shifting the input and then applying \(f\) gives the same result as applying \(f\) and then shifting the output:

(157)#\[ f\!\left(T_\Delta \mathbf{X}\right) \;=\; T_\Delta\, f(\mathbf{X}). \]

A 2D cross-correlation whose kernel is shared across all positions satisfies this exactly. (Equivariance is weaker than translation invariance: an equivariant layer moves the feature when the input moves; an invariant layer would ignore the move altogether. Pooling, not convolution, is what buys invariance.)

The scale of the saving is not subtle. The same single hidden layer that needed ~150 million dense weights needs only a \(3 \times 3 \times 3\) kernel per output channel — \(27\) weights per channel, or \(27{,}000\) for \(1{,}000\) output channels. That is a five-thousand-fold reduction, before any cleverness in the rest of the network.

1h_in, w_in, c_in = 224, 224, 3
2units = 1000
3
4dense_params = h_in * w_in * c_in * units
5conv_params = 3 * 3 * c_in * units      # one 3x3 kernel per output channel
6
7print(f"dense layer params: {dense_params:,}")   # 150,528,000
8print(f"conv layer params:    {conv_params:,}")    # 27,000
9print(f"reduction factor:     {dense_params // conv_params:,}x")

dense layer params: 150,528,000
conv layer params:    27,000
reduction factor:     5,575x

The next page, the concept chapter, makes the sliding-window operator precise — entry by entry, then with stride, padding, and multiple channels. The page after that, the implementation chapter, turns the operator into twenty lines of NumPy you can step through yourself.

Summary

If this page had to be one sentence: a convolutional layer beats a dense layer for images because it bakes in three true priors — locality, parameter sharing, and translation equivariance — and in doing so it replaces a billion-weight first layer with a few thousand shared kernel weights.

  • A dense layer flattens the image, explodes the parameter count, and treats a shifted cat as unrelated to the original.

  • A convolutional layer keeps the 2D structure, looks only at local neighbourhoods, reuses one kernel everywhere, and propagates shifts faithfully through the feature map.

  • Equivariance (\(f(T_\Delta \mathbf{X}) = T_\Delta f(\mathbf{X})\)) is the formal statement of “shifts propagate”; it is weaker than, and different from, shift invariance.

Next: cross-correlation, stride, and padding; then the from-scratch NumPy implementation.

Further reading