CNN Design Principles: Locality, Weight Sharing, Equivariance

CNN Design Principles: Locality, Weight Sharing, Equivariance#

Twitter Handle LinkedIn Profile GitHub Profile Tag Tag

A convolutional layer is not defined only by its arithmetic — there is a separate page for that. It is defined by four design choices that distinguish it from a dense layer: it looks at local neighborhoods, it shares weights across positions, it responds equivariantly to shifts, and its representations grow more abstract with depth. This page is the concise reference for those principles and the vocabulary they introduce. Read it for the why; read the convolution chapter for the how.

By the end you will be able to state each principle in one sentence, explain the difference between translation equivariance and translation invariance (a distinction the literature routinely blurs), and decode the CNN vocabulary — kernel versus filter, feature map, receptive field, padding, stride — without confusion.

Prerequisites

This assumes the computer-vision chapter introduction. The full arithmetic of cross-correlation, padding, stride, and the output-shape formula lives in the convolution concept page; this page summarizes and links rather than re-deriving.

The four design principles#

Property 1 (Translation equivariance)

A layer is translation equivariant when a shift of the input produces the same shift of the output: \(f(\text{shift}(\mathbf{X})) = \text{shift}(f(\mathbf{X}))\). A convolutional layer is translation equivariant because the same kernel is applied at every position, so whatever a shift moves in the input moves identically in the feature map [Zhang et al., 2023].

Equivariance is not invariance

The literature (and many course notes) use translation invariance and translation equivariance interchangeably — they are not the same. Equivariance means the output moves with the input; invariance means the output is unchanged by the input shift. A convolutional layer is equivariant. Translation invariance is introduced later, by pooling layers that collapse spatial location, not by the convolution itself. Conflating the two is the single most common CNN vocabulary error.

Property 2 (Locality)

The earliest layers should attend to local regions of the input and ignore distant content [Zhang et al., 2023]. A kernel of size \(K \times K\) enforces this: each output element depends only on a \(K \times K\) patch of the input. Local patterns — edges, corners, textures — are aggregated across layers into whole-image understanding, so locality at the bottom is what enables globally meaningful features at the top.

Property 3 (Weight (parameter) sharing)

The same kernel weights are reused at every spatial position. An edge detector useful in the top-left corner is useful in the bottom-right too, so the network learns one detector and applies it everywhere rather than relearning it per location. This is what slashes the parameter count relative to a dense layer and what makes the layer translation equivariant — both consequences follow from the single decision to share.

Property 4 (Increasing abstraction)

As depth grows, representations become more abstract and less tied to exact pixel positions. Early layers detect edges and color blobs; deeper layers compose them into textures, parts, and objects. Depth, combined with locality and pooling, is what turns pixels into semantics.

Intuition — the four principles as one story

Locality says “look nearby”; weight sharing says “look the same way everywhere”; equivariance is what follows mechanically from sharing; abstraction is what depth builds on top. State them in that order and the convolutional layer’s design reads as a single argument, not four disconnected rules.

A CNN glossary#

These terms recur throughout the chapter; the full derivations live in the convolution concept page.

Term

Meaning

Image

A tensor \(\mathbf{X} \in \R^{C \times H \times W}\): \(C\) channels, \(H\) rows, \(W\) columns. A grayscale image has \(C = 1\); RGB has \(C = 3\).

Kernel

A small 2D weight matrix, typically \(K \times K\), slid across the image.

Filter

A 3D weight tensor of shape \(C_{\text{in}} \times K \times K\) — a stack of one kernel per input channel. See Remark 48.

Feature map

The output of applying one filter across the image; “feature map” and “output channel” are used interchangeably [Zhang et al., 2023].

Receptive field

The region of the input that contributes to a single output element. It grows with depth: a unit two layers deep sees a patch of the original image larger than one layer’s kernel.

Stride \(s\)

How many pixels the kernel shifts per step. \(s > 1\) downsamples.

Padding \(p\)

Zeros added around the input. Valid padding adds none (output shrinks); same padding adds \(p = (K - 1)/2\) (for odd \(K\)) so the output keeps the input’s spatial size.

Output size

For input \(n \times n\), kernel \(k\), padding \(p\), stride \(s\): \(\lfloor (n + 2p - k) / s \rfloor + 1\) per spatial dimension.

Two distinctions worth memorizing#

Remark 48 (A “5×5 filter” is not a 5×5 matrix)

Saying “a \(5 \times 5\) filter” does not mean the filter is a \(5 \times 5\) matrix. It means the filter is a \(C_{\text{in}} \times 5 \times 5\) tensor — one \(5 \times 5\) kernel per input channel. The \(5 \times 5\) refers to the spatial extent only; the channel dimension is implied by the previous layer.

Remark 49 (The filters are learned, not designed)

Hand-designing kernels (Sobel, Gaussian, etc.) is tedious and brittle. In a CNN every entry of every filter is a learnable parameter, set by backpropagation just like the weights of a dense layer. The network discovers its own edge and texture detectors from the data.

Two patterns the glossary implies#

Remark 50 (1×1 convolutions mix channels)

A \(1 \times 1\) kernel still spans all \(C_{\text{in}}\) channels, so a \(1 \times 1\) convolution computes a learned weighted sum across channels at each position — a per-pixel linear projection of the channel dimension. It changes the number of channels without touching the spatial size, and is the standard tool for cheap channel expansion and bottleneck blocks.

Remark 51 (Parameters come from the filters, not the positions)

Because weights are shared across positions, a layer’s parameter count is the filter size times the number of filters — independent of the image size. A \(3 \times 3\) filter bank on a \(224 \times 224\) image has the same number of weights as on a \(32 \times 32\) image. This is the practical payoff of Property 3.

Summary#

If this page had to be one sentence: a convolutional layer is local, shares weights across space, is translation equivariant (not invariant), and builds abstraction with depth — and its vocabulary (kernel, filter, feature map, receptive field, padding, stride) is just the language for those four decisions. For the arithmetic behind the glossary — the cross-correlation slide, the output-shape formula, multi-channel volumes — continue to the convolution concept page.

Further reading