Why Neural Network Initialization Matters

Why Neural Network Initialization Matters#

Twitter Handle LinkedIn Profile GitHub Profile Tag Tag

Train the same network architecture twice from two different sets of initial weights and you will sometimes see one run converge cleanly while the other stalls at a poor solution — or fails to learn at all. The architecture is identical, the data is identical, the optimizer is identical. The only thing that differs is the starting point. Why should that decide whether training works?

This series answers that question. It has two halves. The first, permutation symmetry, proves a structural fact about every multilayer network: the units within a hidden layer are interchangeable — relabel them and the network computes exactly the same function. The second, weight initialization, turns that fact into a practical warning: if you initialize those interchangeable units identically, they stay identical forever, and the layer collapses to a single unit. The fix — random, scale-aware initialization — exists precisely to break that symmetry and to keep activation and gradient magnitudes stable as they cross each layer.

By the end of the series you will be able to predict, before training, whether a given initialization scheme can learn at all; explain why initializing every weight to the same constant paralyses a network; and choose between schemes like Xavier/Glorot and Kaiming/He based on the activation function and depth of your model, rather than copying a default and hoping.

Prerequisites

You need comfort with matrix multiplication and the forward and backward passes of a one-hidden-layer MLP — we restate every equation we use, but we do not teach the chain rule from scratch. The notation follows the deep-learning notation page; a passing familiarity with cross-entropy loss helps for the gradient discussion but is not required.

The two failures initialization must prevent#

A poorly initialized network fails in one of two qualitatively different ways, and the rest of the series is organized around distinguishing them.

Failure

Mechanism

Where it is resolved

Symmetry lock

Units start identical, receive identical gradients, and so update identically — the layer never discovers that its units could specialize.

Permutation Symmetry in Neural Networks Explained and Weight Initialization: Why Constant and Zero Init Fail

Scale runaway

Activation or gradient variance shrinks to zero (vanishing) or balloons (exploding) as it crosses layers, so deep networks lose signal or numerical stability.

Weight Initialization: Why Constant and Zero Init Fail

The two failures are independent. A network can be symmetry-locked yet perfectly scaled, or well-symmetry-broken yet vanishing. Good initialization prevents both at once.

Intuition — why “identical start” is a trap

Imagine two cooks in a kitchen handed the same recipe, the same ingredients, and — crucially — told to make exactly the same moves. Whatever each learns, they learn together; neither ever explores a move the other has not already made. Two cooks collapse into one. Hidden units initialized identically are those two cooks: identical inputs, identical weights, identical updates. The capacity to specialize is there in the architecture, but the training dynamics can never reach it. Symmetry must be broken by the initializer, not earned by the optimizer — because, as the next chapter shows, the optimizer provably preserves it.

Roadmap#

  1. Permutation symmetry in Neural Networks establishes the interchangeable-units property rigorously: we permute the hidden units of a one-hidden-layer MLP and prove, forward and backward, that the output and every gradient are unchanged. The proof is short and concrete — a permutation matrix, four lines of algebra — but its consequence (permutation symmetry is an exact redundancy the optimizer can never remove) is the load-bearing fact for everything that follows.

  2. Weight Initialization: Why Constant and Zero Init Fail applies that fact. We prove that constant initialization keeps a layer’s units identical forever, then derive the scale-aware schemes — Xavier/Glorot for saturating activations, Kaiming/He for ReLU families — that both break symmetry and control variance across depth.

Summary#

If this introduction had to be one sentence: initialization matters because the optimizer cannot fix a starting point that is either symmetry-locked or scale-broken — those are properties the initializer must guarantee up front. The next chapter proves the symmetry property that makes identical init fatal; the chapter after turns that proof into the practical rules (Xavier, Kaiming) you reach for every time you call nn.Linear.

Further reading