---
kind: concept
slug: weight-initialization
title: Weight initialization
summary: >-
  How the starting weights of a neural network are drawn before training. The rules compared are fan-in and fan-out scaled variances, a Glorot-style uniform rule and a Gaussian with variance 1/fan_in. Only one card covers it: an MNIST MLP study that finds the differences small and mixed across depth. Single split, author-reported, bounds partly not stated.
tags: [neural-networks, optimization, deep-learning]
papers: ["1804.02763"]
related: [activation-function, multi-layer-perceptron]
status: draft
updated_at: 2026-11-08
---

## Overview

Before training, every weight matrix needs starting values. The rule used to
pick their scale is the initialiser. The variants covered here differ in which
layer size sets that scale: incoming connections (fan_in), outgoing
connections (fan_out), or both (Glorot-style). The card's motivation is that
this choice matters more as the network gets deeper [arxiv:1804.02763 §1].

The only card on this page is a small study of ELU multilayer perceptrons on
MNIST. It compares four initialisers at depths 2 to 8 [arxiv:1804.02763 §4]
[arxiv:1804.02763 tab. 5]. It does not compare against other schemes, and it
does not analyse the effect of connection counts beyond those four rules
[arxiv:1804.02763 §1].

## Variants

### Glorot uniform

A uniform rule that the card attributes to Glorot and Bengio. Its bounds are
not stated in the card, so an implementer has to take them from the original
source and record the choice. In the depth study it has the lowest error at
2 layers (0.100) and ties for the best accuracy at 7 layers (0.983)
[arxiv:1804.02763 tab. 5] [arxiv:1804.02763 tab. 6]. Card:
[Pedamonti 2018, MNIST MLPs](/papers/1804.02763).

### Scaled by fan_in

A rule whose variance depends on the number of incoming connections only.
The card's text calls it a variance rule [arxiv:1804.02763 §1], and whether
its samples are uniform or Gaussian is not stated. At 5 layers it has the best ELU accuracy of the four rules (0.983), though
the other three are within 0.001 at that depth [arxiv:1804.02763 tab. 6].
Its error is the highest of the four at 7 layers (0.135)
[arxiv:1804.02763 tab. 5]. Card: [Pedamonti 2018, MNIST MLPs](/papers/1804.02763).

### Scaled by fan_out

A rule whose variance depends on the number of outgoing connections only.
Its distribution and bounds are also not stated [arxiv:1804.02763 §1]. It
has the highest validation error of the four at 2, 3, 5, 6 and 8 layers
[arxiv:1804.02763 tab. 5], and it is tied for the lowest accuracy at 2 and 6 layers
[arxiv:1804.02763 tab. 6]. Card: [Pedamonti 2018, MNIST MLPs](/papers/1804.02763).

### Gaussian with variance 1/fan_in

A zero-mean normal distribution with variance one over the number of incoming
connections. The card gives this as the initialiser used with SELU
[arxiv:1804.02763 §4]. In the SELU depth runs it has lower validation error
than the uniform rule at 3, 5, 6, 7 and 8 layers, but higher error at 2 and 4
layers, and its accuracy ordering flips across depths [arxiv:1804.02763
tab. 7]. Card: [Pedamonti 2018, MNIST MLPs](/papers/1804.02763).

## When to use which

The study does not separate the initialisers reliably. Its best differences
are a few tenths of a point on one validation split, and the paper gives no
seed count or spread [arxiv:1804.02763 §3]. Choose on the constraints below
and treat the ranking as a starting point, not a result.

| Situation | Use | Why |
|---|---|---|
| Shallow ELU MLP (2 layers) | Glorot uniform | Lowest error at 2 layers, 0.100 [arxiv:1804.02763 tab. 5] |
| Deeper ELU MLP (5 to 7 layers), no preference | Glorot uniform, fan_in or Gaussian | Accuracy within 0.001 of each other at 5 and 7 layers [arxiv:1804.02763 tab. 6] |
| Avoid fan_out as the default | Any of the other three | Highest error at 2, 3, 5, 6 and 8 layers [arxiv:1804.02763 tab. 5] |
| SELU MLP | Gaussian with variance 1/fan_in | Lower error at 5 of 7 depths, with the accuracy gap at most 0.2 points [arxiv:1804.02763 tab. 7] |
| Very deep networks, more than 8 layers, or other widths | Not established | Only depths 2 to 8 were tested in the depth study, and its width is not stated [arxiv:1804.02763 §4] |
| Any setting needing a tested guarantee | Run your own seeds | One split, no test-set numbers [arxiv:1804.02763 §1] |

## Implementation sketch

The common core is one function that fills a weight matrix from a rule.
Only the Gaussian rule has its scale given in the card. The uniform rules
have their bounds left open in this paper, so the sketch leaves them as
placeholders.

```pseudocode
# Draw one weight matrix for a layer  [arxiv:1804.02763 §1] [§4]
# Input:  rule name, fan_in (incoming), fan_out (outgoing)
# Output: matrix W of shape (fan_out, fan_in)

def init_weights(rule, fan_in, fan_out):
    # Gaussian for SELU: mean 0, variance 1 / fan_in  [§4]
    if rule == "selu_gaussian":
        return normal(mean=0.0, var=1.0 / fan_in, shape=(fan_out, fan_in))
    # Uniform rules: bounds not stated in the card; take them from the
    # original source and record which one was used
    if rule == "glorot_uniform":
        return uniform(bounds_from_glorot(fan_in, fan_out), shape=...)
    if rule == "fan_in":
        return uniform(bounds_from_fan_in(fan_in), shape=...)
    if rule == "fan_out":
        return uniform(bounds_from_fan_out(fan_out), shape=...)
    raise ValueError(rule)
```

Pair the initialiser with the activation. The ELU depth runs cover all four
rules [arxiv:1804.02763 tab. 5], and the SELU depth runs cover the uniform and
Gaussian rules [arxiv:1804.02763 tab. 7].

## Contradictions

- **Gaussian versus uniform for SELU.** The text says the Gaussian rule gives
  better accuracy [arxiv:1804.02763 §4]. The table shows the accuracy sign
  changes across depths, with gaps of 0.2 points or less [arxiv:1804.02763
  tab. 7]. Both are reported as the paper states them.
- **Fan-based rules and the abstract.** The abstract promises an analysis of
  how incoming and outgoing connection counts affect the network
  [arxiv:1804.02763 abstract]. The body compares the four rules at fixed
  depth and does not give that analysis [arxiv:1804.02763 §1].
- **Learning rate for the ELU depth runs.** The paper picks learning rate 0.1
  from the sweep [arxiv:1804.02763 §3], while the SELU depth runs use 0.05
  [arxiv:1804.02763 §4]. The rate for the ELU depth runs is not stated, so
  the ELU initialiser comparison may not be at the same rate as the SELU one.

## Papers

- [Pedamonti 2018, MNIST MLPs](/papers/1804.02763): compares ELU and SELU MLPs on MNIST at depths 2 to 8 under four initialisers, with a single split and author-reported numbers.