Weight initialization

Overview

Before training, every weight matrix needs starting values. The rule used to pick their scale is the initialiser. The variants covered here differ in which layer size sets that scale: incoming connections (fan_in), outgoing connections (fan_out), or both (Glorot-style). The card's motivation is that this choice matters more as the network gets deeper 1804.02763 §1.

The only card on this page is a small study of ELU multilayer perceptrons on MNIST. It compares four initialisers at depths 2 to 8 1804.02763 §4 1804.02763 tab. 5. It does not compare against other schemes, and it does not analyse the effect of connection counts beyond those four rules 1804.02763 §1.

Variants

Glorot uniform

A uniform rule that the card attributes to Glorot and Bengio. Its bounds are not stated in the card, so an implementer has to take them from the original source and record the choice. In the depth study it has the lowest error at 2 layers (0.100) and ties for the best accuracy at 7 layers (0.983) 1804.02763 tab. 5 1804.02763 tab. 6. Card: Pedamonti 2018, MNIST MLPs.

Scaled by fan_in

A rule whose variance depends on the number of incoming connections only. The card's text calls it a variance rule 1804.02763 §1, and whether its samples are uniform or Gaussian is not stated. At 5 layers it has the best ELU accuracy of the four rules (0.983), though the other three are within 0.001 at that depth 1804.02763 tab. 6. Its error is the highest of the four at 7 layers (0.135) 1804.02763 tab. 5. Card: Pedamonti 2018, MNIST MLPs.

Scaled by fan_out

A rule whose variance depends on the number of outgoing connections only. Its distribution and bounds are also not stated 1804.02763 §1. It has the highest validation error of the four at 2, 3, 5, 6 and 8 layers 1804.02763 tab. 5, and it is tied for the lowest accuracy at 2 and 6 layers 1804.02763 tab. 6. Card: Pedamonti 2018, MNIST MLPs.

Gaussian with variance 1/fan_in

A zero-mean normal distribution with variance one over the number of incoming connections. The card gives this as the initialiser used with SELU 1804.02763 §4. In the SELU depth runs it has lower validation error than the uniform rule at 3, 5, 6, 7 and 8 layers, but higher error at 2 and 4 layers, and its accuracy ordering flips across depths [arxiv:1804.02763 tab. 7]. Card: Pedamonti 2018, MNIST MLPs.

When to use which

The study does not separate the initialisers reliably. Its best differences are a few tenths of a point on one validation split, and the paper gives no seed count or spread 1804.02763 §3. Choose on the constraints below and treat the ranking as a starting point, not a result.

Implementation sketch

The common core is one function that fills a weight matrix from a rule. Only the Gaussian rule has its scale given in the card. The uniform rules have their bounds left open in this paper, so the sketch leaves them as placeholders.

# Draw one weight matrix for a layer  [arxiv:1804.02763 §1] [§4]
# Input:  rule name, fan_in (incoming), fan_out (outgoing)
# Output: matrix W of shape (fan_out, fan_in)

def init_weights(rule, fan_in, fan_out):
    # Gaussian for SELU: mean 0, variance 1 / fan_in  [§4]
    if rule == "selu_gaussian":
        return normal(mean=0.0, var=1.0 / fan_in, shape=(fan_out, fan_in))
    # Uniform rules: bounds not stated in the card; take them from the
    # original source and record which one was used
    if rule == "glorot_uniform":
        return uniform(bounds_from_glorot(fan_in, fan_out), shape=...)
    if rule == "fan_in":
        return uniform(bounds_from_fan_in(fan_in), shape=...)
    if rule == "fan_out":
        return uniform(bounds_from_fan_out(fan_out), shape=...)
    raise ValueError(rule)

Pair the initialiser with the activation. The ELU depth runs cover all four rules 1804.02763 tab. 5, and the SELU depth runs cover the uniform and Gaussian rules 1804.02763 tab. 7.

Contradictions

• Gaussian versus uniform for SELU. The text says the Gaussian rule gives better accuracy 1804.02763 §4. The table shows the accuracy sign changes across depths, with gaps of 0.2 points or less [arxiv:1804.02763 tab. 7]. Both are reported as the paper states them.
• Fan-based rules and the abstract. The abstract promises an analysis of how incoming and outgoing connection counts affect the network 1804.02763 abstract. The body compares the four rules at fixed depth and does not give that analysis 1804.02763 §1.
• Learning rate for the ELU depth runs. The paper picks learning rate 0.1 from the sweep 1804.02763 §3, while the SELU depth runs use 0.05 1804.02763 §4. The rate for the ELU depth runs is not stated, so the ELU initialiser comparison may not be at the same rate as the SELU one.

Papers

• Pedamonti 2018, MNIST MLPs: compares ELU and SELU MLPs on MNIST at depths 2 to 8 under four initialisers, with a single split and author-reported numbers.