Comparison of non-linear activation functions for deep neural networks on MNIST classification task

In plain words

A neural network is built from layers of simple units. Each unit adds up its inputs and passes the sum through a bend, the activation function. The classic bend is the sigmoid, which squashes everything into a fixed range. Newer bends (ReLU and its variants) are simpler and keep a useful signal flowing when a network has many layers.

This paper tests five such bends on a small handwritten-digit task (MNIST). It runs each one at several learning rates, then picks one bend and makes the network deeper, one layer at a time. It also tries several ways of choosing the starting weights, since the start matters more as the network gets deeper. The main lesson the card records is that the bends the author tested all beat the sigmoid on this task, and that deeper networks were a little more accurate up to a point, with the best starting-weight rule depending on the depth.

TL;DR

• Five activation functions (sigmoid, ReLU, Leaky ReLU, ELU, SELU) are compared as 2-hidden-layer, 100-unit MLPs on MNIST at four learning rates §3.
• Every ReLU-family function beats sigmoid on validation accuracy at the same learning rate tab. 4. ELU is chosen for the depth study §3.
• With ELU, validation accuracy rises from 2 to about 5–7 hidden layers, then flattens; the best value is 0.983 tab. 6.
• The paper is a single-student coursework report with one split, no test-set numbers and no code. Treat the differences between activations (about 0.2 to 0.5 points) as within noise until repeated.

Problem

Activation functions shape how gradients flow through deep networks. The paper notes that the sigmoid lets gradients vanish in deep stacks, that ReLU passes gradient 1 for positive inputs, and that ReLU units can "die" (stay at zero for every input) §2. It asks which of the ReLU variants and SELU helps on a standard classification task, and whether depth and the starting-weight rule change the result §1. It states no assumptions beyond the MNIST setup.

Method

The study is an empirical comparison, not a new method. It has three parts.

1. Activation sweep §3. Five activations are trained as MLPs with two hidden layers of 100 units. The learning rate takes the values 0.01, 0.05, 0.1 and 0.2. Training and validation error and accuracy are recorded tab. 1–tab. 4. The text describes the sweep as four runs per activation, one per learning rate; whether these were single seeds is not stated.
2. ELU vs SELU §3 fig. 3 fig. 4. ELU is kept over SELU because it gave better loss and accuracy in most of the author's runs. The SELU runs that were better sometimes, mostly at learning rate 0.05, are not reported.
3. Depth and initialisation §4. ELU networks with 2 to 8 hidden layers are trained under four initialisers: a uniform rule ("Glorot uniform"), a variance scaled by fan_in, a variance scaled by fan_out, and a Gaussian Glorot rule tab. 5 tab. 6. A SELU run with 2 to 8 layers compares a uniform and a Gaussian initialiser tab. 7 fig. 6.

The activation functions, with the derivative the paper gives for each:

# Activation functions used in the comparison  [§2] [eq. 1]-[eq. 8]
# Input:  x (scalar pre-activation)
# Output: activation value; the derivative is noted in each comment

def relu(x):
    # max(0, x); derivative is 0 for x <= 0 and 1 for x > 0  [eq. 1] [eq. 2]
    return max(0.0, x)

def leaky_relu(x, alpha=0.01):
    # alpha * x for x <= 0, x otherwise; alpha = 0.01  [eq. 3] [eq. 4]
    # derivative is alpha for x <= 0 and 1 for x > 0
    if x > 0:
        return x
    return alpha * x

def elu(x, alpha=1.0):
    # alpha * (exp(x) - 1) for x <= 0, x otherwise; alpha = 1  [eq. 5]
    # derivative is elu(x) + alpha for x <= 0 and 1 for x > 0  [eq. 6]
    if x > 0:
        return x
    return alpha * (exp(x) - 1.0)

def selu(x, alpha=1.6733, lam=1.0507):
    # lam times elu(x), with the constants as printed  [eq. 7] [eq. 8]
    # derivative is elu(x) + lam * alpha for x <= 0, and lam for x > 0
    return lam * elu(x, alpha)

The sigmoid baseline is not given a formula in the paper §2.

The starting-weight rules, as the paper describes them. The scale is only given for the SELU Gaussian case.

# Weight initialisers compared  [§1] [§4]
# Input:  fan_in (incoming connections), fan_out (outgoing connections)
# Output: weight matrix of one layer

def init_fan_in(fan_in, fan_out):
    # Uniform rule whose variance depends on incoming connections only
    # Exact bounds are not stated  [§1]
    return uniform_bounds_not_stated(fan_in)

def init_fan_out(fan_in, fan_out):
    # Uniform rule whose gradient variance depends on outgoing
    # connections only; exact bounds not stated  [§1]
    return uniform_bounds_not_stated(fan_out)

def init_glorot_uniform(fan_in, fan_out):
    # Glorot and Bengio uniform rule; bounds not stated in this paper  [§4]
    return uniform_bounds_not_stated(fan_in, fan_out)

def init_selu_gaussian(fan_in):
    # Normal with mean 0 and variance 1 / n_i, n_i = incoming connections  [§4]
    return normal(mean=0.0, variance=1.0 / fan_in)

The training loop is not described beyond these settings:

# Experimental protocol  [§1] [§3] [§4]
# Input:  activation, initialiser, depth, learning rate
# Output: validation error and accuracy after training

def run_configuration(activation, init, depth, learning_rate):
    # Sweep: depth 2, 100 units per layer, 100 epochs  [§3]
    # Depth study: depth 2 to 8, 50 epochs  [§1] [§4]
    # Batch size 50 in both parts  [§1]
    network = build_mlp(depth, activation, init)
    # Optimiser, loss and the learning-rate schedule are not stated
    train(network, learning_rate=learning_rate, batch_size=50)
    return validation_error(network), validation_accuracy(network)

Key equations. The paper's own equations are the eight in the activation section eq. 1–eq. 8; they are reproduced in the pseudocode above.

Hyperparameters

Results

Author-reported. Setup: MNIST, validation split, 2 hidden layers of 100 units for the activation sweep §3 and ELU or SELU 2–8 hidden-layer networks for the depth study §4. The baseline is the sigmoid network and, for depth, the 2-layer network with the same initialiser. Hardware and training time are not stated. Test-set accuracy is not reported.

The full tables follow. Values are written as in the paper, with the printed e_-k read as e-k. Rows are learning rates (activation sweep) or depths.

Training error by learning rate (activation sweep) tab. 1:

Validation error by learning rate tab. 2:

Training accuracy by learning rate tab. 3:

Validation accuracy by learning rate tab. 4:

Validation error for ELU by depth and initialiser tab. 5:

Validation accuracy for ELU by depth and initialiser tab. 6:

SELU, Glorot uniform vs Gaussian (LR 0.05) tab. 7:

Figures are described, not reproduced: fig. 1 and fig. 2 show ELU error and accuracy against learning rate; fig. 3 and fig. 4 compare ELU and SELU over training; fig. 5 shows the four ELU initialisers at 5 layers; fig. 6 shows the SELU comparison at 7 layers. The text reads these figures for overfitting onset, which it places around epoch 20 §3.

Paper vs code

No official code. No link to a code repository appears in the paper, its arXiv metadata or the LaTeX source. The GitHub repository search for the title returned nothing, and a search for the author's surname returned nothing. The source tree contains only the LaTeX sources and figure PDFs.

Pitfalls

• Learning-rate trend contradicts the table. The text says the loss rises with the learning rate §3. Training error tab. 1 falls as the rate rises, by two or more orders of magnitude for every activation. Only validation error tab. 2 rises for most columns.
• Accuracy gain from LR 0.05 to 0.1 is overstated. The text says validation accuracy rises by 2–4% §3. Table 4 shows rises of 0.1 to 0.5 percentage points. The card uses the table.
• ELU vs SELU "2% better". The text says ELU is about 2% better in accuracy §3 fig. 3. No table gives this comparison, and the activation-sweep numbers in tab. 4 differ by under one point. The figures' learning rate and epoch count are not stated.
• Selection on the validation set. The choice of ELU and of learning rate, and the early-stopping point, are made on the validation split. The test set is never reported, so these numbers are optimistic.
• Selective reporting. The text says SELU sometimes beat ELU, but those runs are "not reported" §3. The ELU choice rests on runs the author chose to show.
• Gaussian vs uniform (SELU). The text says the Gaussian gives better accuracy §4. Table 7 shows the sign of the accuracy difference changes across depths, with gaps of 0.2 points or less. Error is lower for Gaussian at most depths, but the gap is small.
• Single values, no spread. The tables give one value per cell and no standard deviation or seed count. The text says results varied between runs but does not quantify it §3, so differences of 0.001 to 0.002 accuracy cannot be judged from the card.
• Initialiser formulas are missing. The fan_in and fan_out rules, and the Glorot uniform bounds, are named but not written out §1 §4. Implement them from the cited originals, and say which variant you used.
• Width and optimiser for depth runs. The depth-experiment width, optimiser, loss and learning-rate schedule are not stated. An implementer has to choose them, and the depth curve may depend on that choice.
• Abstract vs body. The abstract promises an analysis of how the number of incoming and outgoing connections affects the network [abstract]. The body compares four initialisers at fixed depth and gives no connection-count analysis beyond that.
• Typesetting. Numbers in the source are printed as e_-1 (the underscore stands for a minus sign). Table 4 is cited as "table 4" in the text and as "Table 4" elsewhere; both mean the same table.
• Equation numbers. Equations are numbered in source order eq. 1–eq. 8 because the article class numbers displayed equations. This has not been checked against the compiled PDF.

Lineage

• Builds on: ReLU, Leaky ReLU, ELU, SELU and Glorot initialisation, all cited in §1 and §2 without arXiv ids in the source.
• Compared against: sigmoid (baseline) tab. 1–tab. 4; ReLU and Leaky ReLU within the same sweep; Glorot uniform and fan-based initialisers in the depth study tab. 5–tab. 7.
• Superseded by: not stated.