Activation function

Overview

A unit in a neural network computes a weighted sum of its inputs and then passes it through an activation function. Without that nonlinearity, a stack of layers collapses to one linear map. The problem the choice addresses is how gradients flow back through many such layers. The paper notes that the sigmoid lets gradients vanish in deep stacks, that ReLU passes gradient 1 for positive inputs, and that a ReLU unit can "die", staying at zero for every input 1804.02763 §2. All variants share the same slot in the network; they differ in the shape of the curve, especially on the negative side.

The concept is distinct from learnable activation functions, where the nonlinearity itself is trained. The page here covers fixed, hand-chosen functions applied to each unit.

Variants

Five fixed functions are compared in the single card for this concept. Their formulas, with the paper's stated parameter values, are in the implementation sketch below 1804.02763 §2 eq. 1–eq. 8.

Sigmoid (baseline)

The classic squashing function, used as the reference in every comparison. The paper gives no formula for it 1804.02763 §2. It is the weakest of the five at learning rates 0.01 and 0.05 on validation accuracy, and it ties the others at 0.1 1804.02763 tab. 4. Card: Pedamonti 2018, activation comparison on MNIST.

ReLU

Zero for negative inputs and the identity for positive ones. Its derivative is 0 for x ≤ 0 and 1 for x > 0 1804.02763 §2 eq. 1 eq. 2. At learning rate 0.01 it reaches 0.977 validation accuracy, above sigmoid's 0.955 1804.02763 tab. 4. Card: Pedamonti 2018, activation comparison on MNIST.

Leaky ReLU

ReLU with a small slope on the negative side. The paper sets the slope to 0.01 1804.02763 §2 eq. 3 eq. 4. It has the highest validation accuracy at learning rate 0.2 (0.982) in the sweep, and it trails ReLU and ELU at 0.01 1804.02763 tab. 4. Card: Pedamonti 2018, activation comparison on MNIST.

ELU

Exponential on the negative side, scaled by alpha = 1, and the identity for positive inputs 1804.02763 §2 eq. 5. The paper's derivative on the negative side is elu(x) + alpha, which stays nonzero there eq. 6. ELU is the best validation accuracy at learning rate 0.01 (0.978) and ties for the best at 0.1 (0.980) 1804.02763 tab. 4. The author keeps ELU for the depth study 1804.02763 §3. Card: Pedamonti 2018, activation comparison on MNIST.

SELU

A scaled ELU with alpha = 1.6733 and lambda = 1.0507, both as printed in the paper 1804.02763 §2 eq. 7 eq. 8. Its validation accuracy is within about one point of the others in the sweep 1804.02763 tab. 4, and it is the lowest of the ReLU family at learning rate 0.01 (0.974). The paper gives no argument for why these particular constants were chosen. Card: Pedamonti 2018, activation comparison on MNIST.

When to use which

The evidence is one dataset (MNIST), one author and one train/validation split, with no test-set numbers 1804.02763 §1 tab. 4. The table below states what the paper's numbers support for that setting.

Implementation sketch

The four activations written as the paper defines them. Parameter values are the paper's; the sigmoid is omitted because the paper gives no formula.

# ReLU: zero below zero, identity above  [§2] [eq. 1]
def relu(x):
    return max(0.0, x)

# Leaky ReLU with a small negative slope alpha = 0.01  [§2] [eq. 3]
def leaky_relu(x, alpha=0.01):
    if x > 0:
        return x
    return alpha * x

# ELU with alpha = 1; negative side is exponential  [§2] [eq. 5]
def elu(x, alpha=1.0):
    if x > 0:
        return x
    return alpha * (exp(x) - 1.0)

# SELU: lambda times ELU, with the printed constants  [§2] [eq. 7]
def selu(x, alpha=1.6733, lam=1.0507):
    return lam * elu(x, alpha)

The paper's starting-weight rules are named but not written out, so an implementation has to pick the exact bounds itself 1804.02763 §1 §4.

Contradictions

• Learning rate and loss. The text says the loss rises with the learning rate 1804.02763 §3. Training error in tab. 1 falls as the rate rises, by two or more orders of magnitude for every activation. Only the validation error in tab. 2 rises for most columns.
• ELU vs SELU accuracy. The text says ELU is about 2% more accurate 1804.02763 §3 fig. 3. No table gives that gap. The activation sweep in tab. 4 differs by under one point.
• Gaussian vs uniform initialiser for SELU. The text says the Gaussian gives better accuracy 1804.02763 §4. In tab. 7 the sign of the accuracy difference changes with depth, and the gaps are 0.2 points or less.

Papers

• Pedamonti 2018, activation comparison on MNIST: compares sigmoid, ReLU, Leaky ReLU, ELU and SELU as MNIST MLPs, picks ELU, and compares four starting-weight rules across 2 to 8 layers.