---
kind: concept
slug: activation-function
title: Activation function
summary: >-
  The fixed nonlinearity applied to each unit of a neural network after its weighted sum. Sigmoid, ReLU, Leaky ReLU, ELU and SELU differ in gradient flow and in the negative side. The one card here tests them on MNIST MLPs, finds the ReLU family ahead of sigmoid at low learning rates, and picks ELU. Single split, author-reported.
tags: [neural-networks, nonlinearity, deep-learning]
papers: ["1804.02763"]
related: [multi-layer-perceptron, learnable-activation-functions]
status: draft
updated_at: 2026-11-08
---

## Overview

A unit in a neural network computes a weighted sum of its inputs and then
passes it through an activation function. Without that nonlinearity, a stack
of layers collapses to one linear map. The problem the choice addresses is
how gradients flow back through many such layers. The paper notes that the
sigmoid lets gradients vanish in deep stacks, that ReLU passes gradient 1 for
positive inputs, and that a ReLU unit can "die", staying at zero for every
input [arxiv:1804.02763 §2]. All variants share the same slot in the network;
they differ in the shape of the curve, especially on the negative side.

The concept is distinct from [learnable activation functions](/concepts/learnable-activation-functions),
where the nonlinearity itself is trained. The page here covers fixed,
hand-chosen functions applied to each unit.

## Variants

Five fixed functions are compared in the single card for this concept. Their
formulas, with the paper's stated parameter values, are in the implementation
sketch below [arxiv:1804.02763 §2] [eq. 1]–[eq. 8].

### Sigmoid (baseline)

The classic squashing function, used as the reference in every comparison.
The paper gives no formula for it [arxiv:1804.02763 §2]. It is the weakest
of the five at learning rates 0.01 and 0.05 on validation accuracy, and it
ties the others at 0.1 [arxiv:1804.02763 tab. 4]. Card: [Pedamonti 2018, activation comparison on MNIST](/papers/1804.02763).

### ReLU

Zero for negative inputs and the identity for positive ones. Its derivative
is 0 for x ≤ 0 and 1 for x > 0 [arxiv:1804.02763 §2] [eq. 1] [eq. 2]. At
learning rate 0.01 it reaches 0.977 validation accuracy, above sigmoid's 0.955
[arxiv:1804.02763 tab. 4]. Card: [Pedamonti 2018, activation comparison on MNIST](/papers/1804.02763).

### Leaky ReLU

ReLU with a small slope on the negative side. The paper sets the slope to
0.01 [arxiv:1804.02763 §2] [eq. 3] [eq. 4]. It has the highest validation
accuracy at learning rate 0.2 (0.982) in the sweep, and it trails ReLU and
ELU at 0.01 [arxiv:1804.02763 tab. 4]. Card: [Pedamonti 2018, activation comparison on MNIST](/papers/1804.02763).

### ELU

Exponential on the negative side, scaled by alpha = 1, and the identity for
positive inputs [arxiv:1804.02763 §2] [eq. 5]. The paper's derivative on the
negative side is elu(x) + alpha, which stays nonzero there [eq. 6]. ELU is
the best validation accuracy at learning rate 0.01 (0.978) and ties for the
best at 0.1 (0.980) [arxiv:1804.02763 tab. 4]. The author keeps ELU for the
depth study [arxiv:1804.02763 §3]. Card: [Pedamonti 2018, activation comparison on MNIST](/papers/1804.02763).

### SELU

A scaled ELU with alpha = 1.6733 and lambda = 1.0507, both as printed in the
paper [arxiv:1804.02763 §2] [eq. 7] [eq. 8]. Its validation accuracy is
within about one point of the others in the sweep [arxiv:1804.02763 tab. 4],
and it is the lowest of the ReLU family at learning rate 0.01 (0.974). The
paper gives no argument for why these particular constants were chosen.
Card: [Pedamonti 2018, activation comparison on MNIST](/papers/1804.02763).

## When to use which

The evidence is one dataset (MNIST), one author and one train/validation
split, with no test-set numbers [arxiv:1804.02763 §1] [tab. 4]. The table
below states what the paper's numbers support for that setting.

| Situation | Use | Why |
|---|---|---|
| Plain MLP, small classification task, learning rate about 0.1 | ReLU or ELU; sigmoid is also fine | At 0.1 the five validation accuracies lie between 0.978 and 0.980 [arxiv:1804.02763 tab. 4]. The gap is within the noise the card describes. |
| Low learning rate (0.01) | ELU, then ReLU | ELU 0.978 and ReLU 0.977 against sigmoid 0.955 [arxiv:1804.02763 tab. 4]. The sigmoid gap is the largest in the paper. |
| Deeper ELU stack (5 to 8 hidden layers) | Two layers may be enough | Validation accuracy stays between 0.978 and 0.983 across 2 to 8 layers and all four initialisers. For Glorot uniform, the lowest validation error is at 2 layers (0.100) [arxiv:1804.02763 tab. 5] [arxiv:1804.02763 tab. 6]. |
| Choosing between ELU and SELU | No evidence here for SELU | The author picks ELU because it did better in most runs, and the SELU runs that did better are not shown [arxiv:1804.02763 §3]. |
| Choosing the starting-weight rule for a deep ELU net | Not settled by this paper | The best depth differs by rule (7 layers for Glorot uniform, 5 for fan_in), and at any one depth the four rules differ by at most 0.2 points [arxiv:1804.02763 tab. 6]. |
| Compute budget, memory, training time | Not stated | The paper reports no timing or hardware [arxiv:1804.02763 §1]. |

## Implementation sketch

The four activations written as the paper defines them. Parameter values
are the paper's; the sigmoid is omitted because the paper gives no formula.

```pseudocode
# ReLU: zero below zero, identity above  [§2] [eq. 1]
def relu(x):
    return max(0.0, x)

# Leaky ReLU with a small negative slope alpha = 0.01  [§2] [eq. 3]
def leaky_relu(x, alpha=0.01):
    if x > 0:
        return x
    return alpha * x

# ELU with alpha = 1; negative side is exponential  [§2] [eq. 5]
def elu(x, alpha=1.0):
    if x > 0:
        return x
    return alpha * (exp(x) - 1.0)

# SELU: lambda times ELU, with the printed constants  [§2] [eq. 7]
def selu(x, alpha=1.6733, lam=1.0507):
    return lam * elu(x, alpha)
```

The paper's starting-weight rules are named but not written out, so an
implementation has to pick the exact bounds itself [arxiv:1804.02763 §1] [§4].

## Contradictions

- **Learning rate and loss.** The text says the loss rises with the learning
  rate [arxiv:1804.02763 §3]. Training error in [tab. 1] falls as the rate
  rises, by two or more orders of magnitude for every activation. Only the
  validation error in [tab. 2] rises for most columns.
- **ELU vs SELU accuracy.** The text says ELU is about 2% more accurate
  [arxiv:1804.02763 §3] [fig. 3]. No table gives that gap. The activation
  sweep in [tab. 4] differs by under one point.
- **Gaussian vs uniform initialiser for SELU.** The text says the Gaussian
  gives better accuracy [arxiv:1804.02763 §4]. In [tab. 7] the sign of the
  accuracy difference changes with depth, and the gaps are 0.2 points or less.

## Papers

- [Pedamonti 2018, activation comparison on MNIST](/papers/1804.02763): compares sigmoid, ReLU, Leaky ReLU, ELU and SELU as MNIST MLPs, picks ELU, and compares four starting-weight rules across 2 to 8 layers.