---
kind: concept
slug: surrogate-gradient
title: Surrogate gradient
summary: >-
  Trains spiking neural networks with backpropagation by replacing the zero or infinite derivative of the spike threshold with a smooth stand-in on the backward pass only. The forward pass keeps the hard spike. The trade-offs are memory for the stored time steps and no update for neurons that never fire.
tags: [spiking-neural-networks, training, backpropagation]
papers: ["1910.07407", "2102.04159", "2109.12894", "2209.13929", "2509.24852"]
related: [backpropagation-through-time, spiking-neural-network, quantization-aware-training, learnable-delays, residual-learning, attention-mechanism]
status: draft
updated_at: 2026-11-01
---

## Overview

A spiking neuron emits a binary spike when its membrane potential crosses a
threshold. The spike is a step function, so its derivative is zero almost
everywhere and unbounded at the threshold. Backpropagation then gives no
useful signal to neurons that have not fired [arxiv:2109.12894 §4.3]
[arxiv:2109.12894 fig. 9].

The shared fix is to keep the hard threshold in the forward pass and replace
its derivative with a smooth function σ̃′ in the backward pass
[arxiv:2109.12894 eq. 8] [arxiv:2109.12894 eq. 9]. The network is unrolled
over time steps and trained with backpropagation through time (BPTT)
[arxiv:2109.12894 §4.3]. The paper calls this the central recipe for
training SNNs with gradients. The reset term of the neuron is excluded from
the backward pass [arxiv:2109.12894 §4.3.3]. Card:
[Training Spiking Neural Networks Using Lessons From Deep Learning](/papers/2109.12894).

The Heidelberg spiking datasets paper trains recurrent and feed-forward LIF
networks on spike-coded speech with the same recipe, using a fast-sigmoid
surrogate [arxiv:1910.07407 §2.4.4]. Card:
[The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](/papers/1910.07407).

DelRec applies the same recipe to recurrent spiking networks whose
connections have learned delays. Its threshold is non-differentiable, so it
is trained with a surrogate gradient [arxiv:2509.24852 §2.1]. Card:
[DelRec: learning delays in recurrent spiking neural networks](/papers/2509.24852).

The Attention SNN paper trains the spikes of its convolutional layers with a
surrogate gradient, but its main text does not give the surrogate's shape
[arxiv:2209.13929 §3.2]. Card:
[Attention Spiking Neural Networks](/papers/2209.13929).

Deep spiking networks add a second problem. Through a stack of residual
blocks, the input gradient is a product of surrogate derivatives, and the
authors of SEW ResNet argue this product vanishes for the common surrogates
[arxiv:2102.04159 §3.2] [arxiv:2102.04159 eq. 8]. Their fix changes how the
shortcut is combined with the block output, so that gradients pass through
identity blocks unchanged [arxiv:2102.04159 §3.3]. Card:
[Deep Residual Learning in Spiking Neural Networks](/papers/2102.04159).

## Variants

The tutorial card has no experiments of its own. The datasets card is a
dataset paper whose own SNNs are trained with a surrogate, next to SVM, LSTM
and CNN baselines.
The SEW ResNet and DelRec cards are method papers with benchmark results.
The Attention SNN card is a method paper that uses a surrogate without
stating its shape in the main text. Together they describe the following
variants of the gradient path.

### Smooth surrogate on the backward pass

The forward pass uses the Heaviside step of membrane minus threshold. The
backward pass uses a smooth curve. The paper describes a sigmoid form and an
arctan form, and gives the arctan derivative as the default choice
[arxiv:2109.12894 §4.3.1]. Its slope parameter is a hyperparameter; the
implementation the card reviewed uses α = 2 by default
[arxiv:2109.12894 §4.3.1] (see the card's Hyperparameters section).

The SEW ResNet paper uses the same arctan form with α = 2 in all its
experiments [arxiv:2102.04159 app. A.1]. Its CIFAR-10 ablation changes only
the surrogate shape. For the SEW ResNet, arctan reaches 0.8263, a rectangular
surrogate 0.8256, and a constant derivative of 1 only 0.1256. For the
Spiking ResNet baseline, the same three give 0.7733, 0.6601 and 0.1
[arxiv:2102.04159 tab. 9]. These are author-reported single runs on a small
network that the paper does not present as state of the art
[arxiv:2102.04159 app. A.5]. The repository does not set a surrogate, so the
α = 2 value rests on the paper alone [arxiv:2102.04159 app. A.1]. Card:
[Deep Residual Learning in Spiking Neural Networks](/papers/2102.04159).

### Fast-sigmoid surrogate with a steepness parameter

The datasets paper writes its surrogate as a fast sigmoid, u / (1 + β|u|),
with steepness β, and trains with backpropagation through time
[arxiv:1910.07407 §2.4.4]. The text sets β = 40 for all SNNs unless stated
otherwise, while the parameter table lists β = 100; the paper does not say
which value the reported SNN runs used [arxiv:1910.07407 §3.1]
[arxiv:1910.07407 tab. 2]. The paper also does not say whether the printed
expression is the surrogate itself or the function whose derivative is taken
[arxiv:1910.07407 §2.4.4]. Steepness was chosen on a validation set
[arxiv:1910.07407 §3.1]. The card records no comparison of surrogate shapes
in this paper. Its reported recurrent SNN reaches 71.4% on SHD, against 89.0%
for an LSTM on binned spikes; both are author-reported [arxiv:1910.07407
tab. 1]. Card: [The Heidelberg spiking datasets](/papers/1910.07407).

### Rectangular window surrogate

The main text of the Attention SNN paper does not state a surrogate shape
[arxiv:2209.13929 §3.2]. The released convolutional code instead uses a
rectangle around the threshold, of height 2 and with a width parameter of 0.25
as the card reads it. The card records this only from the code, not from the
paper [Attention Spiking Neural Networks](/papers/2209.13929), Paper vs code,
item 6. This is a rectangle-type surrogate, the shape that the SEW ResNet
ablation tests against arctan. Its numbers come from a different network, so
they do not settle the choice here [arxiv:2102.04159 tab. 9].

### Triangular surrogate with a per-task width

DelRec uses a triangle-shaped surrogate, with its width α set per benchmark:
1.0 for SSC and PS-MNIST, 0.4 for AL and 0.6 for HAR [arxiv:2509.24852 supp.
Hyperparameters]. Its Mackey-Glass runs use an arctan surrogate with α = 5
instead [arxiv:2509.24852 §4.3]. The card does not give the triangle's
formula, so take it from the paper before porting. The surrogate width α is a
different quantity from the kernel width σ, which DelRec anneals to spread
each delayed spike over neighbouring time steps [arxiv:2509.24852 §2.1]. Card:
[DelRec](/papers/2509.24852).

### Residual shortcut that bypasses the surrogate

The SEW block computes a residual branch A with a spiking neuron at its end,
then combines A with the block input S by a rule g [arxiv:2102.04159 eq. 9].
Three rules are compared: ADD, AND, and IAND, which is (not A) and S
[arxiv:2102.04159 tab. 1]. For ADD and IAND, setting the residual branch to
a constant zero gives identity mapping. Under identity mapping, the gradient
through k blocks is exactly 1, so no surrogate derivative multiplies along the
path [arxiv:2102.04159 §3.3] [arxiv:2102.04159 eq. 11]. AND needs the residual
branch held at a constant one, and the paper reaches it by zeroing a
BatchNorm weight and setting its bias to a large constant [arxiv:2102.04159
§3.3]. The gradient result holds only while the branch is constant; the
paper gives no bound for ordinary training [arxiv:2102.04159 §3.3].

### Straight-through view

The paper relates the surrogate to the straight-through estimator used for
binary and quantised networks. It reports that SNNs tolerate
quantisation-aware training, with example bit widths of 8 for weights and 4
for membrane states [arxiv:2109.12894 §4.3.2].

### Spike-time backpropagation

SpikeProp differentiates the spike time instead of the spike. The paper says
this gives an unbiased gradient in multi-layer networks, but a neuron must
fire for the gradient to exist, so inactive neurons freeze
[arxiv:2109.12894 §4.2].

### Online variants

BPTT needs the stored history of every time step. Real-time recurrent
learning tracks the influence of each weight forward in time instead, and
the paper surveys RTRL-derived SNN rules named e-prop, decolle, OSTL, ETLP,
OSTTP and FPTT [arxiv:2109.12894 §5.2] [arxiv:2109.12894 §5.2.1]. The paper
reports no numbers comparing them.

### Shadow training

Train a non-spiking network and convert its activations to rates or
latencies. This avoids the zero derivative, but the conversion is an
approximation and usually needs many time steps [arxiv:2109.12894 §4.1].

## When to use which

Only the SEW ResNet card compares surrogate shapes, and those come from one
architecture. The other rows rest on constraints and on the papers' own
qualitative statements or single reported settings.

| Situation | Use | Why |
|---|---|---|
| Fixed, short time horizon and memory for the unrolled history | Surrogate BPTT | Standard autodiff over stored steps [arxiv:2109.12894 §4.3] |
| Directly trained SNN with many residual blocks | Surrogate BPTT with an ADD or IAND shortcut | Author-reported ImageNet at T = 4: SEW ResNet-152 reaches 69.26% top-1, against 10.03% for a Spiking ResNet-152 with the ordinary shortcut [arxiv:2102.04159 tab. 2] [arxiv:2102.04159 §3.3] |
| Residual SNN with the AND shortcut | Only with zero initialisation and the large bias | Without zero initialisation the AND variant goes silent in deep layers [arxiv:2102.04159 §4.1] [arxiv:2102.04159 fig. 5]; on DVS Gesture it reaches 70.49% against 97.92% for ADD [arxiv:2102.04159 tab. 4] |
| Surrogate shape to pick for a deep residual SNN | Arctan with α = 2 | A constant-1 derivative fails on both networks in the ablation, and the rectangular one hurts the Spiking ResNet most [arxiv:2102.04159 tab. 9] |
| Recurrent SNN on spike-coded speech, where a published fast-sigmoid setup is the reference | Fast-sigmoid surrogate with steepness tuned on a validation set | Steepness is 40 in the text and 100 in the table, so check it against the code [arxiv:1910.07407 §3.1] [arxiv:1910.07407 tab. 2]; author-reported 71.4% on SHD for a recurrent SNN [arxiv:1910.07407 tab. 1] |
| Memory too small to store the history, or learning must run online | Online RTRL-family rules | The stored history is what BPTT needs; the paper gives only scaling statements [arxiv:2109.12894 §5.2] |
| Static images, time dynamics not needed | Shadow training | The paper says it leads on CIFAR-10 and ImageNet, but notes those benchmarks rarely use temporal dynamics [arxiv:2109.12894 §4.1] |
| Many neurons stay silent | Threshold adjustment or activity regularisation, with the surrogate | A surrogate gives no weight update for a neuron that never spikes [arxiv:2109.12894 §4.3.1] [arxiv:2109.12894 §3.5] |
| Weights or states must be low precision | Surrogate with straight-through style quantisation | Example bit widths and the STE link [arxiv:2109.12894 §4.3.2] |
| Input is an event stream from a sensor | Delta-modulation input encoding, then surrogate BPTT | Spikes only on change, as event cameras do [arxiv:2109.12894 §3.1] |
| Convolutional SNN with attention on the membrane potential, event-camera input | Arctan with α = 2 by default; the rectangle only to reproduce the released code | Neither paper ablates the shape in this setting. The arctan evidence is from the SEW ablation on another network [arxiv:2102.04159 tab. 9]; the attention paper gives no shape in its text [arxiv:2209.13929 §3.2] |
| Recurrent SNN on temporal tasks, where the synaptic or axonal delay should be learned | Surrogate BPTT with learned delays, using a triangle surrogate with a tuned width | Author-reported: learned delays beat fixed ones in six of eight dataset-type comparisons at p < 0.05, and synaptic delays give no significant gain on SSC or AL [arxiv:2509.24852 tab. 3] [arxiv:2509.24852 supp. Statistical significance] |
| Target hardware needs integer delays only | Round delays after training | Rounding has little effect on SSC, PS-MNIST and HAR, but a large one on AL, which keeps fractional delays [arxiv:2509.24852 §4.4] |

## Implementation sketch

The forward pass keeps the hard threshold. Only the derivative changes.

```pseudocode
# Surrogate derivative, used only on the backward pass  [arxiv:2109.12894 §4.3.1]
# Arctan form; alpha sets the slope (default 2 in the reviewed implementation)
def surrogate_derivative(membrane, theta, alpha=2.0):
    # Position relative to the threshold, as in the forward Heaviside
    shifted = membrane - theta
    # Peaks at its maximum when shifted = 0; decays on both sides
    return (alpha / 2) / (1 + (pi / 2 * alpha * shifted) ** 2)
```

The residual variant changes only how the block output is formed from its
input. Here A is the spiking residual branch and S the block input.

```pseudocode
# Residual combine for a SEW block, output over one time step  [arxiv:2102.04159 eq. 9]
# ADD is not binary; AND and IAND keep the output binary for binary inputs
def sew_combine(residual_spikes, block_input, rule="IAND"):
    # Element-wise rule g applied to A and S  [arxiv:2102.04159 tab. 1]
    if rule == "ADD":
        return residual_spikes + block_input
    if rule == "AND":
        return residual_spikes * block_input
    # IAND: 1 where the branch is silent and the input spikes  [arxiv:2102.04159 tab. 1]
    return (1 - residual_spikes) * block_input
```

```pseudocode
# Surrogate-gradient BPTT over T steps  [arxiv:2109.12894 §4.3] [arxiv:2109.12894 eq. 6]
# Forward uses the hard spike; backward uses the surrogate above
def train_step(inputs, targets, weights, T):
    # Run the neuron dynamics forward and keep every step  [arxiv:2109.12894 §4.3]
    spikes_per_step, membranes_per_step = run_lif_forward(inputs, weights, T)
    # Sum the instantaneous losses over time  [arxiv:2109.12894 eq. 6]
    loss = output_loss(spikes_per_step, membranes_per_step, targets)
    # Backpropagate; the Heaviside derivative is swapped for the surrogate
    # and the reset term is detached  [arxiv:2109.12894 §4.3.1] [arxiv:2109.12894 §4.3.3]
    return backprop_through_time(loss, surrogate_fn=surrogate_derivative)
```

Implementers choose the output loss from [arxiv:2109.12894 §3.3]. The paper
notes that the best loss depends on the network and task, and that it is not
settled.

## Contradictions

- **Peak and scale of the arctan derivative.** The paper's practical note
  prints the arctan derivative with a factor of 1/π, while its own integral
  of the arctan has a derivative without that factor [arxiv:2109.12894
  §4.3.1]. The reviewed implementation follows the integral form, so its
  peak is 1 rather than 1/π. Porting the printed formula gives surrogate
  gradients about π times smaller. See the
  [card's Paper vs code section](/papers/2109.12894) for detail.
- **Steepness of the fast sigmoid.** The datasets paper's text uses β = 40 for
  all SNNs unless stated otherwise [arxiv:1910.07407 §3.1]. Its parameter
  table lists β = 100 [arxiv:1910.07407 tab. 2]. The paper does not say which
  value produced its reported SNN results.
- **Cross-entropy on spike counts.** The paper's table and listing describe
  cross-entropy on spike counts used as logits [arxiv:2109.12894 §3.3]. The
  reviewed implementation of the rate-code cross-entropy applies a log-softmax
  at each step instead. The card explains the difference.

## Papers

- [Training Spiking Neural Networks Using Lessons From Deep Learning](/papers/2109.12894): tutorial and perspective that formalises surrogate-gradient BPTT for LIF neurons, lists encodings, losses and learning rules, and links the method to STDP.
- [Deep Residual Learning in Spiking Neural Networks](/papers/2102.04159): SEW residual block that combines the spiking branch with the input by ADD, AND or IAND, trained with an arctan surrogate (α = 2); author-reported ImageNet and neuromorphic results.
- [DelRec: learning delays in recurrent spiking neural networks](/papers/2509.24852): trains learned delays and weights of recurrent LIF networks with a triangle or arctan surrogate; author-reported accuracy on four temporal benchmarks.
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](/papers/1910.07407): two spike-coded speech datasets, and feed-forward and recurrent SNNs trained with a fast-sigmoid surrogate in backpropagation through time; author-reported accuracies.
- [Attention Spiking Neural Networks](/papers/2209.13929): attention on the membrane potential of convolutional SNNs, trained with a surrogate whose shape the main text does not state; author-reported DVS128 and ImageNet results.