Surrogate gradient

Overview

A spiking neuron emits a binary spike when its membrane potential crosses a threshold. The spike is a step function, so its derivative is zero almost everywhere and unbounded at the threshold. Backpropagation then gives no useful signal to neurons that have not fired 2109.12894 §4.3 2109.12894 fig. 9.

The shared fix is to keep the hard threshold in the forward pass and replace its derivative with a smooth function σ̃′ in the backward pass 2109.12894 eq. 8 2109.12894 eq. 9. The network is unrolled over time steps and trained with backpropagation through time (BPTT) 2109.12894 §4.3. The paper calls this the central recipe for training SNNs with gradients. The reset term of the neuron is excluded from the backward pass 2109.12894 §4.3.3. Card: Training Spiking Neural Networks Using Lessons From Deep Learning.

The Heidelberg spiking datasets paper trains recurrent and feed-forward LIF networks on spike-coded speech with the same recipe, using a fast-sigmoid surrogate 1910.07407 §2.4.4. Card: The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks.

DelRec applies the same recipe to recurrent spiking networks whose connections have learned delays. Its threshold is non-differentiable, so it is trained with a surrogate gradient 2509.24852 §2.1. Card: DelRec: learning delays in recurrent spiking neural networks.

The Attention SNN paper trains the spikes of its convolutional layers with a surrogate gradient, but its main text does not give the surrogate's shape 2209.13929 §3.2. Card: Attention Spiking Neural Networks.

Deep spiking networks add a second problem. Through a stack of residual blocks, the input gradient is a product of surrogate derivatives, and the authors of SEW ResNet argue this product vanishes for the common surrogates 2102.04159 §3.2 2102.04159 eq. 8. Their fix changes how the shortcut is combined with the block output, so that gradients pass through identity blocks unchanged 2102.04159 §3.3. Card: Deep Residual Learning in Spiking Neural Networks.

Variants

The tutorial card has no experiments of its own. The datasets card is a dataset paper whose own SNNs are trained with a surrogate, next to SVM, LSTM and CNN baselines. The SEW ResNet and DelRec cards are method papers with benchmark results. The Attention SNN card is a method paper that uses a surrogate without stating its shape in the main text. Together they describe the following variants of the gradient path.

Smooth surrogate on the backward pass

The forward pass uses the Heaviside step of membrane minus threshold. The backward pass uses a smooth curve. The paper describes a sigmoid form and an arctan form, and gives the arctan derivative as the default choice 2109.12894 §4.3.1. Its slope parameter is a hyperparameter; the implementation the card reviewed uses α = 2 by default 2109.12894 §4.3.1 (see the card's Hyperparameters section).

The SEW ResNet paper uses the same arctan form with α = 2 in all its experiments 2102.04159 app. A.1. Its CIFAR-10 ablation changes only the surrogate shape. For the SEW ResNet, arctan reaches 0.8263, a rectangular surrogate 0.8256, and a constant derivative of 1 only 0.1256. For the Spiking ResNet baseline, the same three give 0.7733, 0.6601 and 0.1 2102.04159 tab. 9. These are author-reported single runs on a small network that the paper does not present as state of the art 2102.04159 app. A.5. The repository does not set a surrogate, so the α = 2 value rests on the paper alone 2102.04159 app. A.1. Card: Deep Residual Learning in Spiking Neural Networks.

Fast-sigmoid surrogate with a steepness parameter

The datasets paper writes its surrogate as a fast sigmoid, u / (1 + β|u|), with steepness β, and trains with backpropagation through time 1910.07407 §2.4.4. The text sets β = 40 for all SNNs unless stated otherwise, while the parameter table lists β = 100; the paper does not say which value the reported SNN runs used 1910.07407 §3.1 1910.07407 tab. 2. The paper also does not say whether the printed expression is the surrogate itself or the function whose derivative is taken 1910.07407 §2.4.4. Steepness was chosen on a validation set 1910.07407 §3.1. The card records no comparison of surrogate shapes in this paper. Its reported recurrent SNN reaches 71.4% on SHD, against 89.0% for an LSTM on binned spikes; both are author-reported [arxiv:1910.07407 tab. 1]. Card: The Heidelberg spiking datasets.

Rectangular window surrogate

The main text of the Attention SNN paper does not state a surrogate shape 2209.13929 §3.2. The released convolutional code instead uses a rectangle around the threshold, of height 2 and with a width parameter of 0.25 as the card reads it. The card records this only from the code, not from the paper Attention Spiking Neural Networks, Paper vs code, item 6. This is a rectangle-type surrogate, the shape that the SEW ResNet ablation tests against arctan. Its numbers come from a different network, so they do not settle the choice here 2102.04159 tab. 9.

Triangular surrogate with a per-task width

DelRec uses a triangle-shaped surrogate, with its width α set per benchmark: 1.0 for SSC and PS-MNIST, 0.4 for AL and 0.6 for HAR [arxiv:2509.24852 supp. Hyperparameters]. Its Mackey-Glass runs use an arctan surrogate with α = 5 instead 2509.24852 §4.3. The card does not give the triangle's formula, so take it from the paper before porting. The surrogate width α is a different quantity from the kernel width σ, which DelRec anneals to spread each delayed spike over neighbouring time steps 2509.24852 §2.1. Card: DelRec.

Residual shortcut that bypasses the surrogate

The SEW block computes a residual branch A with a spiking neuron at its end, then combines A with the block input S by a rule g 2102.04159 eq. 9. Three rules are compared: ADD, AND, and IAND, which is (not A) and S 2102.04159 tab. 1. For ADD and IAND, setting the residual branch to a constant zero gives identity mapping. Under identity mapping, the gradient through k blocks is exactly 1, so no surrogate derivative multiplies along the path 2102.04159 §3.3 2102.04159 eq. 11. AND needs the residual branch held at a constant one, and the paper reaches it by zeroing a BatchNorm weight and setting its bias to a large constant [arxiv:2102.04159 §3.3]. The gradient result holds only while the branch is constant; the paper gives no bound for ordinary training 2102.04159 §3.3.

Straight-through view

The paper relates the surrogate to the straight-through estimator used for binary and quantised networks. It reports that SNNs tolerate quantisation-aware training, with example bit widths of 8 for weights and 4 for membrane states 2109.12894 §4.3.2.

Spike-time backpropagation

SpikeProp differentiates the spike time instead of the spike. The paper says this gives an unbiased gradient in multi-layer networks, but a neuron must fire for the gradient to exist, so inactive neurons freeze 2109.12894 §4.2.

Online variants

BPTT needs the stored history of every time step. Real-time recurrent learning tracks the influence of each weight forward in time instead, and the paper surveys RTRL-derived SNN rules named e-prop, decolle, OSTL, ETLP, OSTTP and FPTT 2109.12894 §5.2 2109.12894 §5.2.1. The paper reports no numbers comparing them.

Shadow training

Train a non-spiking network and convert its activations to rates or latencies. This avoids the zero derivative, but the conversion is an approximation and usually needs many time steps 2109.12894 §4.1.

When to use which

Only the SEW ResNet card compares surrogate shapes, and those come from one architecture. The other rows rest on constraints and on the papers' own qualitative statements or single reported settings.

Implementation sketch

The forward pass keeps the hard threshold. Only the derivative changes.

# Surrogate derivative, used only on the backward pass  [arxiv:2109.12894 §4.3.1]
# Arctan form; alpha sets the slope (default 2 in the reviewed implementation)
def surrogate_derivative(membrane, theta, alpha=2.0):
    # Position relative to the threshold, as in the forward Heaviside
    shifted = membrane - theta
    # Peaks at its maximum when shifted = 0; decays on both sides
    return (alpha / 2) / (1 + (pi / 2 * alpha * shifted) ** 2)

The residual variant changes only how the block output is formed from its input. Here A is the spiking residual branch and S the block input.

# Residual combine for a SEW block, output over one time step  [arxiv:2102.04159 eq. 9]
# ADD is not binary; AND and IAND keep the output binary for binary inputs
def sew_combine(residual_spikes, block_input, rule="IAND"):
    # Element-wise rule g applied to A and S  [arxiv:2102.04159 tab. 1]
    if rule == "ADD":
        return residual_spikes + block_input
    if rule == "AND":
        return residual_spikes * block_input
    # IAND: 1 where the branch is silent and the input spikes  [arxiv:2102.04159 tab. 1]
    return (1 - residual_spikes) * block_input

# Surrogate-gradient BPTT over T steps  [arxiv:2109.12894 §4.3] [arxiv:2109.12894 eq. 6]
# Forward uses the hard spike; backward uses the surrogate above
def train_step(inputs, targets, weights, T):
    # Run the neuron dynamics forward and keep every step  [arxiv:2109.12894 §4.3]
    spikes_per_step, membranes_per_step = run_lif_forward(inputs, weights, T)
    # Sum the instantaneous losses over time  [arxiv:2109.12894 eq. 6]
    loss = output_loss(spikes_per_step, membranes_per_step, targets)
    # Backpropagate; the Heaviside derivative is swapped for the surrogate
    # and the reset term is detached  [arxiv:2109.12894 §4.3.1] [arxiv:2109.12894 §4.3.3]
    return backprop_through_time(loss, surrogate_fn=surrogate_derivative)

Implementers choose the output loss from 2109.12894 §3.3. The paper notes that the best loss depends on the network and task, and that it is not settled.

Contradictions

• Peak and scale of the arctan derivative. The paper's practical note prints the arctan derivative with a factor of 1/π, while its own integral of the arctan has a derivative without that factor [arxiv:2109.12894 §4.3.1]. The reviewed implementation follows the integral form, so its peak is 1 rather than 1/π. Porting the printed formula gives surrogate gradients about π times smaller. See the card's Paper vs code section for detail.
• Steepness of the fast sigmoid. The datasets paper's text uses β = 40 for all SNNs unless stated otherwise 1910.07407 §3.1. Its parameter table lists β = 100 1910.07407 tab. 2. The paper does not say which value produced its reported SNN results.
• Cross-entropy on spike counts. The paper's table and listing describe cross-entropy on spike counts used as logits 2109.12894 §3.3. The reviewed implementation of the rate-code cross-entropy applies a log-softmax at each step instead. The card explains the difference.

Papers

• Training Spiking Neural Networks Using Lessons From Deep Learning: tutorial and perspective that formalises surrogate-gradient BPTT for LIF neurons, lists encodings, losses and learning rules, and links the method to STDP.
• Deep Residual Learning in Spiking Neural Networks: SEW residual block that combines the spiking branch with the input by ADD, AND or IAND, trained with an arctan surrogate (α = 2); author-reported ImageNet and neuromorphic results.
• DelRec: learning delays in recurrent spiking neural networks: trains learned delays and weights of recurrent LIF networks with a triangle or arctan surrogate; author-reported accuracy on four temporal benchmarks.
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks: two spike-coded speech datasets, and feed-forward and recurrent SNNs trained with a fast-sigmoid surrogate in backpropagation through time; author-reported accuracies.
• Attention Spiking Neural Networks: attention on the membrane potential of convolutional SNNs, trained with a surrogate whose shape the main text does not state; author-reported DVS128 and ImageNet results.