---
kind: concept
slug: spiking-neural-network
title: Spiking neural networks
summary: >-
  Spiking neural networks send sparse on-off spikes through leaky neurons. Training is hard because a spike has no useful gradient. Options: surrogate-gradient BPTT, with optional residual blocks for depth and optional attention on the membrane potential; ANN-to-SNN conversion; spike-time gradients; online learning; learned delays. Speech benchmarks: SHD and SSC. Results are author-reported.
tags: [spiking-neural-networks, neuromorphic, surrogate-gradients, backpropagation-through-time, online-learning, delays, residual-networks, attention, speech, benchmarks]
papers: ["1910.07407", "2102.04159", "2109.12894", "2209.13929", "2509.24852"]
related: [backpropagation-through-time, recurrent-neural-network, quantization-aware-training, learnable-delays, residual-learning, surrogate-gradient, attention-mechanism, channel-attention, keyword-spotting, benchmark-validity]
status: draft
updated_at: 2026-11-01
---

## Overview

A spiking neural network (SNN) passes information as discrete spikes in time
rather than as real-valued activations. Each neuron integrates its input into a
membrane potential, leaks part of that potential at every step, and emits a
spike when the potential crosses a threshold [arxiv:2109.12894 §2.1]. Because
most neurons are silent at a given step, activity is sparse, which is the basis
for the efficiency argument [arxiv:2109.12894 §1] [arxiv:2109.12894 §2].

The shared difficulty is training. A spike is a step function of the membrane
potential. Its derivative is zero almost everywhere and unbounded at the
threshold, so standard gradients give no useful signal to neurons that have
not fired [arxiv:2109.12894 §4.3]. Most training methods below are a different
answer to this problem, or a way to avoid it. Learned delays keep the surrogate
gradient and add a trainable parameter [arxiv:2509.24852 §2.1]. Residual wiring
addresses a second problem, depth, and keeps the surrogate gradient
[arxiv:2102.04159 §3.3].

The sources are a tutorial and perspective, three method papers, and a dataset
paper. The tutorial reports no benchmark results of its own
[arxiv:2109.12894 §1]. The method papers and the dataset paper report benchmark
accuracy for their own models, which is author-reported
[arxiv:2509.24852 tab. 1] [arxiv:2102.04159 tab. 2] [arxiv:2209.13929 tab. 1]
[arxiv:1910.07407 tab. 1].
Figures in this page are the papers' statements, not measurements from our side.

## Variants

### Leaky integrate-and-fire neuron

The usual neuron model. The membrane decays by a factor β each step, takes the
weighted input, and is reduced by the threshold after a spike. Its discrete
update is derived from an RC circuit with forward Euler [arxiv:2109.12894 §2.1]
[arxiv:2109.12894 app. A]. The paper recommends this neuron as the default for
gradient-based training, with richer neuron models when more capacity is needed
[arxiv:2109.12894 §2.2]. Card: [Training Spiking Neural Networks Using Lessons From Deep Learning](/papers/2109.12894).

### Input and output codes

Rate coding turns an intensity into a spike probability per step. Latency coding
turns it into the time of one spike. Delta modulation emits a spike only when the
input changes, as an event camera does [arxiv:2109.12894 §3.1] [arxiv:2109.12894 app. B].
On the output side, the class can be read from the spike count, from the
first neuron to fire, or from a population of neurons per class
[arxiv:2109.12894 §3.2]. The paper notes that encoding ordinary static data
usually discards information, and that rate codes lose less accuracy than
latency codes [arxiv:2109.12894 §3.1].

### Surrogate-gradient backpropagation through time

The most widely used route. The network is unrolled over the time steps, and the
loss is summed over time [arxiv:2109.12894 §4.3] [arxiv:2109.12894 eq. 6]. In
the forward pass the hard threshold stays. In the backward pass it is replaced
by a smooth function whose derivative supplies the gradient
[arxiv:2109.12894 §4.3.1] [arxiv:2109.12894 eq. 8]. The paper names an arctan
curve as its starting point for this surrogate [arxiv:2109.12894 §4.3.1]. The
reset term is excluded from the backward pass [arxiv:2109.12894 §4.3.3].
The paper also relates the resulting update to the straight-through estimator
used for quantised networks [arxiv:2109.12894 §4.3.2], which links this variant
to [Quantization-aware training](/concepts/quantization-aware-training).

The shape of the surrogate matters in practice. On CIFAR-10, the SEW ResNet
paper reports accuracy 0.8263 with arctan, 0.8256 with a rectangular surrogate,
and 0.1256 with a constant derivative of 1 [arxiv:2102.04159 app. A.5]
[arxiv:2102.04159 tab. 9]. The speech-benchmark paper trains with the same
family of surrogate, a fast sigmoid u / (1 + β|u|), and does not say which of
its two stated steepness values produced its SNN results
[arxiv:1910.07407 §2.4.4] [arxiv:1910.07407 tab. 2].

### Spike-element-wise residual blocks

A plain residual spiking block applies the residual branch, then a spiking
neuron to the sum. The authors show that such a block cannot reproduce its
input for every neuron model, and that the gradient through k stacked blocks is
a product of surrogate derivatives, which vanishes or explodes with depth
[arxiv:2102.04159 §3.2] [arxiv:2102.04159 eq. 8]. The result is a deeper
Spiking ResNet that trains worse than a shallower one [arxiv:2102.04159 §4.1].

The spike-element-wise (SEW) block combines the spiking branch output A with the
block input S by an element-wise rule g, so the output is g(A, S)
[arxiv:2102.04159 eq. 9]. The three rules are ADD (A + S, which is not binary),
AND (A ∧ S, which equals A·S for binary inputs) and IAND ((¬A) ∧ S, which equals
(1 − A)·S for binary inputs). AND and IAND keep the output binary
[arxiv:2102.04159 §3.3] [arxiv:2102.04159 tab. 1]. For ADD and IAND, identity is
reached by making the branch output zero, which the paper does by zeroing the
last BatchNorm's weight and bias. Under identity the gradient through the block
is exactly 1 [arxiv:2102.04159 §3.3] [arxiv:2102.04159 eq. 11]. Card:
[Deep Residual Learning in Spiking Neural Networks](/papers/2102.04159). Related
concept: [Residual learning](/concepts/residual-learning).

On ImageNet, trained directly at T = 4, the SEW ResNet reaches 69.26% top-1 at
depth 152, where a Spiking ResNet of the same depth reaches 10.03%
[arxiv:2102.04159 tab. 2]. On DVS Gesture, the ADD rule scores 97.92%, IAND
95.49% and AND 70.49%, against 91.67% for a plain network without residual
connections [arxiv:2102.04159 tab. 4].

### Attention on the membrane potential

Attention gates the activity of a spiking layer, so that fewer neurons reach
threshold. The paper applies CBAM-style channel and spatial gates, plus a
temporal gate, to the membrane potential, with a shared MLP and a 7×7 spatial
convolution [arxiv:2209.13929 §3.3] [arxiv:2209.13929 eq. 10] [arxiv:2209.13929 eq. 12].
The three dimensions can be used alone, in pairs, or together as TCSA. The
combined form gives the highest accuracy in the paper's three-layer Gesture
ablation, 96.53% against 90.63% for the vanilla model, with spike count reduced
by 84.9% [arxiv:2209.13929 tab. 6] [arxiv:2209.13929 tab. 1]. The energy ratio
is estimated from operation counts, not measured on hardware
[arxiv:2209.13929 §4]. Card: [Attention Spiking Neural Networks](/papers/2209.13929).
Related concepts: [Attention mechanism](/concepts/attention-mechanism) and
[Channel attention](/concepts/channel-attention).

The residual form places the channel and spatial gates on the residual branch,
before the shortcut is added. On ImageNet-1K, with one time step, it adds 2.3
points over MS-Res-SNN-18 (63.97% against 61.70%), and with four steps it reaches
77.08% top-1 [arxiv:2209.13929 tab. 3] [arxiv:2209.13929 tab. 2]. These
ImageNet figures are the paper's, from its own re-implementation.

### Shadow training (ANN-to-SNN conversion)

Train a conventional network and convert its activations to rates or latencies.
This avoids the gradient problem but the conversion is approximate and usually
needs many time steps [arxiv:2109.12894 §4.1]. The paper says this approach
currently leads on static-image benchmarks, while noting those benchmarks rarely
test temporal dynamics [arxiv:2109.12894 §4.1]. The SEW paper's comparison table
lists conversion methods that use 250 to 4096 time steps for ResNet-34 and -50
[arxiv:2102.04159 tab. 3], against 4 steps for its own direct training.

### Spike-time gradients (SpikeProp)

Differentiate the spike time instead of the spike. The paper describes this as
giving an unbiased gradient in multi-layer networks. A neuron must fire for the
gradient to exist, so silent neurons receive no update [arxiv:2109.12894 §4.2].

### Online learning

Backpropagation through time stores the whole history of the network. Real-time
recurrent learning instead tracks the influence of each weight forward in time,
so the update can be made at each step [arxiv:2109.12894 §5] [arxiv:2109.12894 §5.2].
The paper lists SNN variants of this family: e-prop, decolle, OSTL, ETLP, OSTTP
and FPTT [arxiv:2109.12894 §5.2.1].

### Local rules and STDP

Perturbation learning, random feedback, per-layer local losses and
forward-forward style rules avoid the global backward pass. The paper reports
that they work on simple tasks but scale less well than gradient
backpropagation [arxiv:2109.12894 §3.4]. It also argues that the
backpropagation-through-time update for a spike pair decays with their time
difference, as spike-timing dependent plasticity curves do [arxiv:2109.12894 §5.3]
[arxiv:2109.12894 §4.3].

### Learned delays

Each recurrent connection gets a trainable delay, learned with its weight by
surrogate gradients. A delay can fall between two steps, so during training a
spike is shared between the two neighbouring steps by a triangular kernel
[arxiv:2509.24852 §2.1] [arxiv:2509.24852 eq. 7]. The sharing narrows over
training, and the delays are rounded to whole steps for inference
[arxiv:2509.24852 §4.4]. Delays can be axonal (one per neuron) or synaptic (one
per connection) [arxiv:2509.24852 §2.1]. The author-reported test accuracy with
plain LIF neurons is 82.42% on Spiking Speech Commands and 96.63% on PS-MNIST,
the latter with synaptic delays [arxiv:2509.24852 tab. 1]. Card:
[DelRec: learning delays in recurrent spiking neural networks](/papers/2509.24852).
Details are on [Learnable delays](/concepts/learnable-delays).

## Benchmarks

### Spiking Heidelberg Digits and Spiking Speech Commands

Two speech sets already converted to spikes [arxiv:1910.07407 §2]. Spiking
Heidelberg Digits (SHD) is recorded spoken digits in English and German, from
12 speakers. Two speakers are held out for the test set, so speaker
generalisation is measured [arxiv:1910.07407 §2.1.1]. Spiking Speech Commands
(SSC) is derived from Speech Commands v0.02 [arxiv:1910.07407 §2.1.2]. Both are
produced by one fixed audio-to-spike model with 700 channels, a basilar-membrane
filter bank, hair cells and bushy cells [arxiv:1910.07407 §2.2] [arxiv:1910.07407 app. A].
The fixed chain is the paper's answer to inconsistent encoders across studies
[arxiv:1910.07407 §1.1].

The paper's own results are test accuracy, author-reported. A recurrent SNN with
max-over-time readout reaches 71.4% on SHD and 50.9% on SSC. Tuning width, channel
merging, time constants and input jitter brings SHD to 83.2%. The LSTM and CNN
baselines, on binned spikes, reach 89.0% and 92.4% on SHD [arxiv:1910.07407 tab. 1].
Linear SVMs on spike counts stay near 56% on SHD. The authors take this gap as
evidence that spike timing carries information [arxiv:1910.07407 §3].

The conversion code is LGPL-2.1, so this page does not reproduce it
[arxiv:1910.07407 Paper vs code]. The paper's parameter table mixes unit systems,
and some of its values differ from the released conversion
[arxiv:1910.07407 tab. 2] [arxiv:1910.07407 app. A.1]. Check units before reusing
the table.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Accuracy matters most, memory is available | Surrogate-gradient BPTT | Paper calls gradient backpropagation the most accurate of the learning rules and recommends the surrogate as the starting point [arxiv:2109.12894 §3.4] [arxiv:2109.12894 §4.3.1] |
| Deep directly trained network, many spiking layers | Surrogate-gradient BPTT with SEW residual blocks, rule ADD | Plain spiking residual blocks degrade with depth; SEW ResNet-152 reaches 69.26% at T = 4 where Spiking ResNet-152 reaches 10.03% [arxiv:2102.04159 tab. 2] [arxiv:2102.04159 §4.1] |
| Each layer must pass binary spikes | SEW residual block with IAND, or AND with care | ADD output is not binary [arxiv:2102.04159 §3.3]. On DVS Gesture, IAND scores 95.49% against 97.92% for ADD, and AND scores 70.49% [arxiv:2102.04159 tab. 4] |
| Event-camera or other spiking input, and fewer spikes matter as much as accuracy | Attention on the membrane potential (TCSA form) | Author-reported: +5.9 points and 84.9% fewer spikes on three-layer Gesture, with the same time window as the vanilla model [arxiv:2209.13929 tab. 1]. Attention cost is small in parameters, but SimAM adds many MACs [arxiv:2209.13929 tab. 5] |
| Only a few time steps are affordable | Surrogate-gradient BPTT, with or without SEW blocks | SEW ResNet-34 reaches 67.04% at T = 4, while the conversion rows of the same table use 250 to 4096 steps [arxiv:2102.04159 tab. 3] |
| A good non-spiking model already exists and time steps are cheap | Shadow training | Conversion avoids the gradient, but needs many steps [arxiv:2109.12894 §4.1] |
| Long sequences or a memory budget that rules out storing history | Online learning (RTRL family) | Avoids storing the history; the paper gives the memory scaling only qualitatively [arxiv:2109.12894 §5.2] |
| Input is an event stream, such as from an event camera | Delta-modulation input with the same training choice | Encodes only changes in the input [arxiv:2109.12894 §3.1] |
| Static data is the input | Rate or latency code, with a warning | Encoding static data usually loses information; rate codes lose less [arxiv:2109.12894 §3.1] |
| Local, biologically motivated learning is the goal | Local rules or STDP-like updates | The paper reports they fit the constraints but lag on hard tasks [arxiv:2109.12894 §3.4] [arxiv:2109.12894 §5.3] |
| Spike timing carries the information | Spike-time gradients | Gradient exists only where neurons fire, so check firing first [arxiv:2109.12894 §4.2] |
| Speech input, and a shared spiking benchmark is wanted | SHD and SSC, with the published conversion | Held-out speakers on SHD; spike-count classifiers score below timing-aware ones [arxiv:1910.07407 §2.2] [arxiv:1910.07407 §3] |
| Accuracy on SHD is the goal and spiking is not required | Non-spiking LSTM or CNN on binned spikes | In the dataset paper, the LSTM (89.0%) and CNN (92.4%) beat its best SNN (83.2%) [arxiv:1910.07407 tab. 1] |
| Recurrent spiking network on temporal data, and delays on connections are allowed | Learned delays, axonal or synaptic | Author-reported gains over fixed delays on several temporal tasks; the better delay type changes by task [arxiv:2509.24852 tab. 3] |
| Deployment needs whole-step delays and the task has long delays | Learned delays, rounded after training | Rounding has little effect on most of the paper's tasks, but a large effect on AL, which keeps fractional delays [arxiv:2509.24852 §4.4] |

The paper does not settle the best loss function or code choice. It says the
best output loss depends on the network and task [arxiv:2109.12894 §3.3]. For
learning rates, batch sizes and epoch counts it gives no recipe
[arxiv:2109.12894 §4.3.3].

## Implementation sketch

Surrogate-gradient BPTT for one mini-batch. Neuron update and surrogate follow
the paper [arxiv:2109.12894 §2.1] [arxiv:2109.12894 §4.3.1]. The loss is any
output loss from [arxiv:2109.12894 §3.3].

```pseudocode
# Start every neuron at rest with no spikes  [arxiv:2109.12894 §2.1]
def snn_bptt(inputs, targets, weights, T, beta, theta):
    # inputs: (T, B, N_in); membrane and spikes: (B, N)
    membrane = zeros(B, N)
    spikes = zeros(B, N)
    spike_record = []
    membrane_record = []
    for t in range(T):
        # Leaky update: decay, add input, subtract last step's reset  [eq. 4]
        membrane = beta * membrane + inputs[t] @ weights
        membrane = membrane - spikes * theta
        # Hard threshold on the forward pass  [eq. 5]
        spikes = (membrane > theta).float()
        spike_record.append(spikes)
        membrane_record.append(membrane)
    # Loss summed over time  [arxiv:2109.12894 §4.3] [eq. 6]
    loss = output_loss(spike_record, membrane_record, targets)
    # Backward pass uses a smooth derivative in place of the step  [§4.3.1]
    # The reset term is treated as a constant  [§4.3.3]
    return backprop_through_time(loss, surrogate_derivative)
```

```pseudocode
# Smooth derivative used only on the backward pass  [§4.3.1]
# Peaks at 1 when the membrane sits exactly at the threshold
def surrogate_derivative(membrane, theta):
    # Distance from threshold, the argument of the arctan curve  [§4.3.1]
    shifted = membrane - theta
    # Form to confirm before use; see Contradictions  [§4.3.1]
    return 1 / (1 + (pi * shifted) ** 2)
```

Notes on the sketch:

- A weight update still needs a spike. If a neuron never fires, the surrogate
  passes error to earlier layers, but the weights on that neuron's output get
  no update [arxiv:2109.12894 §4.3.1].
- The printed reset listing in the paper resets to zero on every call, so the
  reset never takes effect. Keep the previous step's spikes as state
  [arxiv:2109.12894 §2.1].
- Online learning replaces the stored history with a per-weight influence that
  decays by β and accumulates the immediate input [arxiv:2109.12894 §5.2].
- For a residual SNN, the layer output becomes g(A, S) from
  [arxiv:2102.04159 eq. 9], where A is the spiking branch output and S the block
  input. Use ADD or IAND for the identity start. Zero the last BatchNorm so that
  A is zero at initialisation [arxiv:2102.04159 §3.3].

## Contradictions

- **Arctan surrogate derivative, within the paper.** The box in §4.3.1 prints the
  derivative as (1/π) · 1/(1+(πU)²), and gives the integral as (1/π)·arctan(πU).
  The derivative of that integral is 1/(1+(πU)²), without the 1/π factor. The
  paper does not resolve which form it intends [arxiv:2109.12894 §4.3.1]. Check
  the scale against your own framework before relying on it.
- **Surrogate shapes differ between sources.** The tutorial and the SEW paper use
  an arctan with α = 2 [arxiv:2109.12894 §4.3.1] [arxiv:2102.04159 app. A.1].
  DelRec uses a triangular surrogate on its four benchmarks, with width from 0.4
  to 1.0, and an arctan with α = 5 on Mackey-Glass [arxiv:2509.24852 §4.3]
  [arxiv:2509.24852 supp. Hyperparameters]. The three sources do not compare
  these shapes on a shared task, apart from the SEW CIFAR-10 ablation
  [arxiv:2102.04159 app. A.5]. The attention paper's main text states no
  surrogate shape [arxiv:2209.13929 §3.2].
- **Attention placement, paper versus released code.** The attention paper
  recommends channel and spatial gates on the integrated membrane potential, and
  temporal gate on the convolution output [arxiv:2209.13929 §6.1]
  [arxiv:2209.13929 fig. 4]. The released code applies all three gates to the
  batch-normalised convolution output, before the spiking step, and its channel
  gate has no membrane term [arxiv:2209.13929 Paper vs code]. Both are reported
  as the authors' work; the page does not say which one the reported numbers
  came from.
- **Same dataset name, different setups.** SSC appears in two sources. The
  dataset paper reports 50.9% for its recurrent SNN [arxiv:1910.07407 tab. 1].
  DelRec reports 82.42% with learned delays [arxiv:2509.24852 tab. 1]. The inputs
  differ: the dataset paper uses 700 channels at 0.5 ms steps over 1.0 s
  [arxiv:1910.07407 tab. 2], while DelRec bins to 140 channels and 250 steps of
  5.6 ms [arxiv:2509.24852 supp. Datasets]. The numbers are not comparable.
- **SHD, two reports.** The dataset paper's best recurrent SNN reaches 83.2%
  [arxiv:1910.07407 tab. 1]. DelRec reports 93.39% on SHD in its appendix, with
  augmentation, and leaves SHD out of its main table as saturated
  [arxiv:2509.24852 supp. Results on SHD] [arxiv:2509.24852 supp. The case of the SHD dataset].
  The two SNNs differ in network and training, so the gap does not measure the
  effect of delays.
- **No shared task across variants.** The tutorial reports no benchmarks of its
  own [arxiv:2109.12894 §1]. The learned-delay, SEW and speech numbers come from
  different tasks and are author-reported, so they do not rank the variants
  against each other. For HAR, the DelRec paper takes its reported number from a
  sweep scored on test accuracy [arxiv:2509.24852 §4.4], so treat that number as
  optimistic.

## Papers

- [Training Spiking Neural Networks Using Lessons From Deep Learning](/papers/2109.12894): tutorial and perspective on training SNNs with deep-learning tools; covers neuron models, codes, losses, surrogate-gradient BPTT, online learning and the STDP link. No new method or experiments.
- [Deep Residual Learning in Spiking Neural Networks](/papers/2102.04159): spike-element-wise residual blocks (ADD, AND, IAND) that let directly trained SNNs reach 152 layers; author-reported ImageNet and DVS Gesture results.
- [Attention Spiking Neural Networks](/papers/2209.13929): multi-dimensional attention (temporal, channel, spatial) on the membrane potential of conv-based SNNs, and an attention residual variant; author-reported Gesture, Gait and ImageNet results, with operation-count energy estimates.
- [DelRec: learning delays in recurrent spiking neural networks](/papers/2509.24852): trainable axonal or synaptic delays in recurrent LIF networks, trained with surrogate gradients; author-reported results on four temporal benchmarks.
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](/papers/1910.07407): SHD and SSC, two spike-based speech sets made with a fixed audio-to-spike model; author-reported baselines and recurrent SNN results.