Spiking neural networks

Overview

A spiking neural network (SNN) passes information as discrete spikes in time rather than as real-valued activations. Each neuron integrates its input into a membrane potential, leaks part of that potential at every step, and emits a spike when the potential crosses a threshold 2109.12894 §2.1. Because most neurons are silent at a given step, activity is sparse, which is the basis for the efficiency argument 2109.12894 §1 2109.12894 §2.

The shared difficulty is training. A spike is a step function of the membrane potential. Its derivative is zero almost everywhere and unbounded at the threshold, so standard gradients give no useful signal to neurons that have not fired 2109.12894 §4.3. Most training methods below are a different answer to this problem, or a way to avoid it. Learned delays keep the surrogate gradient and add a trainable parameter 2509.24852 §2.1. Residual wiring addresses a second problem, depth, and keeps the surrogate gradient 2102.04159 §3.3.

The sources are a tutorial and perspective, three method papers, and a dataset paper. The tutorial reports no benchmark results of its own 2109.12894 §1. The method papers and the dataset paper report benchmark accuracy for their own models, which is author-reported 2509.24852 tab. 1 2102.04159 tab. 2 2209.13929 tab. 1 1910.07407 tab. 1. Figures in this page are the papers' statements, not measurements from our side.

Variants

Leaky integrate-and-fire neuron

The usual neuron model. The membrane decays by a factor β each step, takes the weighted input, and is reduced by the threshold after a spike. Its discrete update is derived from an RC circuit with forward Euler 2109.12894 §2.1 2109.12894 app. A. The paper recommends this neuron as the default for gradient-based training, with richer neuron models when more capacity is needed 2109.12894 §2.2. Card: Training Spiking Neural Networks Using Lessons From Deep Learning.

Input and output codes

Rate coding turns an intensity into a spike probability per step. Latency coding turns it into the time of one spike. Delta modulation emits a spike only when the input changes, as an event camera does 2109.12894 §3.1 2109.12894 app. B. On the output side, the class can be read from the spike count, from the first neuron to fire, or from a population of neurons per class 2109.12894 §3.2. The paper notes that encoding ordinary static data usually discards information, and that rate codes lose less accuracy than latency codes 2109.12894 §3.1.

Surrogate-gradient backpropagation through time

The most widely used route. The network is unrolled over the time steps, and the loss is summed over time 2109.12894 §4.3 2109.12894 eq. 6. In the forward pass the hard threshold stays. In the backward pass it is replaced by a smooth function whose derivative supplies the gradient 2109.12894 §4.3.1 2109.12894 eq. 8. The paper names an arctan curve as its starting point for this surrogate 2109.12894 §4.3.1. The reset term is excluded from the backward pass 2109.12894 §4.3.3. The paper also relates the resulting update to the straight-through estimator used for quantised networks 2109.12894 §4.3.2, which links this variant to Quantization-aware training.

The shape of the surrogate matters in practice. On CIFAR-10, the SEW ResNet paper reports accuracy 0.8263 with arctan, 0.8256 with a rectangular surrogate, and 0.1256 with a constant derivative of 1 2102.04159 app. A.5 2102.04159 tab. 9. The speech-benchmark paper trains with the same family of surrogate, a fast sigmoid u / (1 + β|u|), and does not say which of its two stated steepness values produced its SNN results 1910.07407 §2.4.4 1910.07407 tab. 2.

Spike-element-wise residual blocks

A plain residual spiking block applies the residual branch, then a spiking neuron to the sum. The authors show that such a block cannot reproduce its input for every neuron model, and that the gradient through k stacked blocks is a product of surrogate derivatives, which vanishes or explodes with depth 2102.04159 §3.2 2102.04159 eq. 8. The result is a deeper Spiking ResNet that trains worse than a shallower one 2102.04159 §4.1.

The spike-element-wise (SEW) block combines the spiking branch output A with the block input S by an element-wise rule g, so the output is g(A, S) 2102.04159 eq. 9. The three rules are ADD (A + S, which is not binary), AND (A ∧ S, which equals A·S for binary inputs) and IAND ((¬A) ∧ S, which equals (1 − A)·S for binary inputs). AND and IAND keep the output binary 2102.04159 §3.3 2102.04159 tab. 1. For ADD and IAND, identity is reached by making the branch output zero, which the paper does by zeroing the last BatchNorm's weight and bias. Under identity the gradient through the block is exactly 1 2102.04159 §3.3 2102.04159 eq. 11. Card: Deep Residual Learning in Spiking Neural Networks. Related concept: Residual learning.

On ImageNet, trained directly at T = 4, the SEW ResNet reaches 69.26% top-1 at depth 152, where a Spiking ResNet of the same depth reaches 10.03% 2102.04159 tab. 2. On DVS Gesture, the ADD rule scores 97.92%, IAND 95.49% and AND 70.49%, against 91.67% for a plain network without residual connections 2102.04159 tab. 4.

Attention on the membrane potential

Attention gates the activity of a spiking layer, so that fewer neurons reach threshold. The paper applies CBAM-style channel and spatial gates, plus a temporal gate, to the membrane potential, with a shared MLP and a 7×7 spatial convolution 2209.13929 §3.3 2209.13929 eq. 10 2209.13929 eq. 12. The three dimensions can be used alone, in pairs, or together as TCSA. The combined form gives the highest accuracy in the paper's three-layer Gesture ablation, 96.53% against 90.63% for the vanilla model, with spike count reduced by 84.9% 2209.13929 tab. 6 2209.13929 tab. 1. The energy ratio is estimated from operation counts, not measured on hardware 2209.13929 §4. Card: Attention Spiking Neural Networks. Related concepts: Attention mechanism and Channel attention.

The residual form places the channel and spatial gates on the residual branch, before the shortcut is added. On ImageNet-1K, with one time step, it adds 2.3 points over MS-Res-SNN-18 (63.97% against 61.70%), and with four steps it reaches 77.08% top-1 2209.13929 tab. 3 2209.13929 tab. 2. These ImageNet figures are the paper's, from its own re-implementation.

Shadow training (ANN-to-SNN conversion)

Train a conventional network and convert its activations to rates or latencies. This avoids the gradient problem but the conversion is approximate and usually needs many time steps 2109.12894 §4.1. The paper says this approach currently leads on static-image benchmarks, while noting those benchmarks rarely test temporal dynamics 2109.12894 §4.1. The SEW paper's comparison table lists conversion methods that use 250 to 4096 time steps for ResNet-34 and -50 2102.04159 tab. 3, against 4 steps for its own direct training.

Spike-time gradients (SpikeProp)

Differentiate the spike time instead of the spike. The paper describes this as giving an unbiased gradient in multi-layer networks. A neuron must fire for the gradient to exist, so silent neurons receive no update 2109.12894 §4.2.

Online learning

Backpropagation through time stores the whole history of the network. Real-time recurrent learning instead tracks the influence of each weight forward in time, so the update can be made at each step 2109.12894 §5 2109.12894 §5.2. The paper lists SNN variants of this family: e-prop, decolle, OSTL, ETLP, OSTTP and FPTT 2109.12894 §5.2.1.

Local rules and STDP

Perturbation learning, random feedback, per-layer local losses and forward-forward style rules avoid the global backward pass. The paper reports that they work on simple tasks but scale less well than gradient backpropagation 2109.12894 §3.4. It also argues that the backpropagation-through-time update for a spike pair decays with their time difference, as spike-timing dependent plasticity curves do 2109.12894 §5.3 2109.12894 §4.3.

Learned delays

Each recurrent connection gets a trainable delay, learned with its weight by surrogate gradients. A delay can fall between two steps, so during training a spike is shared between the two neighbouring steps by a triangular kernel 2509.24852 §2.1 2509.24852 eq. 7. The sharing narrows over training, and the delays are rounded to whole steps for inference 2509.24852 §4.4. Delays can be axonal (one per neuron) or synaptic (one per connection) 2509.24852 §2.1. The author-reported test accuracy with plain LIF neurons is 82.42% on Spiking Speech Commands and 96.63% on PS-MNIST, the latter with synaptic delays 2509.24852 tab. 1. Card: DelRec: learning delays in recurrent spiking neural networks. Details are on Learnable delays.

Benchmarks

Spiking Heidelberg Digits and Spiking Speech Commands

Two speech sets already converted to spikes 1910.07407 §2. Spiking Heidelberg Digits (SHD) is recorded spoken digits in English and German, from 12 speakers. Two speakers are held out for the test set, so speaker generalisation is measured 1910.07407 §2.1.1. Spiking Speech Commands (SSC) is derived from Speech Commands v0.02 1910.07407 §2.1.2. Both are produced by one fixed audio-to-spike model with 700 channels, a basilar-membrane filter bank, hair cells and bushy cells 1910.07407 §2.2 1910.07407 app. A. The fixed chain is the paper's answer to inconsistent encoders across studies 1910.07407 §1.1.

The paper's own results are test accuracy, author-reported. A recurrent SNN with max-over-time readout reaches 71.4% on SHD and 50.9% on SSC. Tuning width, channel merging, time constants and input jitter brings SHD to 83.2%. The LSTM and CNN baselines, on binned spikes, reach 89.0% and 92.4% on SHD 1910.07407 tab. 1. Linear SVMs on spike counts stay near 56% on SHD. The authors take this gap as evidence that spike timing carries information 1910.07407 §3.

The conversion code is LGPL-2.1, so this page does not reproduce it 1910.07407 Paper vs code. The paper's parameter table mixes unit systems, and some of its values differ from the released conversion 1910.07407 tab. 2 1910.07407 app. A.1. Check units before reusing the table.

When to use which

The paper does not settle the best loss function or code choice. It says the best output loss depends on the network and task 2109.12894 §3.3. For learning rates, batch sizes and epoch counts it gives no recipe 2109.12894 §4.3.3.

Implementation sketch

Surrogate-gradient BPTT for one mini-batch. Neuron update and surrogate follow the paper 2109.12894 §2.1 2109.12894 §4.3.1. The loss is any output loss from 2109.12894 §3.3.

# Start every neuron at rest with no spikes  [arxiv:2109.12894 §2.1]
def snn_bptt(inputs, targets, weights, T, beta, theta):
    # inputs: (T, B, N_in); membrane and spikes: (B, N)
    membrane = zeros(B, N)
    spikes = zeros(B, N)
    spike_record = []
    membrane_record = []
    for t in range(T):
        # Leaky update: decay, add input, subtract last step's reset  [eq. 4]
        membrane = beta * membrane + inputs[t] @ weights
        membrane = membrane - spikes * theta
        # Hard threshold on the forward pass  [eq. 5]
        spikes = (membrane > theta).float()
        spike_record.append(spikes)
        membrane_record.append(membrane)
    # Loss summed over time  [arxiv:2109.12894 §4.3] [eq. 6]
    loss = output_loss(spike_record, membrane_record, targets)
    # Backward pass uses a smooth derivative in place of the step  [§4.3.1]
    # The reset term is treated as a constant  [§4.3.3]
    return backprop_through_time(loss, surrogate_derivative)

# Smooth derivative used only on the backward pass  [§4.3.1]
# Peaks at 1 when the membrane sits exactly at the threshold
def surrogate_derivative(membrane, theta):
    # Distance from threshold, the argument of the arctan curve  [§4.3.1]
    shifted = membrane - theta
    # Form to confirm before use; see Contradictions  [§4.3.1]
    return 1 / (1 + (pi * shifted) ** 2)

Notes on the sketch:

• A weight update still needs a spike. If a neuron never fires, the surrogate passes error to earlier layers, but the weights on that neuron's output get no update 2109.12894 §4.3.1.
• The printed reset listing in the paper resets to zero on every call, so the reset never takes effect. Keep the previous step's spikes as state 2109.12894 §2.1.
• Online learning replaces the stored history with a per-weight influence that decays by β and accumulates the immediate input 2109.12894 §5.2.
• For a residual SNN, the layer output becomes g(A, S) from 2102.04159 eq. 9, where A is the spiking branch output and S the block input. Use ADD or IAND for the identity start. Zero the last BatchNorm so that A is zero at initialisation 2102.04159 §3.3.

Contradictions

• Arctan surrogate derivative, within the paper. The box in §4.3.1 prints the derivative as (1/π) · 1/(1+(πU)²), and gives the integral as (1/π)·arctan(πU). The derivative of that integral is 1/(1+(πU)²), without the 1/π factor. The paper does not resolve which form it intends 2109.12894 §4.3.1. Check the scale against your own framework before relying on it.
• Surrogate shapes differ between sources. The tutorial and the SEW paper use an arctan with α = 2 2109.12894 §4.3.1 2102.04159 app. A.1. DelRec uses a triangular surrogate on its four benchmarks, with width from 0.4 to 1.0, and an arctan with α = 5 on Mackey-Glass 2509.24852 §4.3 2509.24852 supp. Hyperparameters. The three sources do not compare these shapes on a shared task, apart from the SEW CIFAR-10 ablation 2102.04159 app. A.5. The attention paper's main text states no surrogate shape 2209.13929 §3.2.
• Attention placement, paper versus released code. The attention paper recommends channel and spatial gates on the integrated membrane potential, and temporal gate on the convolution output 2209.13929 §6.1 2209.13929 fig. 4. The released code applies all three gates to the batch-normalised convolution output, before the spiking step, and its channel gate has no membrane term 2209.13929 Paper vs code. Both are reported as the authors' work; the page does not say which one the reported numbers came from.
• Same dataset name, different setups. SSC appears in two sources. The dataset paper reports 50.9% for its recurrent SNN 1910.07407 tab. 1. DelRec reports 82.42% with learned delays 2509.24852 tab. 1. The inputs differ: the dataset paper uses 700 channels at 0.5 ms steps over 1.0 s 1910.07407 tab. 2, while DelRec bins to 140 channels and 250 steps of 5.6 ms 2509.24852 supp. Datasets. The numbers are not comparable.
• SHD, two reports. The dataset paper's best recurrent SNN reaches 83.2% 1910.07407 tab. 1. DelRec reports 93.39% on SHD in its appendix, with augmentation, and leaves SHD out of its main table as saturated 2509.24852 supp. Results on SHD 2509.24852 supp. The case of the SHD dataset. The two SNNs differ in network and training, so the gap does not measure the effect of delays.
• No shared task across variants. The tutorial reports no benchmarks of its own 2109.12894 §1. The learned-delay, SEW and speech numbers come from different tasks and are author-reported, so they do not rank the variants against each other. For HAR, the DelRec paper takes its reported number from a sweep scored on test accuracy 2509.24852 §4.4, so treat that number as optimistic.

Papers

• Training Spiking Neural Networks Using Lessons From Deep Learning: tutorial and perspective on training SNNs with deep-learning tools; covers neuron models, codes, losses, surrogate-gradient BPTT, online learning and the STDP link. No new method or experiments.
• Deep Residual Learning in Spiking Neural Networks: spike-element-wise residual blocks (ADD, AND, IAND) that let directly trained SNNs reach 152 layers; author-reported ImageNet and DVS Gesture results.
• Attention Spiking Neural Networks: multi-dimensional attention (temporal, channel, spatial) on the membrane potential of conv-based SNNs, and an attention residual variant; author-reported Gesture, Gait and ImageNet results, with operation-count energy estimates.
• DelRec: learning delays in recurrent spiking neural networks: trainable axonal or synaptic delays in recurrent LIF networks, trained with surrogate gradients; author-reported results on four temporal benchmarks.
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks: SHD and SSC, two spike-based speech sets made with a fixed audio-to-spike model; author-reported baselines and recurrent SNN results.