Deep Residual Learning in Spiking Neural Networks

In plain words

Spiking neural networks compute with on/off pulses instead of smooth numbers, which makes them cheap on suitable hardware. Training very deep ones has been hard. A standard trick for deep networks is the residual block: the block adds its input back to its output, so the network can pass information straight through when a layer has nothing useful to add. Earlier attempts to copy this into spiking networks simply swapped the activation for a spiking neuron, and the result got worse as it got deeper.

The authors change how the shortcut is combined with the block's output. Instead of adding two numbers, they combine two spike trains element by element, with a simple logical rule. Because the rule works on on/off signals, the block can pass its input through unchanged by making the inner branch stay silent. Gradients then flow back through the shortcut without shrinking or blowing up. With this change the authors train networks with more than 100 layers directly, and report that deeper versions do better, as in ordinary ResNets.

TL;DR

Each SEW block outputs g(A, S), where A is a spiking residual branch and S the block input, and g is ADD, AND or IAND eq. 9 tab. 1. For ADD and IAND, identity mapping is reachable for every neuron model, and gradients through identity blocks are exactly 1 §3.3 eq. 11. Trained with SGD and a cosine schedule for 320 epochs on ImageNet at T=4, SEW ResNet-152 reaches 69.26% top-1 tab. 2. The released code matches the block definition, but the paper leaves several training details out, and some of its numbers disagree with its own code [Paper vs code].

Problem

Spiking ResNet copies the ANN block and swaps ReLU for spiking neurons §2.2 eq. 7. The authors argue this block cannot pass its input through unchanged for neurons with complex dynamics such as learnable-τ LIF §3.2. Its gradient through k stacked blocks is a product of surrogate derivatives, so it vanishes or explodes depending on whether that derivative is below or above 1 §3.2 eq. 8. The result is a degradation problem: deeper directly trained Spiking ResNets have higher training loss than shallower ones §4.1 fig. 3. Converted (ANN-to-SNN) networks sidestep this training problem, but need longer simulation to match the ANN §2.1.

Method

Neuron model §3.1. A single discrete-time model covers the spiking neurons used here. The membrane potential after dynamics is H[t]. A spike is emitted when H[t] reaches the threshold, and the membrane is then reset eq. 2 eq. 3. The dynamics function f differs per neuron: IF integrates the input directly eq. 4, and LIF leaks towards the reset potential with time constant τ eq. 5. The Heaviside step has zero derivative almost everywhere, so backpropagation uses a surrogate σ in its place §3.1 app. A.1.

S[t] = \Theta(H[t] - V_{th}) eq. 2

V[t] = H[t] \left(1 - S[t]\right) + V_{reset} S[t] eq. 3

Spiking ResNet block and its drawbacks §3.2. The Spiking ResNet block applies the residual branch and then a spiking neuron to the sum eq. 7. The authors show that identity mapping needs the last neuron to reproduce its input spike exactly. That holds for IF with 0 < V_th ≤ 1 and an input of 0 or 1 §3.2. For LIF with learnable τ, no fixed threshold guarantees it §3.2.

O^{l}[t] = SN(\mathcal{F}^{l}(S^{l}[t]) + S^{l}[t]) eq. 7

For k blocks in identity regions, the gradient with respect to the input is a product of surrogate derivatives eq. 8. The authors argue that since a spike is 0 or 1 and the common surrogates have derivative below 1 at the spike points, this product vanishes §3.2.

SEW block §3.3. The block computes the residual mapping A = SN(F(S)), then combines it with the input S by an element-wise function g eq. 9. The three choices in the paper's Table 1 are:

• ADD: A + S. Not binary; k stacked blocks give outputs at most k + 1 §3.3.
• AND: A ∧ S, which equals A·S for binary inputs. Output stays binary §3.3.
• IAND: (¬A) ∧ S, which equals (1 − A)·S for binary inputs. Output stays binary §3.3.

O^{l}[t] = g(A^{l}[t], S^{l}[t]), \quad A^{l}[t] = SN(\mathcal{F}^{l}(S^{l}[t])) eq. 9

For ADD and IAND, identity mapping is reached by setting A ≡ 0, which means zeroing the last BatchNorm's weight and bias in F §3.3. For AND, identity needs A ≡ 1, which the authors reach by zeroing the BatchNorm weight and setting its bias to a large constant (V_th for IF neurons) §3.3. Under identity mapping, the gradient through k SEW blocks is exactly 1 eq. 11.

\frac{\partial O^{l+k-1}_{j}[t]}{\partial S^{l}_{j}[t]} = 1 eq. 11 (under identity mapping)

When the input and output shapes differ, the shortcut is a strided convolution followed by BatchNorm and a spiking neuron, which the paper calls a downsample SEW block §3.3 fig. 2.

# One time-step of an IF neuron, V_th = 1, V_reset = 0  [app. A.1]
# Input:  input current x (B, ...), membrane v (B, ...)
# Output: spikes s (B, ...), next membrane v (B, ...)
def if_step(x, v):
    # Integrate the input into the membrane  [eq. 4]
    h = v + x
    # Fire where h reaches the threshold; the backward pass uses the
    # surrogate derivative instead of the step's derivative  [§3.1]
    s = heaviside(h - 1.0)
    # Reset to 0 where a spike fired; S is detached in the reset  [app. A.1]
    v = h * (1.0 - s.detach())
    return s, v

# SEW residual block over all time-steps  [eq. 9] [tab. 1]
# Input:  spikes (T, B, C, H, W); residual_branch = conv, BN, SN stack
# Output: (T, B, C, H, W); binary for AND and IAND, not binary for ADD
def sew_block(spikes, residual_branch, connect):
    # Residual mapping A: the branch ends in a spiking neuron  [eq. 9]
    residual = residual_branch(spikes)
    # Combine A with the block input S by the element-wise rule g  [tab. 1]
    if connect == "ADD":
        return residual + spikes
    if connect == "AND":
        return residual * spikes
    if connect == "IAND":
        return (1.0 - residual) * spikes
    raise ValueError(connect)

Surrogate function app. A.1. The paper uses an arctan surrogate with slope α = 2 in all experiments. Its forward is a smoothed step and its backward is the derivative below:

\sigma(x) = \frac{1}{\pi} \arctan\left(\frac{\pi}{2} \alpha x\right) + \frac{1}{2} app. A.1

\sigma'(x) = \frac{\alpha}{2 \left(1 + \left(\frac{\pi}{2} \alpha x\right)^{2}\right)} app. A.1

ImageNet networks §4.1 sew_resnet.py. The architecture is the ResNet family with each residual block replaced by a SEW block (ADD) and the downsample shortcut given a spiking neuron. ResNet-18 and -34 use the basic block; ResNet-50, -101 and -152 use the bottleneck block. Block counts per stage are 2-2-2-2, 3-4-6-3, 3-4-6-3, 3-4-23-3 and 3-8-36-3 sew_resnet.py. All neurons are IF. T = 4 time-steps §4.1 tab. 7.

DVS Gesture network 7B-Net §4.2. The stated structure is c32k3s1-BN-PLIF-{SEW block-MPk2s2}×7-FC11, with PLIF neurons (LIF with learnable τ) §4.2. Each SEW block here has two 3×3 conv-BN-PLIF stages smodels.py. The paper trains with T = 16 and random temporal delete (keep 12 of 16 frames during training) app. A.2.

CIFAR10-DVS network Wide-7B-Net §4.3. The stated structure is c64k3s1-BN-PLIF-{SEW(c64)-MPk2s2}×4-c128k3s1-BN-PLIF-{SEW(c128)-MPk2s2}×3-FC10 §4.3. Random temporal delete is not used here app. A.1.

Reproducibility app. A.6. The paper says identical random seeds are used throughout, and that all experiments use SpikingJelly on PyTorch app. A.6.

Hyperparameters

Unless a row says otherwise, values are the paper's. "Code" entries are the repository defaults or logged runs.

Results

Author-reported. Each block lists the setup. The paper reports single numbers per setting. No error bars appear in the compiled text.

ImageNet, SEW ResNet (ADD) vs Spiking ResNet, T = 4, ImageNet validation set used as test §4.1 tab. 2:

ImageNet, comparison with other SNNs §4.1 tab. 3:

The text states that SEW ResNet-34 uses 21.8M parameters against 85.5M for the larger td-BN Spiking ResNet-34 §4.1.

DVS Gesture, ablation of g and the plain block §4.2 tab. 4:

DVS Gesture, comparison §4.2 tab. 5: 7B-Net 97.92% with 0.13M parameters at T = 16; the cited Fang et al. 2020 network 97.57% with 1.70M at T = 20; Spiking ResNet-17 with td-BN 96.87% with 11.18M at T = 40; the cited HE2020108 network 93.40% with 23.23M at T = 60.

CIFAR10-DVS, Wide-7B-Net §4.3 tab. 6: 64.8, 70.2 and 74.4% at T = 4, 8 and 16, with 1.19M parameters. Spiking ResNet-19 with td-BN is 67.8% at T = 10 with 11.18M parameters. The Fang et al. 2020 network is 74.8% at T = 20 with 17.4M parameters.

Surrogate ablation on CIFAR-10, 0/1 gradients app. A.5 tab. 9: the SEW ResNet (ADD) reaches 0.8263 with arctan, 0.8256 with rectangular and 0.1256 with constant 1. The Spiking ResNet reaches 0.7733, 0.6601 and 0.1. The paper treats this as a small lightweight network, not state of the art app. A.5.

Gradient and firing-rate analysis §4.1 fig. 4 fig. 5 fig. 6 fig. 7 fig. 8. Firing rates of the last neuron in each SEW block are low, mostly at or below 0.25 for ResNet-18/34/50 at T = 4 §4.1. Gradient norms are constant through identity regions for SEW under zero initialisation §4.1. Without zero initialisation, the AND variant goes silent in deep layers, and IAND keeps firing §4.1 fig. 5.

Paper vs code

The official repository is fangwei123456/Spike-Element-Wise-ResNet, linked from the abstract. It is licensed under MPL-2.0, a file-level copyleft licence. No code has been quoted verbatim. All code claims below are paraphrased LICENSE. The commit is cfc547c. The repository depends on SpikingJelly at commit 2958519, which is not in the repository. Neuron and surrogate defaults therefore cannot be checked here readme.md.

• Block structure matches. The code's ADD, AND and IAND rules match g in the paper, and the downsample shortcut has a spiking neuron, as the paper describes sew_resnet.py sew_resnet.py smodels.py.
• Surrogate. The paper says arctan with α = 2 everywhere app. A.1. The repository does not set a surrogate for any neuron, so the value comes from the pinned SpikingJelly version, which is not in the repository sew_resnet.py smodels.py.
• DVS stem. The paper's 7B-Net stem is a 3×3 convolution §4.2. The code uses a 1×1 convolution for the first layer, because the entry's up-kernel size is 1 smodels.py smodels.py. The CIFAR10-DVS code does the same smodels.py smodels.py.
• ImageNet test preprocessing. The paper says a 224 resize and centre crop app. A.1. The code resizes to 256 first, then crops to 224 train.py.
• ImageNet training crop. The paper says a random 224 crop with flip app. A.1. The code uses RandomResizedCrop, which samples a scale and aspect ratio before resizing train.py.
• Batch size. The paper table lists batch 32 and 8 GPUs tab. 7. The code's batch size is per process train.py, so an 8-GPU run uses 256 images per step. The paper does not say which is meant.
• Zero initialisation. The paper uses zero initialisation for Spiking ResNet and says nothing for SEW §4.1. The readme command for SEW omits the flag readme.md. The code's zero-init helper sets the last BatchNorm weight to zero, and sets the bias to 1 only for AND sew_resnet.py. For IF neurons with V_th = 1, that matches the paper's "large constant" for AND.
• Readout. The paper does not describe the readout. ImageNet code averages the pooled features over time, then applies the fully connected layer sew_resnet.py. DVS code averages the outputs over time, then applies a linear layer smodels.py.
• Input encoding. The paper does not state it. The ImageNet code feeds the same image at every time-step: the stem convolution runs once and its output is repeated T times sew_resnet.py.
• Residual-block counts. The paper says there are 7, 15 and 15 SEW blocks in ResNet-18, -34 and -50, and 33 and 50 in ResNet-101 and -152 §4.1. The code's block counts are 8, 16, 16, 33 and 50 sew_resnet.py. The 101 and 152 counts match. The paper does not explain the one-block difference for 18, 34 and 50.
• DVS learning rate. The paper's Table 7 says 0.1 for DVS Gesture, and Table 8 gives 0.001 for ADD. The readme and logged run use 0.001 readme.md args.txt.
• CIFAR10-DVS learning rate. The code default is 0.1 train.py. The logged run uses 0.01, matching the paper args.txt.
• Weight decay. The paper does not state a value for ImageNet. The code default and logged run are both 0 train.py.
• Label smoothing. The code uses plain cross-entropy on ImageNet train.py. The paper does not mention label smoothing for SEW.
• Detached reset. The paper detaches S in the reset app. A.1. The code passes detach_reset=True to every neuron sew_resnet.py.
• State reset between batches. The code resets every neuron's state after each batch train.py train.py. The paper does not describe this step.
• Random temporal delete. The paper keeps 12 of 16 slices, and sorts the chosen slices app. A.2. The code uses the same count and sorts the indices train.py.
• Seeds. The code seeds Python, NumPy and PyTorch, and sets cuDNN to deterministic train.py train.py. The paper states identical seeds app. A.6.

Pitfalls

• Surrogate and neuron defaults are not in the repository. The paper's arctan with α = 2 and the code's neuron settings depend on a pinned SpikingJelly version. The readme warns that this version has a bug in MultiStepParametricLIFNode, which the DVS and CIFAR10-DVS models use readme.md. Check the library bug page before reproducing the DVS numbers.
• Identity mapping and the gradient result are conditional. Eq. 11 equals 1 only when A is held constant at 0 or 1 in the identity region §3.3. In general training A is not constant, and the paper gives no bound for that case.
• AND needs care. The AND variant goes silent in deep layers without zero initialisation, and reaches only 70.49% on DVS Gesture §4.1 tab. 4 app. A.3. Prefer ADD or IAND unless you have a reason to use AND.
• Surrogate choice is sensitive. Constant-1 surrogates fail in both networks, and the rectangular surrogate hurts the Spiking ResNet a lot app. A.5 tab. 9. Keep arctan with α = 2.
• Zero initialisation is needed for deep Spiking ResNets, but not for SEW. The paper says the deep Spiking ResNet-50, -101 and -152 do not converge without it §4.1. SEW ResNet is run without it in the released commands readme.md.
• Hyperparameter tables conflict. Table 7 and Table 8 disagree on the DVS learning rate, and the batch size can be read as per GPU or total tab. 7 tab. 8. The code and logs settle both (see Paper vs code).
• Gradient plots cover one setting each. The gradient-amplitude figures are for the 152-layer structure under one (V_th, α) pair each §4.1 fig. 6 fig. 7 fig. 8. They do not show training dynamics.
• Results are single runs with fixed hyperparameters. The paper does not report variance across seeds §4.1 tab. 2. Expect run-to-run differences.
• Compute cost is not stated. The ImageNet logs record 8 GPUs and 320 epochs, but the paper gives no training time or memory figure args.txt.
• The "first over 100 layers" claim is the authors' own. The paper says it is the first to train a directly trained SNN with more than 100 layers §1. It does not survey all prior work, so treat it as a claim.

Lineage

• Builds on: ResNet residual learning §1 §3.2; ReLU-before-addition blocks, which the SEW block extends §3.3; surrogate-gradient training §2.1 §3.1; PLIF neurons, used in the DVS and CIFAR10-DVS networks §3.2 §4.2. Papers without a card are named only.
• Compared against: Spiking ResNet with td-BN (Zheng et al., 2020) §2.2 tab. 3; ANN-to-SNN conversion methods tab. 3; the PLIF-based network of Fang et al. (2020) on DVS Gesture and CIFAR10-DVS tab. 5 tab. 6.
• Superseded by: not stated in the paper. The readme says SpikingJelly implements SEW ResNet for ImageNet in later versions, but does not say it replaces this work readme.md.