Attention Spiking Neural Networks

In plain words

Spiking neural networks compute with sparse on-off pulses instead of dense numbers, which can save a lot of energy on suitable hardware. The catch is that they have historically been less accurate than ordinary networks, and very deep ones are hard to train.

This paper adds attention to spiking networks. Attention lets the network decide which moments in time, which feature channels and which image locations matter most for the current input. Here the attention is applied to the internal charge of each spiking neuron, before the neuron decides whether to fire. Unimportant background activity is pushed below the firing threshold, so fewer neurons fire at all. The authors report that this improves accuracy and reduces spiking at the same time, on event-camera gesture and gait recognition and on ImageNet classification.

TL;DR

• Multi-dimensional Attention (MA) gates a spiking layer over time (when), channels (what) and space (where), using CBAM-style modules applied to the membrane potential §3.3.
• On DVS128 Gesture and Gait, the three-dimensional TCSA variant raises accuracy by 5.9 and 4.7 points and reduces spike counts by 84.9% and 81.6% over the re-implemented vanilla baselines tab. 1.
• A residual variant (Att-Res-SNN, built on MS-Res-SNN) reaches 77.08% top-1 on ImageNet-1K with 4 time steps, and 75.92% with 1 step, both author-reported tab. 2.
• The code releases the conv-based and residual models. It places attention before the spiking step, not on the membrane potential as the paper recommends (see Paper vs code).

Problem

Deep spiking networks trained directly need many time steps, which raises latency, training cost and memory for stored membrane potentials §1. Deep SNNs also suffer spike degradation and gradient vanishing, so adding layers can make accuracy worse §1, §3.4. Earlier attention modules for CNNs act on continuous activations and do not preserve the event-driven, sparse behaviour that gives SNNs their energy advantage §1, §6.1. The authors aim for attention that keeps spiking sparse and improves accuracy together §1, §4.

Method

Input. Event streams are converted to frames: events in each short window are summed per pixel into a 2-channel frame, and frames are grouped into T time steps §3.1 eq. 1 eq. 2. Static images are fed unchanged at every time step §3.1.

Neuron model. The layer is a convolution, batch normalisation and average pooling, followed by a discrete leaky integrate-and-fire update. The membrane accumulates the input, fires when it crosses a threshold, resets fired neurons to a reset value, and decays the others by a factor β §3.2 eq. 4. The decay is β = exp(−dt/τ) §3.2 eq. 4. The paper's Heaviside spike function is trained with a surrogate gradient §2 §3.2.

Multi-dimensional attention. Each dimension computes a sigmoid gate from average- and max-pooled descriptors through a shared MLP with reduction ratio r, and multiplies the gate onto its input §3.3 eq. 6 eq. 7:

• Temporal (TA): one weight per time step, pooled over channel and space, applied to the input block over all time steps §3.3 eq. 8 eq. 9.
• Channel (CA): one weight per channel at each time step, pooled over space, with the MLP shared across time steps. It is the CBAM channel module §3.3 eq. 10 eq. 11.
• Spatial (SA): a 7×7 convolution over the channel-wise average and max maps, shared across time steps §3.3 eq. 12 eq. 13.

The three dimensions can run alone or in pairs, or together as TCSA, which applies TA to the input, then CA on the membrane input, then SA §3.3 eq. 14 tab. 6. The recommended location puts CA and SA on the integrated membrane potential, while TA sits on the convolution output because it cannot act on state that already holds spatio-temporal information §6.1 fig. 4.

Attention residual learning (Att-Res-SNN). The authors take MS-Res-SNN (membrane-potential shortcuts) as the backbone §3.4. In Att-Res-SNN-1, the CSA module acts on the residual branch before the shortcut is added §3.4 eq. 15 fig. 3. In Att-Res-SNN-2, the attention is moved after the shortcut §3.4 eq. 16 fig. 5. The paper recommends variant 1 §3.4.

Energy estimate. Energy is estimated from operation counts, not measured on hardware. The first encoder layer uses multiply-accumulate (MAC) operations; later layers use accumulate (AC) operations, with E_MAC = 4.6 pJ and E_AC = 0.9 pJ for 32-bit operations in 45 nm §4 eq. 17. The energy shift of an attention SNN is computed from added MACs and removed ACs eq. 18. The energy-efficiency ratio is the baseline energy divided by the attention energy eq. 19. The network average spiking activity rate (NASAR) is the average over time steps of the fraction of neurons that spike; spike counts are NASAR times neuron number times T §4.

# LIF step at time t  [eq. 4]
# Input:  U_t (C, H, W) membrane after attention; H_prev (C, H, W)
def lif_step(U_t, u_th, V_reset, beta):
    # fire where the membrane exceeds the threshold  [eq. 4]
    S_t = heaviside(U_t - u_th)
    # reset fired neurons, decay the rest by beta  [eq. 4]
    H_t = V_reset * S_t + (beta * U_t) * (1 - S_t)
    return S_t, H_t

# TCSA layer at time t, paper form  [eq. 8-9] [eq. 10-14]
# Input:  X_all (T, C, H, W) conv output over all steps; H_prev (C, H, W)
def tcsa_membrane(X_all, t, H_prev):
    # temporal weight for step t, pooled over channel and space  [eq. 8-9]
    X_ta = temporal_gate(X_all)[t] * X_all[t]
    # channel gate on the membrane input H + X_ta  [eq. 10-11] [eq. 14]
    pre = H_prev + X_ta
    U_ca = channel_gate(pre) * pre
    # spatial gate on the channel-refined membrane  [eq. 12-13] [eq. 14]
    U_t = spatial_gate(U_ca) * U_ca
    return U_t

# Att-Res-SNN-1 block output  [eq. 15] [§3.4]
# Input:  U_ori (C, H, W) residual branch output; U_prev (C, H, W)
def att_res_block_1(U_ori, U_prev):
    # channel then spatial refinement of the residual branch  [eq. 15]
    U_ca = channel_gate(U_ori) * U_ori
    U_csa = spatial_gate(U_ca) * U_ca
    # add the membrane shortcut from two layers back  [eq. 15]
    return U_csa + U_prev

Hyperparameters

Values stated in the paper body. Anything not listed is marked "not stated" below; the supplementary material (S2.1 to S2.3, Tables S1 and S2) is not in the arXiv source we have.

Results

Author-reported. Setup: DVS128 Gesture (11 classes) and DVS128 Gait; ImageNet-1K (1,000 classes); baselines re-implemented in PyTorch with the same training scheme §5. Energy ratio r_EE is relative to each vanilla baseline §4.

Event-based (Gesture, 3-layer and 5-layer vanilla baselines) tab. 1

Published prior results in the same table include Amir et al. 2017 (92.59), Shrestha et al. 2019 (93.64), Zheng et al. 2021 (96.87) and Fang et al. 2021 (97.92) on Gesture; these are not re-run tab. 1.

ImageNet-1K, direct-training and ANN comparison tab. 2

The authors state 77.08% is the best SNN result in their comparison, above the 75.32% of a 2048-step ANN-to-SNN baseline §5.2.2 tab. 2.

ImageNet-1K, single-step T=1 controlled comparison (224×224) tab. 3

The authors note the same CSA module adds +0.7 points on CBAM's CNN ResNet-34 versus +5.0 on the SNN §5.2.2 tab. 3 caption.

Attention location, three-layer Gesture (dt×T 15×60) tab. 4

Attention modules and dimension combinations, three-layer Gesture tab. 5 tab. 6

Residual placement, T=1, ImageNet-1K tab. 7

Figure-based analysis (average spiking maps over the validation set) is described in §7.3 and fig. 7 fig. 8. The authors report that attention mainly suppresses background channels and lowers spiking, and that on ImageNet some early-stage channels in CSA-Res-SNN never fire §7.3 fig. 8.

Paper vs code

Official code: BICLab/Attention-SNN at code_commit, MIT License (copyright line names an individual, xyz837). The paper's arXiv listing gives no code link. We found the repository with a GitHub search for "attention spiking" inside the BICLab organisation. Its description reads "Offical implementation of Attention Spiking Neural Networks", and the README links the paper's IEEE page. The paper lists Guoqi Li as corresponding author and gives no lab name. We did not verify that BICLab is Guoqi Li's lab, so the licence basis here is the repository itself, not an author check. Confirm before publishing. We did not use MA-SNN/MA-SNN, CHNtao/MA-SNN or other owners returned by the "MA-SNN" name search. The repo contains MA_SNN/ (conv SNN for Gesture, Gait, CIFAR10-DVS) and Att_Res_SNN/ (ImageNet). It has no ablation scripts and no SM files.

Discrepancies (each with both sides):

1. Where attention acts. The paper recommends CA and SA on the integrated membrane potential (Activate-PRE) and TA on the convolution output §6.1 tab. 4 fig. 4. The code applies all attention to the BatchNorm'd convolution output, before the spiking neuron integrates it: the conv2d, BN and attention calls sit in LIF_Module.py and the fully connected layers do the same at LIF_Module.py. Only the spiking step is applied after attention.

2. TCSA formula. The paper's channel gate uses the membrane input H^{t−1} + X_TA eq. 14. The code's TCSA is TA, then CA on the TA output, then SA, then ReLU, with no membrane term Attention.py. The ReLU is not in eq. 14. It clamps negative attention output before the spiking step, so negative BN values are zeroed in code but not in the paper's equations.

3. Channel-attention pooling. The paper pools CA over space at each time step, one gate per step, and shares the MLP across steps eq. 10. The code pools over time, height and width together, giving one gate per sample and channel, broadcast over time Attention.py.

4. Spatial-attention kernel and pooling. The paper uses a 7×7 kernel eq. 12. The code's SpatialAttention defaults to kernel 3, and TCSA, CSA, SA and TSA call it without the argument Attention.py Attention.py. The paper pools channels for each time step; the code pools over the flattened time-channel axis Attention.py.

5. Membrane update. The paper resets fired neurons to V_reset and decays only non-firing neurons by β eq. 4. The code resets fired neurons to Vthres, not to Vreset (which is stored but unused), and multiplies every neuron's membrane by alpha after the reset, with an additive beta LIF.py LIF.py. Config values are alpha 0.3, Vthres 0.3, Vreset 0, beta 0 Config.py. The paper's β = exp(−dt/τ) is not used.

6. Surrogate gradient. Forward is a Heaviside step. The backward pass is a rectangle of width lens = a/2 = 0.25 around the threshold, height 1/(2·lens) = 2 Att_SNN.py Config.py. The paper does not state a surrogate shape in the main text.

7. Time window in the defaults. The paper's Gesture and Gait results use dt×T = 15×60 tab. 1. The Gesture config defaults to dt = 25, T = 60 Config.py. The Gait config defaults to dt = 500, T = 8 Config.py. The paper does not say which settings produced which row, so do not assume the defaults reproduce the table.

8. Attention off by default. The Gesture config sets attention = "no" Config.py. The README tells the reader to run Att_SNN_CNN.py, but the file is Att_SNN.py Att_SNN.py. Running it as is trains the vanilla model, so set the attention flag to TCSA to reproduce the paper.

9. Optimizer and schedule (event tasks). The code uses Adam, lr 1e-4, betas (0.9, 0.999), batch 128, 300 epochs (Gesture) or 100 epochs (Gait), and ReduceLROnPlateau with factor 0.1 and patience 5 Config.py Config.py Att_SNN.py. The paper says only "follow [yao_2021_TASNN]" with details in S2.1.

10. ImageNet training. The code trains with SGD, momentum 0.9, weight decay 1e-5, batch size 100 by default, initial lr 0.1 by default, and a cosine schedule over 250 epochs train_amp.py train_amp.py global_settings.py. The paper defers the ImageNet training details to S2.2, which is not in the source we have.

11. ImageNet neuron and attention settings. The residual model uses lens 0.5, decay 0.25 and time_window 1 resnet_2.py. It places CSA with c_ratio=8 and t_ratio=1 inside the residual branch, before the identity shortcut is added resnet_2.py. This matches Att-Res-SNN-1 eq. 15, but the ratios are not in the paper.

12. Inconsistent numbers in the paper. The text gives a three-layer Gesture vanilla NASAR of 0.214 §5.1.2; Table 1 gives 0.173 and the ablation tables give 0.173 or 0.174 tab. 1 tab. 4 tab. 5. Treat 0.173 as the value in the tables.

13. Unsupported proof steps. For Att-Res-SNN-1 the printed product §7.1 simplifies to 1/16 + c·ε²/256 for the attention block, which equals 1 only when c·ε² = 240. The paper says this holds "when ε, γ and p are set appropriately" without giving them. The code has no such analysis.

Pitfalls

• Location matters more than the paper's equations suggest. The code's attention runs before integration and adds a ReLU. Reproducing the paper's Activate-PRE result requires moving CA and SA onto the membrane potential, a change the paper describes but the code does not implement.
• Energy numbers are operation-count estimates. NASAR and r_EE come from a formula with fixed E_MAC and E_AC §4, not from hardware. The attention overhead on the first encoder layer is counted as MACs. Do not read the ratios as measured power.
• SimAM's added MACs are large. SimAM adds no parameters but +25.6M ΔMAC on the three-layer Gesture model tab. 5. Parameter count alone understates its cost.
• Single-step ImageNet results use a different protocol. The 1-step rows are pre-training for multi-step SNNs §5.2.2 and use T = 1 tab. 3. Compare them only to other single-step numbers.
• ResNet-104 enlarges inputs to 288×288 at inference* tab. 2 caption, so its numbers are not on the 224×224 protocol used in tab. 3.
• Dataset loading. The README expects the DVS128 Gesture archive in a specific folder and a MATLAB-processed CIFAR10-DVS. Paths are relative to the repo layout; check the data folders before a run.
• Hyperparameters live in the supplement. τ, u_th, V_reset, r_c, r_t, the ImageNet optimizer and the Gait schedule are outside the arXiv main text. The repo gives values for some of them (see Paper vs code); other values are "not stated".
• Author-reported, single-team results. The re-implemented baselines are the authors' own. Gesture rows report mean ± std over 10 runs tab. 1; the ImageNet rows report one run each.

Lineage

• Builds on: Multi-dimensional attention is a CBAM-style module (channel and spatial attention) §3.3 §2 and a temporal attention SNN for event streams by the first author and colleagues (Yao et al., ICCV 2021; the paper's baseline, §3.3 and tab. 1).
• Residual backbone: MS-Res-SNN (Hu et al. 2021), chosen for its membrane shortcut and higher accuracy §3.4 §6.4.
• Also compared: vanilla Res-SNN (Zheng et al. 2021) and SEW-Res-SNN (Fang et al. 2021) §3.4 §6.4.
• Training stability: the block dynamical isometry analysis of Chen et al. 2020 §7.1.
• Superseded by: not known from content on this wiki.