Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks

In plain words

Spiking networks communicate with sparse on-off pulses, which suits low-power neuromorphic chips. A common shortcut is to train an ordinary network and then convert it into a spiking one. The problem is that a converted network needs many pulse rounds to match the original. With only a few rounds, its accuracy drops sharply.

The authors split the conversion mistakes into three kinds. They then change the activation used while training the ordinary network, so that it behaves more like the spiking version will. The new activation rounds the signal to a fixed number of levels, caps it, and shifts the rounding by half a level. The spiking copy starts each neuron at half its threshold, which matches that shift. With this, the converted network works well with about four pulse rounds.

TL;DR

• Converted spiking networks lose accuracy when they run for few time-steps. The paper attributes the loss to clipping, quantization (flooring) and unevenness error §3.
• The source ANN is trained with a quantized clip-floor-shift activation §4.2, eq. 15. The converted SNN uses threshold equal to the trained λ and initial potential θ/2 app. training configurations.
• With shift 1/2 and initial potential θ/2, the expectation of the estimated conversion error is zero for any T and L, under a uniform-input assumption §4.2, thm. 2.
• Author-reported: VGG-16 on CIFAR-10 reaches 91.18% at T=2 and 93.96% at T=4 tab. 2.
• The official code has no LICENSE file. Its defaults for epochs, learning rate and batch size differ from the paper; see Paper vs code.

Problem

• Converted SNNs need long simulations to match ANN accuracy. The baseline conversion methods the paper compares against need tens to hundreds of time-steps §1.
• The conversion error is nonzero only when the potential change v(T) − v(0) is nonzero and the weighted input z is positive §3, eq. 9. The paper notes that this change is not negligible when T is small §2.
• Clipping error: the SNN output lies in [0, θ], while the ANN activation can be much larger. Values above the chosen scale λ map to the same top level §3, eq. 10.
• Quantization error: the output moves in steps of θ/T, so nearby activations merge §3, fig. 1.
• Unevenness error: the timing of arriving spikes can change the output rate, adding or removing spikes compared with the ANN value. The paper's example uses weights 2 and −2, inputs [0.6, 0.4] and T=5. The same rates give outputs 0.8, 0.4 or 0.2 depending on spike timing §3, fig. 1.
• The estimated output in eq. 11 assumes the residual potential v(T) falls in [0, θ]. That is, it ignores unevenness error §3.

Method

1. Spiking neuron §2, eq. 2–5. Integrate-and-fire with reset by subtraction. The membrane potential before firing is the previous potential plus the weighted input eq. 2. A spike is emitted when the potential reaches θ eq. 3. The potential then drops by θ eq. 4. Each spike reaches the next layer as an input of size θ eq. 5.

2. Rate identity §2, eq. 6–8. Summing eq. 6 over T steps and dividing by T gives the average input φ^l(T) as W·φ^(l−1)(T) minus the change in potential divided by T eq. 8. The paper neglects the residual term when T is long enough §2.

3. Estimated output §3, eq. 11. Assuming v(T) lies in [0, θ], the SNN output is approximately

   \phi^l(T) \approx \theta^l\,\mathrm{clip}\!\left(\frac{1}{T}\left\lfloor\frac{z^l T + v^l(0)}{\theta^l}\right\rfloor,\,0,\,1\right) \qquad [\text{eq. 11}]

   where z^l = W^l·a^(l−1). The estimated conversion error is this minus ReLU(z^l) eq. 12.

4. Quantization clip-floor activation for training §4.1, eq. 13. The ReLU in the ANN is replaced by

   a^l = \lambda^l\,\mathrm{clip}\!\left(\frac{1}{L}\left\lfloor\frac{z^l L}{\lambda^l}\right\rfloor,\,0,\,1\right) \qquad [\text{eq. 13}]

   If T = L, θ = λ and v(0) = 0, the estimated error is exactly zero §4.1, thm. 1.

5. Quantization clip-floor-shift activation §4.2, eq. 15. A shift φ is added inside the floor:

   a^l = \lambda^l\,\mathrm{clip}\!\left(\frac{1}{L}\left\lfloor\frac{z^l L}{\lambda^l} + \varphi\right\rfloor,\,0,\,1\right) \qquad [\text{eq. 15}]

   With θ = λ, v(0) = θφ and φ = 1/2, the expectation of the estimated error is zero for arbitrary T and L §4.2, thm. 2. The proof assumes z is uniformly distributed inside each quantization interval app. proof of Theorem 2. When T ≠ L the error is not zero pointwise; the paper's Figure 2 shows this fig. 2.

6. Training §4.3, eq. 17; alg. 1. Train the ANN from scratch with SGD. The floor uses a straight-through gradient. The derivative with respect to z is 1 for −λ/(2L) < z < λ − λ/(2L) and 0 otherwise. The derivative with respect to λ is (a − z)/λ on −λ/(2L) ≤ z < λ − λ/(2L), 0 below that range and 1 above it eq. 17. Before training, replace max-pooling with average-pooling and each ReLU with the quantized activation alg. 1.

7. Conversion alg. 1; app. training configurations. Copy the trained weights unchanged. Set each layer's threshold θ to its trained λ. Set the initial potential v(0) to θ/2.

8. Inference app. training configurations. Feed the same (constant) input at every time-step and simulate T steps. The paper does not state how the output is read out.

The activation and the spike rule as pseudocode:

# Quantization clip-floor-shift activation  [eq. 15]
# z: pre-activation (B, C, H, W); lam: trainable scalar lambda > 0
# L: quantization steps; phi: shift, 1/2 in the paper's setting  [§4.2]
def qcfs_shift(z, lam, L, phi):
    # Scale to level units and add the shift  [eq. 15]
    scaled = z * L / lam + phi               # (B, C, H, W)
    # Floor to an integer level and clip to the range 0..L  [eq. 15]
    levels = clip(floor(scaled), 0, L)       # (B, C, H, W)
    # Return to the activation scale  [eq. 15]
    return lam * levels / L

# One time step of a spiking layer, reset by subtraction  [eq. 2]–[eq. 5]
# prev_spikes: 0/1 outputs of layer l-1 at this step
# theta_prev: threshold of layer l-1; theta: threshold of this layer
def spiking_layer_step(potential, prev_spikes, weights, theta_prev, theta):
    # Weighted input; each incoming spike is worth theta_prev  [eq. 5]
    input_current = weights @ (prev_spikes * theta_prev)
    # Integrate onto the membrane  [eq. 2]
    membrane = potential + input_current
    # Fire when the membrane reaches the threshold  [eq. 3]
    spikes = membrane >= theta
    # Subtract one threshold per spike and keep the rest  [eq. 4]
    potential = membrane - spikes * theta
    return spikes, potential

# Convert a trained source ANN into a spiking network  [alg. 1]
def convert_to_snn(source_ann):
    # Copy the trained weights unchanged  [alg. 1]
    snn = copy_weights(source_ann)
    for layer in snn.spiking_layers:
        # Threshold equals the trained lambda  [alg. 1]
        layer.theta = layer.lam
        # Start each membrane at half threshold to match the shift 1/2  [§4.2]
        layer.potential_init = layer.theta / 2
    return snn

Hyperparameters

Results

Author-reported. Setup: CIFAR-10, CIFAR-100 and ImageNet classification §6. Source ANNs are VGG-16, ResNet-18 and ResNet-20 for CIFAR, and ResNet-34 and VGG-16 for ImageNet §6. Baselines are RMP, TSC, RTS, RNL and SNNC-AP, plus direct-training methods §6. The table note says RTS and SNNC-AP use a changed ResNet-18, while this method uses the standard one tab. 2.

CIFAR-10 accuracy tab. 2

Hybrid and direct-training methods on CIFAR-10 tab. S3

ImageNet top-1 tab. 3

CIFAR-100 top-1 tab. S4

Quantization-step sweep tab. S2 (accuracy in %, author-reported)

The paper says L=1 could not be trained. With L=2, VGG-16 reaches 86.53% on CIFAR-10 at T=1 and 61.41% on CIFAR-100 at T=1 app. effect of quantization steps.

Dynamic threshold and shift, CIFAR-100 VGG-16, L=4 tab. S1; fig. 4. The source headers are "DT" (trained threshold) and "w/o shift". Here ✓ means the feature is used, which is the reading under which the proposed setting (both ✓) matches the text.

Energy, all rows tab. S5; app. energy consumption. Author-reported. Energy uses 77 fJ per synaptic operation for SNNs and 12.5 pJ per FLOP for ANNs, taken from the ROLLS chip. Memory access is not counted. The ANN baseline is 0.628 GFLOP and 7.85 mJ for VGG-16 on CIFAR-100.

See Pitfalls for the SNNC-AP entry at T=32.

Paper vs code

The code is linked from the abstract, so it counts as the paper's official repository. It is putshua/SNN_conversion_QCFS at code_commit. Its README says the authors moved to putshua/ANN_SNN_QCFS (cloned at commit eca136bd085087567013240ee14fb6159a2b6da7). That newer repository says it fixes "the biggest unreproducable bug" without naming it, and it is cited below as ANN_SNN_QCFS/.... README.md README.md

Neither repository has a LICENSE file. Both were read; no code is quoted.

• Activation and shift. The floor at the paper's shift 1/2 is implemented as floor(x·t + 0.5), then clipped to [0, 1], then scaled by λ modules.py. The newer repository does the clip first and then floors, with the same +0.5 layer.py. The shift is a constant 0.5 in code, not a trained or per-layer quantity, while the paper calls it a vector §4.2.
• λ gradient. The old code trains λ through autograd on the forward pass with a straight-through floor and a clip modules.py modules.py. Eq. 17 is not written out in code. The README says the paper's λ derivative "has a little problem" and that the code is correct through autograd README.md. I did not verify eq. 17 against autograd at the clip boundaries.
• Weight decay on λ. The optimizer's parameter grouping puts every activation's up parameter in the weight-decay group utils.py, and all three groups use the same weight decay funcs.py. So λ is decayed. The paper does not say this.
• Initial potential. The old code reaches θ/2 with one dummy time-step of input 0.5·θ before the first real step modules.py. The newer code sets the membrane to 0.5·θ directly layer.py.
• Spike rule. The paper says a neuron fires when the potential "exceeds" the threshold §2. The newer code fires when the membrane is at or above the threshold, through input >= 0 on m − θ layer.py layer.py.
• Readout. The old SNN evaluation adds the network output at each step and takes the arg-max of the running sum, recording accuracy after each step funcs.py. The paper does not state a readout.
• Initial λ. When ReLU is replaced, the old code starts λ at 8.0 utils.py. The paper does not state an initial value.
• Pooling. Max-pooling is replaced by average-pooling, as the paper says utils.py.
• ShiftNeuron. The old code defines a ShiftNeuron that learns the initial potential from a running average modules.py. The conversion path does not use it; it imports ScaledNeuron utils.py utils.py.
• Training defaults (old). main.py defaults to 120 epochs, batch size 128, L=16 and T=16 main.py. The paper uses 300 epochs and L per network (see Hyperparameters). The default learning rate of 0.1 and weight decay of 5e-4 match the paper's CIFAR-10 value main.py.
• Training defaults (newer). main_train.py defaults to 300 epochs, batch size 300 and L=8 main_train.py. Its learning-rate comment says 0.05 for CIFAR-100 and 0.1 for CIFAR-10 main_train.py. The paper says 0.02 for CIFAR-100 app. training configurations. The newer README also says "need to change lr to 0.05" for CIFAR-100 README.md.
• Augmentation. The paper says all datasets use random crop, Cutout and AutoAugment app. training configurations. The CIFAR-10 pipeline uses AutoAugment (CIFAR10Policy) getdataloader.py. The CIFAR-100 pipeline does not getdataloader.py. Both use horizontal flip, which the paper does not mention. ImageNet training uses RandomResizedCrop(224), not a resize followed by a crop getdataloader.py.
• Newer test script. main_test.py calls model.set_L(8) regardless of the training L main_test.py. This matters only in ANN mode (T=0), where the layer uses L layer.py. It does not affect SNN mode.
• Reported numbers. The newer README reports VGG-16 on CIFAR-10 at 90.94% (T=2), 94.01% (T=4) and 95.01% (T=8). The paper reports 91.18%, 93.96% and 94.95% README.md tab. 2. The README's CIFAR-100 numbers use lr 0.05 and are higher than the paper's, which uses 0.02 README.md.
• Unused imports. The newer layer.py imports cv2 and sympy without using them, so those packages must be installed to run it layer.py.
• Vendored code. The old repository contains a copy of spikingjelly (for IFNode) with no license of its own in this repository modules.py neuron.py. I did not check its upstream license.
• Not checked. I did not run either repository. The newer README's checkpoint is hosted on Google Drive and was not downloaded.

Pitfalls

• Batch size and λ start are not stated. Plan for them. The code's values are 128 or 300 for batch size, and 8.0 for λ main.py main_train.py utils.py.
• Initial potential must be θ/2 in every spiking layer. The shift only works if the SNN starts at half threshold. Missing this breaks the match with the trained activation §4.2 alg. 1.
• Threshold must equal the trained λ, not a data-based maximum. The paper argues that setting θ to the maximum possible input slows firing and makes the quantization error larger app. dynamic threshold. Its Table S1 shows the trained-threshold setting far ahead at short T tab. S1.
• The estimate ignores unevenness error. Even with L = T, a gap remains between ANN and SNN accuracy §7. The shift makes the expectation zero, not each sample §4.2.
• Theorem 2 is an expectation under an assumption. It needs the inputs to be uniform inside each quantization interval app. proof of Theorem 2. Real inputs are not guaranteed to satisfy this.
• Choose L with the target latency in mind. At T = L the estimate is exact. Smaller L helps at short T but reduces capacity: at L=2 VGG-16 on CIFAR-100 loses accuracy at long T (72.40% at T=128 versus 77.05% for L=4) tab. S2 §6.4.
• T=1 is hard. The paper says high performance at T=1 is difficult §7. L=2 gives 86.53% on CIFAR-10 at T=1 tab. S2.
• Appendix equation label. The appendix title says "Derivation of Equation 12", but the equation it derives is labelled snn_est. By counting the displayed equations in the source, that is eq. 11, and eq. 12 is the estimated error app. derivation of the estimate §3.
• Table S1 headers are ambiguous. The column is headed "w/o shift", but the proposed row (dynamic threshold and shift) has check marks in both columns. Read the check as "used" tab. S1.
• Energy table entry for SNNC-AP at T=32 looks inconsistent. It lists 0.857 GSOP and 0.660 mJ. At the stated 77 fJ per operation, 0.857 GSOP is about 0.066 mJ, which fits the T=64 entry (1.22 GSOP, 0.094 mJ). The text says this method costs about twice SNNC-AP at equal T. That holds at T=64 (0.222 vs 0.094 mJ). At T=32 the table gives 0.111 mJ for this method, lower than 0.660; with the arithmetic value of 0.066 mJ it would be about 1.7 times higher tab. S5 app. energy consumption.
• Readout and input coding. Constant input at each step is stated. How the output is accumulated is not app. training configurations. The code sums outputs over time funcs.py.
• Costs. Each image takes T forward simulation steps. The paper's energy estimate excludes memory access, which it says depends on hardware app. energy consumption.
• Max-pooling replacement changes the source network. The paper does not report the accuracy cost of this change for the ANN alg. 1.

Lineage

• Builds on rate-based ANN-SNN conversion with reset-by-subtraction neurons, cited as Cao et al. (2015), Diehl et al. (2015), Rueckauer et al. (2016) and Sengupta et al. (2019) §1 §5.
• The paper says the clipping and quantization error terms were proposed earlier by Li et al. (2021) §3.
• Contrasted with Deng et al. (2020), which adds a bias to shift the converted network curve; this method shifts the ANN activation instead §7.
• Compared against: RMP (Han et al., 2020), TSC (Han et al., 2020), RTS (Deng et al., 2020), RNL (Ding et al., 2021) and SNNC-AP (Li et al., 2021) on CIFAR and ImageNet §6.
• Superseded by: not known from content in this wiki.