ANN-to-SNN conversion

Overview

A spiking network sends sparse on-off pulses, and its accuracy depends on how many time-steps it is simulated for. Converting a trained ANN avoids training the spiking network directly. The problem is that a plain conversion loses accuracy when few time-steps are allowed 2303.04347 §1. The paper splits the loss into three kinds: clipping error, quantization error and unevenness error 2303.04347 §3. The first two were named earlier by Li et al. (2021) 2303.04347 §3.

The invariant the method aims for is this. If the ANN's activation is a quantized clip-floor-shift function with shift 1/2, and the spiking neuron starts at half its threshold, then the expected conversion error is zero for any number of time-steps T and any number of quantization steps L. The paper assumes the input is uniform inside each quantization interval [arxiv:2303.04347 §4.2 thm. 2]. The zero holds in expectation, not for each input. When T is not equal to L, the error is not zero pointwise 2303.04347 fig. 2. The estimate ignores unevenness error 2303.04347 §3.

Variants

Quantized clip-floor activation without shift

The source ANN uses the activation with no shift. With T = L, threshold equal to λ and initial potential 0, the estimated error is exactly zero 2303.04347 §4.1 thm. 1. The shift is added so that the expected error is zero for arbitrary T and L. The paper reports no separate accuracy table for this variant. The closest data are the ablation with the shift removed and the trained threshold kept: at T=4 the accuracy is 21.57%, against 69.62% with the shift, on CIFAR-100 VGG-16 2303.04347 tab. S1.

Quantized clip-floor-shift activation with half-threshold start

This is the proposed method. Its source ANN uses the shifted activation with shift 1/2, trained from scratch with SGD. The converted network copies the weights, sets each threshold to the trained λ, and starts every membrane at θ/2 2303.04347 §4.2 alg. 1. Card: Bu et al., arXiv:2303.04347.

The threshold must be the trained λ. In the paper's ablation, keeping the shift but dropping the trained threshold gives 1.00% at T=4 on CIFAR-100 VGG-16. Its accuracy reaches 77.10% only at T≥512 2303.04347 tab. S1. The paper also argues against setting the threshold to the largest possible input, because it slows firing and enlarges the quantization error [arxiv:2303.04347 app. dynamic threshold].

When to use which

Every row carries the training cost of the paper's setting: 300 epochs 2303.04347 app. training configurations. The quantization steps L also matter. For VGG-16 on CIFAR-10, L=4 is the highest of the three values tested at T=4 to 16, and L=8 is higher from T=32 on 2303.04347 tab. S2. The paper uses L=4 for all CIFAR-10 networks [arxiv:2303.04347 app. training configurations].

Implementation sketch

The training activation, the spiking step, and the conversion, as the paper describes them. Names are ours.

# Training activation: quantized clip-floor-shift  [arxiv:2303.04347 eq. 15]
# z: pre-activation; lam: trainable threshold lambda
# L: number of quantization steps; phi: shift, 1/2  [§4.2]
def qcfs_activation(z, lam, L, phi=0.5):
    # Map to level units and add the half-level shift  [eq. 15]
    s = z * L / lam + phi
    # Floor to an integer level, then clip to the range 0..L  [eq. 15]
    k = clip(floor(s), 0, L)
    # Return to the activation scale  [eq. 15]
    return lam * k / L

# One time-step of a converted layer, reset by subtraction  [eq. 2]–[eq. 5]
# theta_prev: threshold of the previous layer; each spike carries it  [eq. 5]
def snn_step(v, in_spikes, W, theta_prev, theta):
    # Integrate the weighted input onto the potential  [eq. 2]
    v = v + W @ (in_spikes * theta_prev)
    # Spike where the potential reaches the threshold  [eq. 3]
    out = v >= theta
    # Subtract one threshold per spike  [eq. 4]
    v = v - out * theta
    return out, v

# Convert the trained ANN  [alg. 1]
def convert(ann):
    # Copy the trained weights unchanged  [alg. 1]
    snn = copy(ann.weights)
    for layer in snn.layers:
        # Threshold equals the trained lambda of that activation  [alg. 1]
        layer.theta = layer.lam
        # Start at half threshold to match the shift 1/2  [§4.2]
        layer.v0 = layer.theta / 2
    return snn

The paper does not state how the output is read out 2303.04347 app. training configurations. Choose the readout yourself and document it.

Contradictions

Only one paper lists this slug, so no disagreement between papers is known.

Within the paper's own material, the energy row for SNNC-AP at T=32 does not match the stated cost per operation. The card's check finds 0.660 mJ in the table against about 0.066 mJ from the arithmetic 2303.04347 tab. S5. The table's proposed-method rows are the ones used above.

The paper's numbers and its released code differ in several places, such as the CIFAR-100 learning rate (0.02 in the paper, 0.05 in the later README) and the reported CIFAR-10 accuracy at T=2 (91.18% in the paper, 90.94% in the later README). Details are on the card's Paper vs code section.

Papers

• Bu et al., arXiv:2303.04347: shifted quantized activation and half-threshold start; the only card for this slug.