The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks

In plain words

Spiking neural networks pass information as brief on-off pulses instead of smooth numbers. Good shared test sets for them are rare, and most rely on a conversion from ordinary data to spikes that every researcher does a little differently, which makes results hard to compare. This paper offers two speech datasets that are already in spike form. One is a set of spoken digits the authors recorded themselves. The other is derived from a public collection of spoken words. The spikes come from a model of the inner ear that turns sound into spike trains. The authors show that spike timing carries information: classifiers that only count spikes do poorly, while recurrent networks and LSTMs that see the timing do much better. The authors say the datasets may not prove final.

TL;DR

Two spike-based speech classification datasets, Spiking Heidelberg Digits (SHD) and Spiking Speech Commands (SSC), are produced from audio by a fixed, cochlea-inspired conversion chain. Spike-count classifiers score clearly lower than classifiers that use spike times, so the datasets test temporal processing. All accuracies are author-reported. The conversion code is LGPL-2.1 and is cited here by path only.

Problem

Spiking networks have no shared benchmark. Existing options fail one of the criteria the authors list: unobtainable, too narrow, saturated, or needing heavy preprocessing §1.1. Examples given: the temporal XOR task is saturated §1.2; Neuromorphic MNIST is based on MNIST and nearing saturation §1.2; DASDIGITS depends on TIDIGITS, whose licence for derived data is unclear §1.2; DVS128 Gesture is a continuous stream that needs heavy cutting and downsampling §1.2. Converting MNIST to spikes leaves the encoding to each modeller, which harms comparability §1.1. The authors also assume an ideal benchmark is unsaturated, needs minimal preprocessing, is general, easy to obtain and free to use §1.1.

The authors chose audio because it has a natural time axis and is computationally cheap compared with video §1.2, §4. The conversion is fixed and published, so users need not pick their own encoder §2.2, §4.

Method

Source datasets §2.1

• Heidelberg Digits (HD), recorded by the authors §2.1.1. About 10k recordings of spoken digits zero to nine in English and German, from 12 speakers (six female, six male), aged 21 to 56 (mean 29 (9) years, as written). Recordings were made in a sound-shielded room with three microphones at 48 kHz and 24 bit, then hand-selected, cut and mastered to FLAC. Sequences of ten digits were cut with a speaker-dependent gate; the cutting loss aimed at 10 single-digit files per sequence §2.1.1 fig. 1. Digits are balanced per language fig. 1.
• Speech Commands (SC), version 0.02 §2.1.2. 105,829 one-second WAVE files at 16 kHz from 1,864 speakers, released under CC BY 4.0 §2.1.2. The paper evaluates top-one accuracy over 35 classes. It lists 24 command words and 10 auxiliary words §2.1.2. Splits follow a hashing function described in the Speech Commands paper §2.1.2.
• Split for HD §2.1.1. Two speakers are held out completely for the test set, so speaker generalisation can be measured. The rest of the test set is 5% of the trials of the remaining speakers.

Both sources get a 30 ms Hann taper at the start and end §2.1.1, §2.1.2.

Audio-to-spike conversion §2.2 app. A

The model has three stages: a basilar-membrane (BM) filter bank, a population of hair cells (HCs), and one bushy-cell (BC) layer that sharpens phase locking §2.2, fig. 2. Output: 700 channels of spike trains §2.4.1, app. A.

# Audio-to-spike conversion  [§2.2] [app. A]
# Input:  mono waveform, fs samples/s; N_ch = 700; N_HC = 40
# Output: binary spike trains, (N_ch, T)

def wave_to_spikes(wave, num_channels):
    # 1. Scale the signal to an RMS level of 0.3 cm/s  [app. A]
    wave = wave * 0.3 / rms(wave)

    # 2. Taper both ends with a Hann window (30 ms in the paper)  [§2.1.2]
    wave = hann_taper(wave)

    # 3. Basilar membrane: one velocity trace per place  [app. A.1]
    velocity = basilar_membrane(wave, num_channels)  # (N_ch, T)

    # 4. Hair cells: release probability per channel and step  [app. A.2]
    release = hair_cells(velocity, fs)  # (N_ch, T), values in [0, 1]

    # 5. Bushy cells: one LIF neuron per channel, fed by N_HC hair cells
    #    [app. A.3]
    spike_trains = bushy_cells(release, fs)  # (N_ch, T) binary

    return spike_trains

Step 3, basilar membrane app. A.1. The paper closely follows the steps of Sieroka et al. (2006), a shallow-water cochlea model app. A.1. Stiffness, resistance and effective mass enter a closed-form impedance app. A.1. Places are spaced over a 3.5 cm membrane, and the response is computed in the frequency domain app. A.1. The code takes one FFT of the stimulus and one inverse FFT per place basilar_membrane.py basilar_membrane.py.

Step 4, hair cell app. A.2. Each channel has a transmitter pool (free pool q, cleft c, reuptake store w) based on Meddis's model. Permeability k grows with membrane velocity and is zero when velocity plus A is not positive app. A.2. The pool equations are:

# Hair-cell transmitter pool, one channel, Euler step dt = 1 / fs  [app. A.2]
# Equations: dq/dt = y(1 - q) + n w - k q
#            dc/dt = k q - (l + r) c
#            dw/dt = r c - n w
#            release probability per step = h c dt
def hair_cell(velocity, fs):
    dt = 1 / fs
    # Initial pool state, not stated in the paper  [code: hair_cell.py#L52-L56]
    free, cleft, reuptake = initial_state()
    release = zeros(len(velocity))
    for step, v in enumerate(velocity):
        # Membrane permeability, zero unless v + A > 0  [app. A.2]
        k = g * (v + A) / (v + A + B) if v + A > 0 else 0.0
        # Update the three pools together, using the old values  [app. A.2]
        d_free = y * (1 - free) + n * reuptake - k * free
        d_cleft = k * free - (l + r) * cleft
        d_reuptake = r * cleft - n * reuptake
        free = free + dt * d_free
        cleft = cleft + dt * d_cleft
        reuptake = reuptake + dt * d_reuptake
        # Probability that a transmitter quantum causes a spike  [app. A.2]
        release[step] = h * cleft * dt
    return release

The HC refractory period of 1 ms is stated in the paper as a rule that denies any event within 1 ms of a previous one app. A.2. It is applied in the code on the bushy-cell inputs instead (see Paper vs code).

Step 5, bushy cell app. A.3. Each channel has one LIF neuron. Its input is the 40 HC spike trains, with feed-forward weight 0.54/N_HC for each input app. A.3. There are no recurrent connections app. A.3. In contrast to the bushy-cell model cited from Rothman (1993), the paper uses standard LIF neurons app. A.3. Time constants, threshold and reset for the bushy cell are listed in the parameter table tab. 2.

Event-based data format §2.3

The published files are HDF5, one per partition and dataset. Each holds a spikes group with times and units (variable-length arrays), a labels array of digit IDs, and an extra group with speaker, keys and meta_info (gender, age, body height). meta_info is only given for SHD §2.3. The spike-conversion repository does not write this layout (see Paper vs code).

Spiking network models §2.4

• Architectures §2.4.1. Networks use 700 input channels and hidden layers of N = 128 LIF neurons. Both feed-forward (1 to 3 layers) and recurrent networks are tested. The output is a linear readout of leaky integrators that do not spike §2.4.1.
• Neuron and synapse §2.4.2. Leaky integrate-and-fire membrane with exponentially decaying synaptic current. Spikes are a threshold crossing. After a spike the membrane resets to the leak potential. In discrete time, with kappa = exp(-dt/tau_syn) and lambda = exp(-dt/tau_mem):

# Discrete LIF update, one step n -> n + 1  [§2.4.2]
# Reset to u_leak = 0 after a spike; threshold u_thres = 1
def lif_step(u, i_syn, spikes_below, spikes_rec):
    # Synaptic current: decays, then receives feed-forward and recurrent
    # input  [§2.4.2, continuous form]
    i_syn = kappa * i_syn + W @ spikes_below + V @ spikes_rec
    # Membrane: leaks, integrates the current, and resets after a spike
    u = lam * u * (1 - spikes_rec) + (1 - lam) * i_syn
    # Spike when the membrane reaches the threshold (Heaviside)  [§2.4.2]
    spikes = heaviside(u - u_thres)
    return u, i_syn, spikes

• Weight initialisation §2.4.3. Weights W and V are drawn uniformly from (-sqrt(k), sqrt(k)), with k = 1 / (number of afferent connections).
• Surrogate gradient §2.4.4. The forward pass keeps the hard threshold. The backward pass replaces its derivative with a fast sigmoid. The paper writes the fast sigmoid as u / (1 + beta |u|), with steepness beta §2.4.4. Training uses backpropagation through time §2.4.
• Loss §2.4.5. Cross-entropy on the readout, with two choices of read-out time for each unit: max-over-time (the time step of the largest membrane value) or last time step. Max-over-time is used for all SNN and LSTM results after the comparison in §3.1 §2.4.5, §3.1.
• Regularisation §2.4.6. Two squared-hinge penalties, each scaled by a strength s. The first (L1, threshold theta_l, the paper's "lower threshold") acts on each neuron's mean spike count over time. The second (L2, threshold theta_u, "upper threshold") acts on the population's spike count per sample §2.4.6.

# Supervised surrogate-gradient step  [§2.4.4] [§2.4.5] [§2.4.6]
# Input:  spike inputs (B, T, N_in); labels (B,)
# Output: updated weights W, V and the scalar loss

def train_step(inputs, labels, params):
    # 1. Forward pass with hard-threshold spikes  [§2.4.2]
    spikes, membrane = run_network(inputs, params)  # (B, T, N)
    # 2. Read out each class unit over time  [§2.4.1]
    readout = run_leaky_readout(spikes, params)  # (B, T, N_class)
    # 3. Max-over-time selection per unit  [§2.4.5]
    times = argmax(readout, axis=time)  # (B, N_class)
    logits = pick(readout, times)  # (B, N_class)
    # 4. Cross-entropy plus the two rate regularisers  [§2.4.5] [§2.4.6]
    loss = cross_entropy(logits, labels)
    loss = loss + lower_rate_penalty(spikes) + upper_rate_penalty(spikes)
    # 5. Backward pass with the fast-sigmoid surrogate  [§2.4.4]
    grads = backward_with_surrogate(loss, params)
    # 6. Adamax update with learning rate eta  [§2.4.4] [§3.1]
    params = adamax_update(params, grads, lr=1e-3)
    return params, loss

Non-spiking baselines app. B

• SVM app. B.1. scikit-learn. Input is the spike count of each of the 700 BCs, standardised to zero mean and unit variance. Kernels: linear, polynomial up to degree 3, RBF. Regularisation and kernel parameters are not stated.
• LSTM app. B.2. Input is the 700 BC spike trains binned to 10 ms. One LSTM layer with 128 cells; dropout 0.2 on the input transform and on the recurrent transform; softmax readout. Adamax optimiser; categorical cross-entropy on the last time step and on the time step of maximal activation. TensorFlow 1.14.0 with Keras 2.3.0; other settings are Keras defaults.
• CNN app. B.3. Input is binned to 10 ms and to 64 spatial units. Then: one 2-D convolution, 32 filters of 11x11, ReLU; three blocks, each with two 3x3 convolutions of 32 filters, each followed by batch normalisation and ReLU; then 2x2 max-pooling and dropout 0.2; a dense layer of 128 units with ReLU; softmax readout. Adamax, categorical cross-entropy, Keras defaults otherwise app. B.3. The spatial binning method is not stated.

Hyperparameters

Results

Author-reported. Setup: SHD and SSC with the splits above, using the networks and baselines described in the Method. The paper states that its error measures are standard deviations over 10 experiments §2; the number in parentheses is that spread, in the paper's notation tab. 1. Baselines are the SVM, LSTM and CNN of app. B on spike counts or binned spikes. The paper's own headline numbers:

Other findings:

• Spike timing matters §3, fig. 3. Linear SVMs on spike counts stay near 55% on SHD, and the best SVM, RBF, reaches about 60%. LSTMs and CNNs on binned spikes do much better. Count-based linear SVMs overfit on SHD. The paper concludes that spike counts are not enough for high accuracy on either set.
• Loss choice §3.1, fig. 4. On SHD, only the LSTM generalised well with the last-time-step loss. SNNs and LSTMs both did better with max-over-time. LSTM validation accuracy rose from 95.4 (17) to 97.2 (09) when max-over-time was used §3.1.
• Recurrence §3.1, fig. 6. Recurrent SNNs beat feed-forward SNNs on both sets. On SSC, adding layers to the feed-forward net increases accuracy monotonically. On SHD it does not §3.1.
• Speaker generalisation §3.2, fig. 7. Accuracy drops for speakers four and five, who are only in the test set. The drop is about 18% for the linear SVM, 26% for the RBF SVM, 8% for CNN, 10% for LSTM, 24 to 27% for feed-forward SNNs and 21% for the recurrent SNN §3.2.
• Cross-dataset §3.2, fig. 7. Networks trained on SSC and tested on the English digits of SHD generalise better than networks trained on SHD and tested on SSC. All trained-on-SHD networks stay above chance on SSC §3.2.
• Improvements for the recurrent SNN §3.3, fig. 8. Width 1024 gives 76.5 (10). Channel merge to 70 gives 72.2 (16). Time-constant scale 4 gives 79.9 (28). Input jitter with sigma 20 gives 78.7 (22). Combined, they give 83.2 (13) §3.3, tab. 1.

Paper vs code

The conversion code is in the electronicvisions/lauscher repository, linked from the paper's footnote. Its licence notice names two of the paper's authors (Cramer, Stradmann) and a third contributor Readme.md. It is at commit 16db85ff73252f36f2a8710afa32b0bee155d83b. It is LGPL-2.1, so it is cited by path and line only, with no verbatim code. The repository contains the conversion only. The classifiers, the training code and the dataset writer are not in it. Differences from the paper:

• Hann window length. Paper: 30 ms windows at the start and end of each signal §2.1.1, §2.1.2. Code: the conversion chain calls the Hann taper with its default ramps, which span half the track on each side wave2spike.py window.py. The 30 ms length is not set anywhere in the chain.
• HC parameter values. Paper table: y 11.11, g 1000, l 1250, r 16667, reprocessing n 250 tab. 2. Code defaults: y 5.05, g 2000, l 2500, r 6580, and the reprocessing rate is named x with value 66.3 hair_cell.py hair_cell.py. The paper's input scaling c (1.0) has no counterpart in the code; the code uses velocity directly hair_cell.py.
• HC mass term. The paper's replenishment term is y(1 - q) app. A.2. The code uses an extra parameter m with default 1.0 and sets replenishment to zero once q reaches m hair_cell.py hair_cell.py. m is not in the paper's table.
• HC refractory period. Paper: the 1 ms refractory rule is part of the HC model app. A.2. Code: it is applied to the sampled bushy-cell inputs, after the random draw bushy_cell.py bushy_cell.py.
• BC weight. Paper: 0.54 / N_HC app. A.3. Code: 13e3 divided by n_convergence (40) bushy_cell.py bushy_cell.py. The units differ and the paper gives none, so the two values are not directly comparable.
• BM stiffness constant. Paper table: Greenwood constant a = 35 kg/(s^2 cm^2) tab. 2. Code default: 3500 basilar_membrane.py. The code's C0 is 10e8 basilar_membrane.py, which matches the paper's 1e9 g/(s^2 cm^2). In those same units, 35 kg is 35,000 g, so the code value is ten times smaller. The paper does not explain this.
• BM channel grid. Paper: places in (0, 3.5 cm]. Code: a linear grid that includes 0 basilar_membrane.py.
• Output sample rate. The BM stage tags its output with fs // 2 and uses N // 2 + 1 frequency bins basilar_membrane.py. Later stages therefore run at half the input rate. The paper does not state the conversion's time base.
• Randomness. Bushy-cell inputs use np.random.rand with no seed bushy_cell.py. Repeated runs give different spike trains.
• Output format. Paper: HDF5 with times, units, labels, extra §2.3. Code: SpikeTrain.export writes one NumPy .npz array with rows for spike times and channel indices, and no labels, speakers or metadata spike_train.py spike_train.py. Labels and metadata must come from elsewhere.
• SNN, LSTM and CNN training code. Not in the repository. The paper calls its SNN code custom PyTorch and points to an external tutorial for the surrogate-gradient method §2.4.4. The tutorial is not the code used for the reported runs.
• Discrete synapse equation. The discrete form in the paper uses the same layer's spikes for the feed-forward input, while the continuous form uses the layer below §2.4.2. This card uses the continuous form.

No code for the SNN, LSTM, CNN or SVM experiments was found in the linked repository, so those settings are cited from the paper only.

Pitfalls

• Units. The BM parameters mix cgs units and SI units without a conversion rule tab. 2 app. A.1. The bushy-cell weight has no unit app. A.3. Use the paper's values only after a unit check, and expect the Greenwood-constant mismatch above.
• Surrogate steepness conflicts. The text uses beta = 40 for all SNNs unless stated otherwise §3.1. The parameter table lists beta = 100 tab. 2. The paper does not say which was used for the reported SNN results.
• LSTM accuracies conflict. The table gives 89.0 (02) on SHD and 73.0 (01) on SSC tab. 1. The text gives 85.7 (14) and 75.0 (2) for the same setting §3, §3.1.
• Class count. The paper says 35 classes for SC §2.1.2. It names 24 command words and 10 auxiliary words, which is 34. The 35th class is not named.
• Spike-count baselines. The SVM and LSTM settings are not fully given, so the baseline numbers may not be reproducible app. B.1, app. B.2.
• Max-over-time gradient. The paper does not say how gradients flow through the choice of time step §2.4.5.
• Fast sigmoid. The paper writes u / (1 + beta |u|) as the surrogate and says its derivative is used §2.4.4. It is not stated whether the printed expression is the surrogate itself or the function whose derivative is taken.
• Initial HC state. The paper gives no initial pool values. The code starts each pool at a steady state for zero velocity hair_cell.py.
• Regularisation scope. Both penalties are written for layer l. The paper does not say which layers receive them §2.4.6.
• Speaker split size. The number of test recordings is not stated for SHD. Error bars come from 10 runs §2.
• Selection data. The beta and eta grid search uses a validation set [§3.1, fig. 5]. The text reports both validation and test accuracy for the later choices (width, channel merge, time scale, jitter). Which split selected them is not stated §3.3.
• Compute cost. The README warns that short files can take minutes to convert and that memory use can be very large for long files Readme.md. Each channel runs 40 random HC draws per step.
• Paper's own caveat. The authors expect the datasets may not prove final, and that the reported accuracies are lower bounds §4 §3.
• Mixed frameworks. SNNs use PyTorch; LSTM and CNN use TensorFlow 1.14 and Keras 2.3 app. B.2, app. B.3. Reproducing all rows needs both.

Lineage

• Builds on: the shallow-water basilar-membrane model of Sieroka et al. (2006) app. A.1; the transmitter-pool hair-cell model of Meddis (1986, 1988) app. A.2; surrogate-gradient training as described by Neftci et al. (2019) §2.4.4. None of these has a card on wikillm.
• Compared against: linear, polynomial and RBF SVMs; LSTMs; CNNs; feed-forward and recurrent SNNs; and the spiking neural unit (SNU) §3, app. B, §3.3.
• Related datasets named in the paper: N-MNIST, DVS128 Gesture, DASDIGITS and TIDIGITS §1.2, §4.