DelRec: learning delays in recurrent spiking neural networks

In plain words

Spiking neural networks pass information as short pulses, and a recurrent one feeds its pulses back into itself. In real brains, the time a pulse needs to travel along a connection is not fixed. This paper gives every recurrent connection a travel time, a delay, that the network learns together with the connection's strength.

Training needs gradients, but a pulse is all-or-nothing. The method uses a smooth stand-in gradient for the firing step, the usual trick for spiking networks. A learned delay can fall between two time steps, so each pulse is shared between the two neighbouring steps in proportion to where the delay lies. Over training that sharing narrows, so the final delays are close to whole steps. Whole-step delays are what neuromorphic chips with programmable delays can run.

The reported payoff is accuracy on several temporal benchmarks using ordinary neurons. The authors also report that learned delays reduce memory and spike counts at the same accuracy.

TL;DR

• Adds a trainable delay to each recurrent connection of a spiking RNN and trains it with surrogate-gradient backpropagation, jointly with the weights.
• Works with any spiking neuron model; the paper uses leaky integrate-and-fire (LIF) neurons throughout.
• Author-reported test accuracy: 82.42% on SSC (5 seeds), 96.63% on PS-MNIST (5 seeds), 85.74% on AL (5 seeds). On HAR, 83.53% is below the two best baselines.
• Official MIT-licensed code exists. The Mackey-Glass delay learning rate differs between paper and code, and some reported numbers are selected on the test split (see Pitfalls).

Problem

Feedforward delays can be learned with surrogate gradients, but recurrent delays had been studied little §1. The one earlier recurrent-delay method used EventProp, which the authors say scales poorly and underperforms on real benchmarks §1. The obstacle for recurrent delays is in the simulation loop: standard SNN simulators update each layer over time and apply synapses and delays afterwards, so a delayed recurrent connection needs extra work inside every time step [supp. Difficulty and benefit of recurrent delays in SNN simulators]. Without a fixed maximum delay, the scheduling buffer can grow with the largest delay §4.1.

Method

Neuron model. Each neuron charges its membrane potential, fires when it crosses a threshold, and resets with either a hard or a soft reset §2.1 eq. 1 eq. 2 eq. 3. The paper uses a LIF charge function with leak factor 1/\tau §2.1. The threshold is non-differentiable, so training uses a surrogate gradient §2.1.

Recurrent input and delays. A standard recurrent layer feeds each neuron the previous step's weighted spikes §2.1 eq. 4. With delays, a spike from neuron j at step t reaches neuron i at step t+1+d_j §2.1 eq. 5:

X_i^{\text{rec}}[t] = \sum_{j \in (L)} w_{ij}^{\text{rec}}\, S_j[t-(1+d_j)] eq. 5

• Axonal delays: one delay per presynaptic neuron, N parameters per layer §2.1.
• Synaptic delays: one delay per connection, d_{ij}, N^2 parameters per layer §2.1.
• A delay of 0 equals a standard recurrent connection with one step of latency §2.1.

Relaxing to real-valued delays. During training, delays are real numbers. A spike scheduled at fractional time t+1+d is spread over future steps by a triangular kernel of width \sigma centred on 1+d §2.1 eq. 7:

h_{\sigma,d}(\tau) = \max\!\left(0,\; \frac{1+\sigma-|\tau-(1+d)|}{(1+\sigma)^2}\right) eq. 7

The scheduled input is updated at every step for all future offsets \tau §2.1 eq. 8:

X^{\text{rec}}_{i}[t+\tau] \leftarrow X^{\text{rec}}_{i}[t+\tau] + \sum_{j} w^{\text{rec}}_{ij}\, h_{\sigma_{\text{epoch}}, d_j}(\tau)\, S_j[t] eq. 8

The unrelaxed rule would place each weighted spike only at the integer step 1+d_j §2.1 eq. 6. When \sigma reaches 0, the kernel is linear interpolation between the two integer steps nearest to 1+d, so the delay gets a well-defined gradient §2.1.

Annealing the width. \sigma decays each epoch §2.1 eq. 13:

\sigma_{\text{epoch}} \leftarrow \sigma_{\text{init}} \times \text{decay}^{\,100 \times \frac{\text{epoch}}{N_{\text{epochs}}}} eq. 13

• On SSC and PS-MNIST, each neuron also gets a parameter p_i, initialised at 0, that sets its own width §4.4 eq. 14:

h_{\sigma,d_i,p_i}(\tau) = \max\!\left(0,\; \frac{1+2\,\text{sig}(p_i)\,\sigma-|\tau-(1+d_i)|}{(1+2\,\text{sig}(p_i)\,\sigma)^2}\right) eq. 14

• On AL and HAR, \sigma_{\text{init}} is fixed at 0, because the authors found a positive value did not help §4.4.

Bounded buffer. The kernel has finite support, [d-\sigma;\, 2+d+\sigma] §4.1 eq. 10, so only a window of future steps has to be scheduled §4.1:

\tilde{\mathbf{E}}(\sigma, D) = \big[\,0;\ \lceil 1 + \max_j d_j + (1+\sigma) \rceil\,\big] eq. 11

The buffer is a circular array with a pointer §4.2 alg. 1. Delays are clamped to stay non-negative §4.1.

Pseudocode. The forward pass follows the paper's Algorithm 1 alg. 1. The sketch below is written from the equations above; it is not a line-by-line copy of the paper's listing.

# Delayed recurrent LIF layer, forward pass  [alg. 1]
# Input:  x_seq (T, B, N), weights (N, N), delays (N,) real, sigma scalar
# Output: spikes (T, B, N)

def delayed_recurrent_forward(x_seq, weights, delays, sigma):
    # Number of future slots the spread can reach  [eq. 11]
    slots = ceil(1 + max(delays) + (1 + sigma))
    # Triangular spread of every delay over those slots  [eq. 7]
    spread = triangle_spread(delays, sigma, slots)       # (N, slots)
    # Circular buffer of scheduled recurrent input  [alg. 1]
    buffer = zeros(B, N, slots)
    membrane = zeros(B, N)
    pointer = 0
    spikes = []
    for x_t in x_seq:
        # Recurrent input due at this step  [eq. 5]
        rec_now = buffer[:, :, pointer]                  # (B, N)
        # Leaky integration, threshold and reset  [eq. 1] [eq. 2] [eq. 3]
        membrane, spike = lif_step(membrane, x_t + rec_now)
        # Free the slot that was just read  [alg. 1]
        buffer[:, :, pointer] = 0
        # Spread each spike over the future slots  [eq. 8]
        for offset in range(1, slots):
            contribution = (spike * spread[:, offset]) @ weights.T
            buffer[:, :, (pointer + offset) % slots] += contribution
        # Move the pointer one step forward  [alg. 1]
        pointer = (pointer + 1) % slots
        spikes.append(spike)
    return stack(spikes)


# Triangular spread of each delay over future slots  [eq. 7]
def triangle_spread(delays, sigma, slots):
    # Offsets tau = 0 .. slots-1, one row per delay  [eq. 8]
    taus = arange(slots)                                  # (slots,)
    distance = abs(taus - (1 + delays[:, None]))          # (N, slots)
    # Keep the positive part, scaled by the squared width  [eq. 7]
    width = 1 + sigma
    return maximum(0, (width - distance) / width ** 2)

Loss and readout. A linear readout sums the readout membrane voltage over time for SSC and PS-MNIST and averages it over time for AL and HAR, then applies cross-entropy §4.4 eq. 12. Mackey-Glass uses mean-squared error on a leaky-integrator readout §4.3.

Inference. Learned delays are rounded to the nearest integer, and \sigma is set to 0 §4.4, "Delays at inference". AL is the exception: it keeps fractional delays, so each connection is a weighted mix of its two nearest integer lags §4.4. Integer delays are what neuromorphic hardware implements [supp. Effect of delay rounding]. The authors state that rounding has little effect on SSC, PS-MNIST and HAR, and a large effect on AL §4.4 [supp. Effect of delay rounding].

Gradient probe (analysis only). To measure how far error signals travel back in time, the authors back-propagate the cross-entropy of the final time step alone and log the input-gradient norm at each past step §4.5 eq. 15 eq. 16. The standard RSNN in this comparison uses a lower weight learning rate, 2\times10^{-4} instead of 10^{-3}, to train stably §4.5.

Hyperparameters

Benchmark settings from the paper's tables [supp. Hyperparameters]. "Half-normal" means |\mathcal N(0, s_{\text{init}}^2)|. Rows marked "code" come from the repository and are compared in Paper vs code.

Other settings from the paper's text and tables:

• Shared settings for all four benchmarks: plain LIF neurons, fully connected recurrent hidden layers, no normalisation and no data augmentation §4.4. Two optimizers are used, one for weights and one for delays, and the delay optimizer has no weight decay [supp. Hyperparameters].
• Mackey-Glass §4.3: \beta=0.2, \gamma=0.1, n=10, x_0=1.2, forward Euler with dt=0.1, sampled at unit steps. The first 5000 steps are discarded, leaving 6000 points split 60/20/20 in time order. \tau\in\{17,30,40,50\} and H\in\{20,30,40,50\}, three seeds each. Windows of W=150 steps; one hidden layer of 128 LIF neurons; batch 512; 100 epochs; delays uniform on [0, 20]; \sigma_\text{init}=10 decayed by 0.95 per epoch; weights with Adam at learning rate 5\times10^{-4} and weight decay 10^{-4}; delays with learning rate 10^{-1} (see Paper vs code for the value in the code); cosine schedules; delays rounded to integers each epoch, best validation checkpoint kept, and evaluated at \sigma=0. The readout is a leaky integrator with time constant 20. Hidden neurons use \tau=2, V_\text{threshold}=1, V_\text{reset}=0 and an arctan surrogate with \alpha=5 §4.3. The reset type is not stated.
• Datasets [supp. Datasets]: SSC has 35 classes, 140 input channels (every five of 700 binned), T=250 steps of 5.6 ms. PS-MNIST flattens each 28\times28 image into 784 steps with a fixed pixel permutation. AL has 50k train and 5k test samples of 400 steps, with a binary label. HAR has 26,048 train and 6,512 test samples of 200 steps, 18 classes, from smartwatch gyroscope data.
• Cross-entropy and per-benchmark readout §4.4 eq. 12 as above.

Results

All numbers are author-reported. Each setup is named in the rows.

Table 1: SSC and PS-MNIST tab. 1. Test accuracy, mean ± std over the seeds shown. A single value is one run. DelRec rows use the best delay configuration.

Table 2: AL and HAR tab. 2. Baselines are from the benchmark paper of Chen et al. (single runs, same splits and protocol), all at about 100k parameters. DelRec: mean ± std over 5 seeds (AL) and 3 seeds (HAR).

Table 3: learned vs fixed delays tab. 3. Same architecture, hyperparameters, preprocessing and seeds in each dataset; only the delay configuration differs. Mean ± std over 5 seeds (3 for HAR). Best per dataset in bold in the paper.

Welch t-tests on Table 3 [supp. Statistical significance]: learned minus fixed is significant at p<0.05 in six of eight cases. Exceptions are synaptic SSC (+0.15, p=0.275) and synaptic AL (+1.27, p=0.101). Axonal vs synaptic learned differs on PS-MNIST (-0.31, p=0.020) and HAR (-0.91, p=0.013), so neither delay type wins everywhere.

SHD (appendix only). The authors exclude SHD from the main table because they consider it saturated and lacking a validation split [supp. The case of the SHD dataset]. Their recurrent-delay model reports 93.39 ± 0.45% with 0.17M parameters, trained with augmentation [supp. Results on SHD].

Mackey-Glass forecasting fig. 3. Across all tested (\tau, H) conditions, the full model has lower NMSE than the three baselines (no delays, fixed delays, learned delays without σ-annealing). Relative gains are shown as heat maps. The paper gives no single headline number for this task; per-condition numbers are in the supplement [supp. NMSE values across all τ and H].

Efficiency and robustness (HAR; no retraining for the robustness sweeps) §2.4:

• Memory: at matched accuracy, learned delays reach each accuracy threshold with a shorter maximum delay than the fixed baseline fig. 7c, so a shallower buffer is needed.
• Energy: with a spike penalty, learned synaptic delays need a lower firing rate to hit each accuracy level than fixed synaptic delays do. For axonal delays the two curves are indistinguishable fig. 7e.
• Robustness: learned delays keep an accuracy advantage under input subsampling and deletion fig. 8a,b. Under temporal jitter, the advantage holds for axonal delays but reverses for synaptic delays fig. 8c,d.

Structure learned by the network §2.3 fig. 4 fig. 5 fig. 6. Learned delays move away from their initialisation with depth: the first layer keeps the widest spread, and most delays are short, with a sparse tail of long ones fig. 5a. Permuting either the longest or the shortest delays in layers 1 and 2 degrades accuracy, and learned models degrade more steeply than fixed ones fig. 5b,c. Gradient maps show longer-range sensitivity for learned delays than for fixed random delays fig. 6e.

Paper vs code

Repository: https://github.com/alexmaxad/DelRec, MIT License, copyright the four authors, at commit dfce0992cb660630065b2444d93aa62d24c29efa. The paper links the repository §1.

• Mackey-Glass delay learning rate. Paper: 10^{-1} §4.3. Config: lr_positions = 5e-2 perf_MG.py. A sweep log shows lr_pos: 0.050000 parallel_sweep_MG.py. The other MG values in the config match the Methods text: \sigma_\text{init}=10 and decay 0.95 perf_MG.py, learning rate 5\times10^{-4} and weight decay 10^{-4} perf_MG.py, 100 epochs and batch 512 perf_MG.py.
• Sigma schedule. Code follows Eq. 13 for recurrent delays: \sigma \leftarrow \sigma_\text{init}\cdot \text{decay}^{100\cdot\text{epoch}/N} delay_layers.py. The triangular kernel matches Eq. 7 delay_layers_pytorch.py. The SSC config matches the table: \sigma_\text{init}=1, decay 0.95, s_\text{init}=12, rounding each epoch perf_SSC.py.
• Evaluation sets σ=0. The test pass zeroes σ on the best validation checkpoint train.py.
• Buffer length. The code's window is \lceil 1 + \max d + s_\text{max}\rceil + 1, with one spare slot beyond Eq. 11 delay_layers_pytorch.py. The extra slot is harmless but not in the paper.
• Rounding path. The PyTorch layer has a rounding path that uses the rounded value in the forward pass and passes the gradient to the fractional delay delay_layers_pytorch.py delay_layers_pytorch.py. The appendix describes an in-place projection of delays at each epoch and at test [supp. Effect of delay rounding]. I did not trace which path the training scripts use, so the paper's and the code's rounding behaviour may differ.
• Positivity. Delays are clamped at zero after initialisation and after updates delay_layers.py delay_layers.py.
• Loss. Code sums the readout over time for SSC and PS-MNIST, as the paper states utils.py. The Mackey-Glass loss is MSE on the readout utils.py.
• Triton kernels. The paper says the fused Triton kernels reproduce the PyTorch forward values and gradients [supp. Computational overhead]. I did not verify this. The PyTorch reference is multi_step_forward_v2, described in the file as the canonical version delay_layers_pytorch.py.

Pitfalls

• Test-set selection on HAR. The HAR result is taken from the sweep over the delay-spread initialisation s_\text{init}, choosing the value that maximises test accuracy, and runs are reported by their final test accuracy §4.4 §4.8. The HAR and AL numbers are therefore optimistic. AL and HAR have no validation split, so AL reports the accuracy of the best checkpoint over the training epochs, following the benchmark paper §4.4. With no validation split, that checkpoint is chosen on the test data. SSC and PS-MNIST select a validation checkpoint train.py.
• Two DelRec HAR numbers. Table 2 reports the axonal learned configuration at 83.53% tab. 2. The best learned configuration in Table 3 is synaptic, at 84.44% tab. 3. The Methods say HAR scores use the best configuration per model type §4.4. Treat the 83.53% as the axonal number, not the best one.
• σ is never exactly zero during training. Eq. 13 gives \sigma_\text{init}\times 0.95^{100} at the last epoch, about 0.0059\,\sigma_\text{init} (roughly 0.0059 for SSC, 0.059 for Mackey-Glass). The Methods describe σ as annealed down to zero §4.4, and §2.1 says the kernel ends as linear interpolation. Evaluation then uses σ=0, which the models were not trained with train.py.
• Decay disagreement on PS-MNIST. The Methods give a decay of 0.95 §4.4, but the PS-MNIST table and config use 0.8 [supp. Hyperparameters] perf_PSMNIST.py. The code matches the table.
• Mackey-Glass delay learning rate. Methods say 10^{-1}; the config and logs use 5\times10^{-2} (see Paper vs code). Re-running with the wrong value will not reproduce the figures.
• Integer delays on AL. AL keeps fractional delays at inference, which needs two taps per connection. The appendix states that rounding costs AL a large amount of accuracy, and that the paper's AL numbers were not rounded [supp. Effect of delay rounding]. A hardware target with only integer delays on AL should expect a drop. The appendix's sentence on how much rounding-during-training recovers ("brings the loss down to 3.2 and 1.4 points") is ambiguous, so do not rely on it.
• Synaptic delays are not uniformly better. Synaptic learned delays give no significant gain on SSC or AL in Table 3, and they are more sensitive than fixed synaptic delays to temporal jitter fig. 8d.
• Shared delay per layer does not learn. The supplement's single-delay-per-layer variant barely moves from its initial value on HAR and stays below the fixed families [supp. Comparison with a single delay per layer]. The gain comes from per-neuron or per-connection heterogeneity, not from having a tunable lag.
• Memory and compute scale with the delay. The buffer length scales with the largest delay, and the cost is O(BSL) in the buffer length L [supp. Computational overhead]. In PyTorch, autograd keeps one copy of the buffer per step unless a fused kernel recomputes it. The reference PyTorch path stores more than the fused kernels do [supp. Computational overhead].
• Baseline comparability. Most baselines in Tables 1 and 2 are single runs, so statistical tests against them are not possible [supp. Statistical significance]. The paper says only the AL/HAR comparison runs in one shared pipeline §2.2; it does not say the SSC/PS-MNIST baselines do.
• Borrowed recipe. The SSC and PS-MNIST implementations build on the codebase of another group §4.4, so their hyperparameters follow that recipe.
• Probe gradients are not comparable to the gradient maps. The gradient-flow probe and the gradient maps use different back-propagated scalars and inputs, and the authors say their magnitudes are not comparable §2.3.

Lineage

• Builds on: feedforward delay learning with annealing (DCLS, Hammouamri et al. 2024) §2.1; the triangular kernel (Khalfaoui et al. 2023) §2.1; surrogate-gradient learning (Neftci et al. 2019) §2.1; the SpikingJelly library §1. None of these has an arXiv link in this wiki yet.
• Compared against: EventProp-based delay learning (Meszaros et al. 2025) tab. 1; ASRC-SNN (Xu et al. 2025) tab. 1; SiLIF, SE-adLIF, GSU and Binary S4D tab. 1 tab. 2.
• Superseded by: not stated.