Learnable delays

Overview

A spiking network passes information as spikes. In a recurrent one, a spike reaches the neuron itself or its neighbours after a travel time, the delay. Learnable delays make that time a trainable parameter, learned together with the connection's weight, so the network can shift timing as well as strength 2509.24852 §2.1.

The shared invariant: a delayed spike is scheduled at a real-valued time during training and spread over the neighbouring integer steps by a triangular kernel. The kernel's width is annealed to zero, and the delays are rounded to integers afterwards 2509.24852 §2.1 2509.24852 §4.4.

The obstacle is in the simulator. Standard spiking simulators apply synapses and delays after each layer's update, so a delayed recurrent connection needs extra work inside every time step [arxiv:2509.24852 supp. Difficulty and benefit of recurrent delays in SNN simulators]. Recurrent delays had been studied little. The one earlier recurrent method used EventProp, which the authors say scales poorly 2509.24852 §1. Feedforward delay learning with annealing exists, but is not covered here (see Papers).

Variants

Axonal delays

One delay per presynaptic neuron, so N delays per layer [arxiv:2509.24852 §2.1]. A spike from neuron j reaches all of its targets at the same learned lag. Reported with learned delays: 82.42 ± 0.23% on SSC, 85.74 ± 1.49% on AL and 83.53 ± 0.26% on HAR, all author-reported over several seeds 2509.24852 tab. 2 2509.24852 tab. 3. Card: DelRec.

Synaptic delays

One delay per connection, so N² delays per layer 2509.24852 §2.1. Each weight gets its own lag. Reported with learned delays: 96.63 ± 0.12% on PS-MNIST, 85.00 ± 0.85% on AL and 84.44 ± 0.27% on HAR [arxiv:2509.24852 tab. 1] 2509.24852 tab. 3. Card: DelRec.

Fixed delays (control)

The same architecture with the delays not trained. This is the baseline that separates the effect of having delays from the effect of learning them 2509.24852 tab. 3. Learned minus fixed, author-reported:

Source: 2509.24852 tab. 3. Five seeds, three for HAR. Of the eight learned-versus-fixed comparisons, six are significant at p < 0.05. The two that are not are synaptic SSC (+0.15, p = 0.275) and synaptic AL (+1.27, p = 0.101) 2509.24852 supp. Statistical significance.

Learned delays without σ-annealing (ablation)

The Mackey-Glass forecasting study compares the full model with a version whose kernel width is not annealed. The full model has lower NMSE in every tested condition. The paper gives no single headline number for this task 2509.24852 fig. 3 [arxiv:2509.24852 supp. NMSE values across all τ and H].

Shared delay per layer (negative result)

A single learned delay per layer barely moves from its initial value on HAR and stays below the fixed families. The gain comes from per-neuron or per-connection delays, not from having one tunable lag [arxiv:2509.24852 supp. Comparison with a single delay per layer].

Mechanism details

• Spread kernel. A spike scheduled at 1 + d is shared between future steps by a triangle of width σ. The support is [d − σ, 2 + d + σ], so only a bounded window of future steps must be scheduled 2509.24852 eq. 7 2509.24852 eq. 10. At σ = 0 the kernel is linear interpolation between the two integer steps around 1 + d 2509.24852 §2.1.
• Annealing. σ decays per epoch as σ_init × decay^(100 × epoch / N_epochs) 2509.24852 eq. 13. On SSC and PS-MNIST each neuron also gets a parameter that sets its own width 2509.24852 §4.4 2509.24852 eq. 14. On AL and HAR σ_init is fixed at 0, because the authors found a positive value did not help 2509.24852 §4.4.
• Buffer length. The number of future slots is ⌈1 + max d + (1 + σ)⌉ 2509.24852 eq. 11. It grows with the largest delay, and the per-step cost scales with the buffer length [arxiv:2509.24852 supp. Computational overhead].
• Delay positivity. Delays are clamped to stay non-negative [arxiv:2509.24852 §4.1].

When to use which

Weigh these against the paper's own limits: the rows rest on one paper, and most baselines are single runs 2509.24852 supp. Statistical significance.

Implementation sketch

Pseudocode in our own words, following the forward pass of the card's source 2509.24852 alg. 1. Variants differ only in the shape of delays: (N,) for axonal, (N, N) for synaptic.

# Delayed recurrent LIF layer, forward pass  [arxiv:2509.24852 alg. 1]
# Input:  x_seq (T, B, N), weights (N, N), delays (N,) real, sigma scalar
# Output: spikes (T, B, N)

def delayed_recurrent_forward(x_seq, weights, delays, sigma):
    # Future slots the spread can reach  [arxiv:2509.24852 eq. 11]
    slots = ceil(1 + max(delays) + (1 + sigma))
    # Triangular weight of each delay at each future offset  [eq. 7]
    spread = triangle_spread(delays, sigma, slots)       # (N, slots)
    # Circular buffer of recurrent input still to be delivered
    buffer = zeros(B, N, slots)
    membrane = zeros(B, N)
    pointer = 0
    spikes = []
    for x_t in x_seq:
        # Recurrent input due at this step  [eq. 5]
        rec_now = buffer[:, :, pointer]
        # Leaky integration, threshold and reset  [eq. 1-3]
        membrane, spike = lif_step(membrane, x_t + rec_now)
        # Free the slot that was just read
        buffer[:, :, pointer] = 0
        # Spread each spike over the future slots  [eq. 8]
        for offset in range(1, slots):
            contrib = (spike * spread[:, offset]) @ weights.T
            buffer[:, :, (pointer + offset) % slots] += contrib
        pointer = (pointer + 1) % slots
        spikes.append(spike)
    return stack(spikes)


# Triangular spread of each delay over the future offsets  [eq. 7]
def triangle_spread(delays, sigma, slots):
    taus = arange(slots)                                 # (slots,)
    distance = abs(taus - (1 + delays[:, None]))         # (N, slots)
    width = 1 + sigma
    return maximum(0, (width - distance) / width ** 2)

Training: the threshold has no gradient, so use a surrogate gradient 2509.24852 §2.1. Anneal σ per epoch with eq. 13. At inference, round the delays to integers and set σ = 0 2509.24852 §4.4.

Contradictions

• Mackey-Glass delay learning rate. The Methods give 10⁻¹ for the delay learning rate 2509.24852 §4.3. The paper card records that the repository config and a sweep log use 5 × 10⁻² [code: configs/perf_MG.py#L120]. Both are stated; the paper's figures are not reproduced by the wrong one.
• PS-MNIST σ decay. The Methods give 0.95 2509.24852 §4.4. The PS-MNIST table and config use 0.8 2509.24852 supp. Hyperparameters.
• HAR number for DelRec. Table 2 reports the axonal configuration at 83.53% 2509.24852 tab. 2. Table 3 gives the best learned configuration as synaptic, at 84.44% 2509.24852 tab. 3. The Methods say HAR uses the best configuration per model type 2509.24852 §4.4.
• Rounding in code. The appendix describes an in-place projection of delays each epoch and at test [arxiv:2509.24852 supp. Effect of delay rounding]. The card reports a rounding path in the PyTorch layer that passes the gradient to the fractional delay [code: src/delrec/delay_layers_pytorch.py#L63-L64], and says which path the training scripts use was not traced.

Pitfalls

• Test-set selection on HAR and AL. The HAR result comes from choosing the delay-spread initialisation that maximises test accuracy, and AL reports the best checkpoint over the epochs without a validation split. The authors' HAR and AL numbers are therefore optimistic 2509.24852 §4.4 2509.24852 §4.8.
• σ never reaches zero in training. Eq. 13 leaves about 0.0059 × σ_init at the last epoch on SSC. The Methods describe σ as annealed to zero, yet the evaluation sets σ = 0 for a model that was not trained there [arxiv:2509.24852 §4.4] 2509.24852 eq. 13.
• Integer delays on AL. Keeping fractional delays needs two taps per connection. The appendix's figure for how much rounding-during-training recovers is ambiguous, so do not rely on it [arxiv:2509.24852 supp. Effect of delay rounding].
• Probe gradients are not comparable. The gradient-flow probe and the gradient maps use different back-propagated scalars, and the authors say their magnitudes are not comparable 2509.24852 §2.3.
• Open question in the sketch. The sketch starts the spread at offset 1, as the card's pseudocode does. Whether eq. 8 also writes the current step when d < σ is not settled from the card; check it against the paper before implementing.

Papers

• DelRec: learning delays in recurrent spiking neural networks: learns real-valued axonal or synaptic delays in a recurrent LIF network with surrogate gradients, annealing the spread width to round to integers.

Named but not covered, because no card exists: feedforward delay learning with annealing (DCLS, Hammouamri et al. 2024), which the card's lineage names.