---
kind: concept
slug: learnable-delays
title: Learnable delays
summary: >-
  Gives each recurrent connection of a spiking network a trainable travel time, learned with its weight by surrogate gradients. The delay is spread over neighbouring steps during training, then rounded to integers for inference. Axonal (per neuron) and synaptic (per connection) variants; the reported gains over fixed delays are small and not significant in every case.
tags: [spiking-neural-networks, neuromorphic-computing, delays]
papers: ["2509.24852"]
related: [spiking-neural-network, surrogate-gradient, recurrent-neural-network]
status: draft
updated_at: 2026-10-19
---

## Overview

A spiking network passes information as spikes. In a recurrent one, a spike
reaches the neuron itself or its neighbours after a travel time, the delay.
Learnable delays make that time a trainable parameter, learned together with
the connection's weight, so the network can shift timing as well as strength
[arxiv:2509.24852 §2.1].

The shared invariant: a delayed spike is scheduled at a real-valued time
during training and spread over the neighbouring integer steps by a triangular
kernel. The kernel's width is annealed to zero, and the delays are rounded to
integers afterwards [arxiv:2509.24852 §2.1] [arxiv:2509.24852 §4.4].

The obstacle is in the simulator. Standard spiking simulators apply synapses
and delays after each layer's update, so a delayed recurrent connection needs
extra work inside every time step [arxiv:2509.24852 supp. Difficulty and
benefit of recurrent delays in SNN simulators]. Recurrent delays had been
studied little. The one earlier recurrent method used EventProp, which the
authors say scales poorly [arxiv:2509.24852 §1]. Feedforward delay learning
with annealing exists, but is not covered here (see Papers).

## Variants

### Axonal delays

One delay per presynaptic neuron, so N delays per layer [arxiv:2509.24852
§2.1]. A spike from neuron j reaches all of its targets at the same
learned lag. Reported with learned delays: 82.42 ± 0.23% on SSC, 85.74 ± 1.49%
on AL and 83.53 ± 0.26% on HAR, all author-reported over several seeds
[arxiv:2509.24852 tab. 2] [arxiv:2509.24852 tab. 3]. Card:
[DelRec](/papers/2509.24852).

### Synaptic delays

One delay per connection, so N² delays per layer [arxiv:2509.24852 §2.1].
Each weight gets its own lag. Reported with learned delays: 96.63 ± 0.12% on
PS-MNIST, 85.00 ± 0.85% on AL and 84.44 ± 0.27% on HAR [arxiv:2509.24852 tab.
1] [arxiv:2509.24852 tab. 3]. Card: [DelRec](/papers/2509.24852).

### Fixed delays (control)

The same architecture with the delays not trained. This is the baseline that
separates the effect of having delays from the effect of learning them
[arxiv:2509.24852 tab. 3]. Learned minus fixed, author-reported:

| Delay type | Learning | SSC | PS-MNIST | AL | HAR |
|---|---|---|---|---|---|
| Axonal | fixed | 82.11 ± 0.16 | 95.78 ± 0.27 | 82.72 ± 1.62 | 82.73 ± 0.37 |
| Axonal | learned | 82.42 ± 0.23 | 96.32 ± 0.19 | 85.74 ± 1.49 | 83.53 ± 0.26 |
| Synaptic | fixed | 82.09 ± 0.22 | 95.90 ± 0.22 | 83.74 ± 1.24 | 82.89 ± 0.27 |
| Synaptic | learned | 82.24 ± 0.19 | 96.63 ± 0.12 | 85.00 ± 0.85 | 84.44 ± 0.27 |

Source: [arxiv:2509.24852 tab. 3]. Five seeds, three for HAR. Of the eight
learned-versus-fixed comparisons, six are significant at p < 0.05. The two
that are not are synaptic SSC (+0.15, p = 0.275) and synaptic AL (+1.27,
p = 0.101) [arxiv:2509.24852 supp. Statistical significance].

### Learned delays without σ-annealing (ablation)

The Mackey-Glass forecasting study compares the full model with a version
whose kernel width is not annealed. The full model has lower NMSE in every
tested condition. The paper gives no single headline number for this task
[arxiv:2509.24852 fig. 3] [arxiv:2509.24852 supp. NMSE values across all τ
and H].

### Shared delay per layer (negative result)

A single learned delay per layer barely moves from its initial value on HAR
and stays below the fixed families. The gain comes from per-neuron or
per-connection delays, not from having one tunable lag [arxiv:2509.24852 supp.
Comparison with a single delay per layer].

## Mechanism details

- **Spread kernel.** A spike scheduled at 1 + d is shared between future steps
  by a triangle of width σ. The support is [d − σ, 2 + d + σ], so only a
  bounded window of future steps must be scheduled [arxiv:2509.24852 eq. 7]
  [arxiv:2509.24852 eq. 10]. At σ = 0 the kernel is linear interpolation
  between the two integer steps around 1 + d [arxiv:2509.24852 §2.1].
- **Annealing.** σ decays per epoch as σ_init × decay^(100 × epoch / N_epochs)
  [arxiv:2509.24852 eq. 13]. On SSC and PS-MNIST each neuron also gets a
  parameter that sets its own width [arxiv:2509.24852 §4.4]
  [arxiv:2509.24852 eq. 14]. On AL and HAR σ_init is fixed at 0, because the
  authors found a positive value did not help [arxiv:2509.24852 §4.4].
- **Buffer length.** The number of future slots is ⌈1 + max d + (1 + σ)⌉
  [arxiv:2509.24852 eq. 11]. It grows with the largest delay, and the
  per-step cost scales with the buffer length [arxiv:2509.24852 supp.
  Computational overhead].
- **Delay positivity.** Delays are clamped to stay non-negative [arxiv:2509.24852
  §4.1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Timing jitter is expected in the input | Axonal | Under temporal jitter the advantage holds for axonal delays and reverses for synaptic ones [arxiv:2509.24852 fig. 8c,d] |
| Spike or energy budget is tight | Synaptic learned | At a spike penalty, learned synaptic delays reach each accuracy with a lower firing rate than fixed ones; axonal curves match fixed [arxiv:2509.24852 fig. 7e] |
| Hardware only supports integer delays | Learned, rounded at inference | Rounding has little effect on SSC, PS-MNIST and HAR [arxiv:2509.24852 §4.4] |
| Task like AL, hardware only integer | Expect a drop, or keep fractional taps | Rounding AL costs a large amount of accuracy; the reported AL numbers keep fractional delays, which need two taps per connection [arxiv:2509.24852 §4.4] [arxiv:2509.24852 supp. Effect of delay rounding] |
| Max buffer length is limited | Learned, check max delay | Learned delays reach each accuracy threshold with a shorter maximum delay than fixed ones on HAR [arxiv:2509.24852 fig. 7c]. The buffer scales with the largest delay [arxiv:2509.24852 eq. 11] |
| Need the gain to be significant on SSC or AL with synaptic delays | Check first | Learned minus fixed is not significant for synaptic SSC and AL [arxiv:2509.24852 supp. Statistical significance] |
| Long-range temporal credit needed | Learned | Gradient maps show longer-range sensitivity for learned delays than for fixed random delays [arxiv:2509.24852 fig. 6e] |

Weigh these against the paper's own limits: the rows rest on one paper, and
most baselines are single runs [arxiv:2509.24852 supp. Statistical significance].

## Implementation sketch

Pseudocode in our own words, following the forward pass of the card's
source [arxiv:2509.24852 alg. 1]. Variants differ only in the shape of
`delays`: `(N,)` for axonal, `(N, N)` for synaptic.

```pseudocode
# Delayed recurrent LIF layer, forward pass  [arxiv:2509.24852 alg. 1]
# Input:  x_seq (T, B, N), weights (N, N), delays (N,) real, sigma scalar
# Output: spikes (T, B, N)

def delayed_recurrent_forward(x_seq, weights, delays, sigma):
    # Future slots the spread can reach  [arxiv:2509.24852 eq. 11]
    slots = ceil(1 + max(delays) + (1 + sigma))
    # Triangular weight of each delay at each future offset  [eq. 7]
    spread = triangle_spread(delays, sigma, slots)       # (N, slots)
    # Circular buffer of recurrent input still to be delivered
    buffer = zeros(B, N, slots)
    membrane = zeros(B, N)
    pointer = 0
    spikes = []
    for x_t in x_seq:
        # Recurrent input due at this step  [eq. 5]
        rec_now = buffer[:, :, pointer]
        # Leaky integration, threshold and reset  [eq. 1-3]
        membrane, spike = lif_step(membrane, x_t + rec_now)
        # Free the slot that was just read
        buffer[:, :, pointer] = 0
        # Spread each spike over the future slots  [eq. 8]
        for offset in range(1, slots):
            contrib = (spike * spread[:, offset]) @ weights.T
            buffer[:, :, (pointer + offset) % slots] += contrib
        pointer = (pointer + 1) % slots
        spikes.append(spike)
    return stack(spikes)


# Triangular spread of each delay over the future offsets  [eq. 7]
def triangle_spread(delays, sigma, slots):
    taus = arange(slots)                                 # (slots,)
    distance = abs(taus - (1 + delays[:, None]))         # (N, slots)
    width = 1 + sigma
    return maximum(0, (width - distance) / width ** 2)
```

Training: the threshold has no gradient, so use a surrogate gradient
[arxiv:2509.24852 §2.1]. Anneal σ per epoch with eq. 13. At inference, round
the delays to integers and set σ = 0 [arxiv:2509.24852 §4.4].

## Contradictions

- **Mackey-Glass delay learning rate.** The Methods give 10⁻¹ for the delay
  learning rate [arxiv:2509.24852 §4.3]. The paper card records that the
  repository config and a sweep log use 5 × 10⁻² [code:
  configs/perf_MG.py#L120]. Both are stated; the paper's figures are not
  reproduced by the wrong one.
- **PS-MNIST σ decay.** The Methods give 0.95 [arxiv:2509.24852 §4.4]. The
  PS-MNIST table and config use 0.8 [arxiv:2509.24852 supp. Hyperparameters].
- **HAR number for DelRec.** Table 2 reports the axonal configuration at
  83.53% [arxiv:2509.24852 tab. 2]. Table 3 gives the best learned
  configuration as synaptic, at 84.44% [arxiv:2509.24852 tab. 3]. The Methods
  say HAR uses the best configuration per model type [arxiv:2509.24852 §4.4].
- **Rounding in code.** The appendix describes an in-place projection of
  delays each epoch and at test [arxiv:2509.24852 supp. Effect of delay
  rounding]. The card reports a rounding path in the PyTorch layer that
  passes the gradient to the fractional delay [code:
  src/delrec/delay_layers_pytorch.py#L63-L64], and says which path the
  training scripts use was not traced.

## Pitfalls

- **Test-set selection on HAR and AL.** The HAR result comes from choosing the
  delay-spread initialisation that maximises test accuracy, and AL reports the
  best checkpoint over the epochs without a validation split. The authors' HAR
  and AL numbers are therefore optimistic [arxiv:2509.24852 §4.4]
  [arxiv:2509.24852 §4.8].
- **σ never reaches zero in training.** Eq. 13 leaves about 0.0059 × σ_init at
  the last epoch on SSC. The Methods describe σ as annealed to zero, yet the
  evaluation sets σ = 0 for a model that was not trained there [arxiv:2509.24852
  §4.4] [arxiv:2509.24852 eq. 13].
- **Integer delays on AL.** Keeping fractional delays needs two taps per
  connection. The appendix's figure for how much rounding-during-training
  recovers is ambiguous, so do not rely on it [arxiv:2509.24852 supp. Effect of
  delay rounding].
- **Probe gradients are not comparable.** The gradient-flow probe and the
  gradient maps use different back-propagated scalars, and the authors say
  their magnitudes are not comparable [arxiv:2509.24852 §2.3].
- **Open question in the sketch.** The sketch starts the spread at offset 1,
  as the card's pseudocode does. Whether eq. 8 also writes the current step
  when d < σ is not settled from the card; check it against the paper before
  implementing.

## Papers

- [DelRec: learning delays in recurrent spiking neural networks](/papers/2509.24852):
  learns real-valued axonal or synaptic delays in a recurrent LIF network with
  surrogate gradients, annealing the spread width to round to integers.

Named but not covered, because no card exists: feedforward delay learning with
annealing (DCLS, Hammouamri et al. 2024), which the card's lineage names.