---
kind: concept
slug: neural-ode
title: Neural ODE
summary: >-
  Writes a deep stack of layers as a continuous process described by an ordinary differential equation, solved numerically. Only one card so far: a graph ODE with a restart term that keeps node states from blurring with depth, used for traffic forecasting. Choose it when the data sits on a fixed graph and deep graph stacks over-smooth; check the solver steps, which the paper leaves open.
tags: [continuous-depth, over-smoothing, time-series]
papers: ["2106.12931"]
related: [graph-neural-network, time-series-forecasting, temporal-convolutional-network]
status: draft
updated_at: 2026-10-19
---

## Overview

A neural ODE replaces a stack of residual layers with a continuous process. The
state changes according to a derivative that a network defines, and a numerical
solver integrates it from a start state. Letting the number of layers grow
turns the layer update into an integral and then a differential equation
[arxiv:2106.12931 §4.2] [arxiv:2106.12931 eq. 12] [arxiv:2106.12931 eq. 13].
The original formulation is Chen et al. 2018, cited in the paper as
chen2018neural [arxiv:2106.12931 §2]. It has no card here, so this page does not
summarise it.

The problem the one card addresses is over-smoothing. Stacking graph
convolutions makes node representations converge, so distant nodes look alike
[arxiv:2106.12931 §1]. Adding a restart term that pulls each state back toward
its input at every step keeps the nodes distinct as the process runs
[arxiv:2106.12931 §4.2].

Only one card lists this concept, so this page describes one variant. The
variant is shared by all the cases below: a graph ODE layer inside a
spatio-temporal model.

## Variants

### Graph ODE with restart term

A graph propagation step is written as a tensor ODE. The state is changed along
three modes at once: the node mode through a regularised adjacency, the time
mode through a learned temporal matrix, and the feature mode through a learned
feature matrix. A restart term adds the input state back at every step
[arxiv:2106.12931 §4.2] [arxiv:2106.12931 eq. 8]. The adjacency is regularised
so its eigenvalues lie in [0, α] [arxiv:2106.12931 §3.2] [arxiv:2106.12931 eq. 1].
The paper sets α = 0.8 [arxiv:2106.12931 §5.3]. Dilated temporal convolutions
wrap each ODE block [arxiv:2106.12931 §4.3]. Card: [Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting](/papers/2106.12931).

The paper's reported results come from this variant on traffic data. The
author-reported numbers are in [arxiv:2106.12931 tab. 2].

## When to use which

The choice depends on the data shape and on whether the model can be trained
and solved within the budget. The paper gives no parameter count, run time or
memory use [arxiv:2106.12931 §5.3], so cost cannot be compared with other
methods from the paper alone.

| Situation | Use | Why |
|---|---|---|
| Data lives on a fixed graph with known pairwise distances, and deeper graph stacks lose node detail | Graph ODE with restart term | The restart term keeps states distinct, and the paper reports that its ODE model stays stable as depth grows, although the values are only in a figure [arxiv:2106.12931 §4.2] [arxiv:2106.12931 §5.6.2] [arxiv:2106.12931 fig. 8] |
| No graph exists, or the graph changes per sample | Not covered by this page's variant | The method builds its spatial graph from sensor distances and its semantic graph from DTW similarity [arxiv:2106.12931 §4.1] |
| Forecasting traffic on the six PeMS datasets with one hour of history | Compare with the strongest baseline | The paper reports the lowest RMSE, MAE and MAPE on all six datasets, but margins are small in some rows, for example PeMS04 RMSE 32.82 against 33.65 for STSGCN [arxiv:2106.12931 tab. 2]. No variance or significance tests are reported [arxiv:2106.12931 §5.4] |
| The result must be reproduced exactly | Implement from the equations, then check the code | The solver step count and size are not stated [arxiv:2106.12931 §4.2], and the official code differs from the paper in ways that change the model, listed on the card |
| Single-step or short-horizon forecasting | Not tested here | The paper reports only the 60-minute horizon [arxiv:2106.12931 §5.3] |

## Implementation sketch

The core is an explicit Euler solve of the simplified ODE. The paper replaces
the matrix logarithms with first-order terms, so the derivative is a sum of
mode products plus the restart term [arxiv:2106.12931 §4.2]. The number of
steps and the step size are not stated, so they are the main free choice.

```pseudocode
# Euler solve of the simplified graph ODE  [arxiv:2106.12931 §4.2]
# Input:  H0 (N, T, F) from the upstream network; A_hat (N, N)
#         U (T, T) and W (F, F), eigenvalues in (0, 1)
# Output: state (N, T, F)

def graph_ode_block(H0, A_hat, U, W, num_steps, step_size):
    # The initial state is the upstream output  [arxiv:2106.12931 §4.2]
    state = H0
    # Euler steps; the step count and size are not stated  [arxiv:2106.12931 §4.2]
    for step in range(num_steps):
        # Mode products with the matrices minus identity  [arxiv:2106.12931 §4.2]
        spatial = mode_product(state, A_hat - I, mode=1)
        temporal = mode_product(state, U - I, mode=2)
        feature = mode_product(state, W - I, mode=3)
        # Restart term added at every step  [arxiv:2106.12931 §4.2]
        derivative = spatial + temporal + feature + H0
        state = state + step_size * derivative
    return state
```

U and W are learned. The paper parameterises them as an orthogonal basis times
a diagonal with entries clamped to (0, 1), so their eigenvalues stay bounded
[arxiv:2106.12931 §4.2]. The paper also gives an analytic solution of the ODE
[arxiv:2106.12931 §4.2], which removes the need for a numerical solver.

## Contradictions

None known across papers, because only one card covers this concept. Within
that paper, the text and the figures are not consistent on some labels. The
text calls the spatial threshold σ² while §5.3 gives σ, and the text says
PeMSD3 where the table says PeMS03 [arxiv:2106.12931 §4.1] [arxiv:2106.12931 §5.3]
[arxiv:2106.12931 tab. 1].

## Papers

- [Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting](/papers/2106.12931): a graph ODE with a restart term and dilated temporal convolutions for traffic flow; author-reported lowest error on six PeMS datasets.