Neural ODE

Overview

A neural ODE replaces a stack of residual layers with a continuous process. The state changes according to a derivative that a network defines, and a numerical solver integrates it from a start state. Letting the number of layers grow turns the layer update into an integral and then a differential equation 2106.12931 §4.2 2106.12931 eq. 12 2106.12931 eq. 13. The original formulation is Chen et al. 2018, cited in the paper as chen2018neural 2106.12931 §2. It has no card here, so this page does not summarise it.

The problem the one card addresses is over-smoothing. Stacking graph convolutions makes node representations converge, so distant nodes look alike 2106.12931 §1. Adding a restart term that pulls each state back toward its input at every step keeps the nodes distinct as the process runs 2106.12931 §4.2.

Only one card lists this concept, so this page describes one variant. The variant is shared by all the cases below: a graph ODE layer inside a spatio-temporal model.

Variants

Graph ODE with restart term

A graph propagation step is written as a tensor ODE. The state is changed along three modes at once: the node mode through a regularised adjacency, the time mode through a learned temporal matrix, and the feature mode through a learned feature matrix. A restart term adds the input state back at every step 2106.12931 §4.2 2106.12931 eq. 8. The adjacency is regularised so its eigenvalues lie in [0, α] 2106.12931 §3.2 2106.12931 eq. 1. The paper sets α = 0.8 2106.12931 §5.3. Dilated temporal convolutions wrap each ODE block 2106.12931 §4.3. Card: Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting.

The paper's reported results come from this variant on traffic data. The author-reported numbers are in 2106.12931 tab. 2.

When to use which

The choice depends on the data shape and on whether the model can be trained and solved within the budget. The paper gives no parameter count, run time or memory use 2106.12931 §5.3, so cost cannot be compared with other methods from the paper alone.

Implementation sketch

The core is an explicit Euler solve of the simplified ODE. The paper replaces the matrix logarithms with first-order terms, so the derivative is a sum of mode products plus the restart term 2106.12931 §4.2. The number of steps and the step size are not stated, so they are the main free choice.

# Euler solve of the simplified graph ODE  [arxiv:2106.12931 §4.2]
# Input:  H0 (N, T, F) from the upstream network; A_hat (N, N)
#         U (T, T) and W (F, F), eigenvalues in (0, 1)
# Output: state (N, T, F)

def graph_ode_block(H0, A_hat, U, W, num_steps, step_size):
    # The initial state is the upstream output  [arxiv:2106.12931 §4.2]
    state = H0
    # Euler steps; the step count and size are not stated  [arxiv:2106.12931 §4.2]
    for step in range(num_steps):
        # Mode products with the matrices minus identity  [arxiv:2106.12931 §4.2]
        spatial = mode_product(state, A_hat - I, mode=1)
        temporal = mode_product(state, U - I, mode=2)
        feature = mode_product(state, W - I, mode=3)
        # Restart term added at every step  [arxiv:2106.12931 §4.2]
        derivative = spatial + temporal + feature + H0
        state = state + step_size * derivative
    return state

U and W are learned. The paper parameterises them as an orthogonal basis times a diagonal with entries clamped to (0, 1), so their eigenvalues stay bounded 2106.12931 §4.2. The paper also gives an analytic solution of the ODE 2106.12931 §4.2, which removes the need for a numerical solver.

Contradictions

None known across papers, because only one card covers this concept. Within that paper, the text and the figures are not consistent on some labels. The text calls the spatial threshold σ² while §5.3 gives σ, and the text says PeMSD3 where the table says PeMS03 2106.12931 §4.1 2106.12931 §5.3 2106.12931 tab. 1.

Papers

• Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting: a graph ODE with a restart term and dilated temporal convolutions for traffic flow; author-reported lowest error on six PeMS datasets.