Paper · 2024-02-22 · cs.LG, cs.AI
Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data
Three Neural SDE classes (Langevin-type, linear-noise, geometric) with a spline-driven controlled path in the drift, analysed for stability and for output-distribution shift under missing data. Author-reported: lowest interpolation MSE on PhysioNet Mortality, best Speech Commands accuracy and Sepsis AUROC with observation intensity among the compared models, and robustness across 30 datasets.
Neural SDENeural ODETime series classificationTime-series imputationneural-sdeirregular-time-seriesstochastic-differential-equations
arXivPDFCodeMarkdown21 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Many real measurements arrive at uneven times and with gaps. An ICU chart is updated when a nurse writes something down, and a sensor can drop readings. Recurrent networks expect equal steps, so they cope poorly. Neural differential equations take a different route: they learn a hidden state that changes continuously in time, so the gaps become a question of when to read the state.
A neural SDE adds random noise to that continuous change. The noise can help, but its form is easy to get wrong: a naive choice can stop training from converging or let the hidden state blow up. The paper picks the drift and noise terms from three stochastic equations it calls the Langevin-type, the linear-noise and the geometric design. For these it argues that a small change in the input, such as a missing value, causes only a small change in the predicted output distribution. The geometric design also keeps the hidden state non-negative. The dynamics are fed a spline through the observed values, so the observations shape the path between readings.
The authors test the designs on interpolation, on classification of clinical and speech series, and on a study that removes values from 30 public datasets. They report the best numbers on several of these tests. Those numbers are the authors' own, and the official code differs from the text in places.
TL;DR
The paper proposes three Neural SDE classes (LSDE, LNSDE, GSDE) whose drift and diffusion are chosen for stability, combined with a controlled path in the drift §3.1 §3.4. It proves that solutions exist and are unique under Lipschitz and growth conditions, and bounds how far missing-data perturbations move the output distribution §3.2 §3.3. The authors report the best results among the compared models on PhysioNet interpolation, Sepsis and Speech Commands classification, and a 30-dataset missingness study §4. The official code departs from the text in several places: it scales the diffusion by a learned sigmoid, applies a ReLU to the diffusion network, adds batch normalisation and an L2-type regulariser, replaces the geometric drift factor, and implements the "×100 learning rate" as a gradient hook. Implement from the paper's equations, and check each item under Paper vs code before relying on the code.
Problem
- Irregularly sampled, partly observed series break the fixed-step assumptions of RNNs §1. Neural ODEs and neural CDEs model continuous time, and a neural CDE is driven by a spline through the data §2.1 eq. 2.
- A naive Neural SDE can lack a unique strong solution, destabilise, or give an Euler discretisation that explodes in finite time §2.2. Its performance depends strongly on the diffusion choice: in the paper's test on one dataset at a 50% missing rate, a non-Lipschitz diffusion and the naive network diffusion both fail to stabilise training §2.2 fig. 1.
- Distribution shift caused by irregularity and missingness is a robustness problem. The authors state that this had not been explored for Neural SDEs §1.
- The guarantees rest on Lipschitz continuity, linear growth or boundedness of the networks, and a dissipativity condition on the drift §3.2.
Method
Latent dynamics
All three classes start from a state set by an affine map of the input, z(0) = h(x) §3.1. Drift and diffusion are neural networks with parameters θ_γ and θ_σ §3.1. The latent state is a d_z-dimensional Brownian-driven process, and the stochastic integral is Itô §2.1 eq. 3.
| Class | Equation | Drift | Diffusion | Cite |
|---|---|---|---|---|
| LSDE (Langevin-type) | dz = γ(z̄) dt + σ(t) dW | no explicit time dependence | σ(t), a network of time | eq. 4 §3.1 |
| LNSDE (linear noise) | dz = γ(t, z̄) dt + σ(t) z dW | depends on t and z̄ | σ(t) multiplied by z | eq. 5 §3.1 |
| GSDE (geometric) | dz/z = γ(t, z̄) dt + σ(t) dW | depends on t and z̄ | σ(t) in the dz/z form, so z·σ(t) in dz | eq. 6 §3.1 |
For the LSDE the paper assumes a potential U with ∇U = γ and σ(t) tending to a positive constant. Under mild conditions the process then has a unique invariant (Gibbs) measure §3.1. The robustness argument for the LSDE uses a contraction result for Langevin SDEs §3.3 app. B.2.
Controlled path
The drift receives a controlled map of the state and the path X(t): z̄(t) = ζ(t, z(t), X(t)) eq. 10 §3.4. ζ is a network, and X(t) is a cubic spline through the data. The main text names a natural cubic spline or a Hermite cubic with backward differences §2.1; the appendix reports the Hermite form was selected app. F. The authors show that the controlled versions keep unique strong solutions §3.4 app. A.1.
Read-out
The final state is passed to a two-layer fully connected network with ReLU app. D.3:
y = F(z(T)) [eq. 9]For classification the loss is cross-entropy on that output, with 10% dropout app. D.3. The Sepsis code uses binary cross-entropy [code: benchmark_classification/sepsis-sde.py#L32].
Numerical scheme
The experiments use the explicit Euler method with diagonal noise app. D.2 app. D.4. The appendix compares it with the Milstein and stochastic Runge-Kutta schemes, which the paper says have higher convergence orders but cost more time app. D.4. The step size is not stated in the paper.
Forward pass for the proposed classifier
# Forward pass of a proposed Neural SDE classifier [§3.1] [§3.4] [eq. 9]
# Inputs: times t_0 .. t_n, observed values x_0 .. x_n, controlled path X(t)
# Output: class scores for one series
def classify(times, controlled_path, x0):
# Initial latent state is an affine map of the first observation [§3.1]
z = h(x0) # (d_z,)
# Step from t_0 to T with explicit Euler and diagonal noise [app. D.4]
for t, dt, dW in brownian_steps(times):
# Controlled map, used in place of z inside the drift [eq. 10]
z_bar = zeta(t, z, controlled_path(t)) # (d_z,)
# Drift and diffusion of the chosen class [eq. 4] [§3.1]
drift = gamma(z_bar)
diffusion = sigma(t)
# Explicit Euler-Maruyama step [app. D.4]
z = z + drift * dt + diffusion * dW
# Two-layer read-out of the terminal state [eq. 9] [app. D.3]
return F(z)Training procedure
The paper's Algorithm 1 trains all four parameter groups jointly with cross-entropy and keeps the parameters with the lowest validation loss alg. 1. The text adds early stopping after 10 successive epochs without a validation-loss gain app. D.3.
# Training loop for classification [alg. 1] [app. D.3]
# Inputs: train set, validation set, max epochs
# Output: parameters with the lowest validation loss
def train(train_set, val_set, max_epochs):
# Initialise the controlled map, drift, diffusion and read-out [alg. 1]
params = initialise_all_groups()
best_params = params
best_val_loss = infinity
epochs_without_gain = 0
for epoch in range(max_epochs):
# Minimise cross-entropy on the training split [alg. 1]
for batch in train_set:
loss = cross_entropy(model(batch), batch.labels)
params = adam_step(params, loss)
# Track the lowest validation loss and keep those parameters [alg. 1]
val_loss = evaluate(model, val_set)
if val_loss < best_val_loss:
best_val_loss = val_loss
best_params = params
epochs_without_gain = 0
else:
epochs_without_gain = epochs_without_gain + 1
# Early stopping after 10 epochs without a gain [app. D.3]
if epochs_without_gain >= 10:
break
return best_paramsAssumptions and what is proved
- Lipschitz continuity of each network, with constant L_s §3.2.
- Linear growth, |s(t,x)| ≤ K_s(1 + |x|), when the last layer is ReLU or linear. A tanh or sigmoid last layer gives the bounded case |s| ≤ K_s instead §3.2 eq. 7.
- Dissipativity: ⟨s(t,x), x⟩ ≤ −m|x|² + b for positive m and b §3.2. The paper states that convexity is not required §3.2.
- Under the first two, the LSDE and LNSDE have unique strong solutions. For the GSDE with a tanh or sigmoid activation in γ, the solution is unique, almost surely non-negative, and state 0 is absorbing §3.2 app. A.
- With ρ bounding the input perturbation, √E|x − x̃|² ≤ ρ eq. 8, and F Lipschitz with constant L_F, the paper gives bounds on the output distribution §3.3. The constants L_F, L_h (initial map), L_γ and K_γ are not given numerically. c_1 and c_2 are positive and independent of T, but no values are stated.
For the LSDE, with W_1 and W_2 the Wasserstein distances between the output distributions for x and x̃ §3.3:
W_1 \le \sqrt{3}\, L_F L_h c_1 e^{-c_2 T} \sqrt{5 + 2\mathbb{E}|x|^4 + 2\mathbb{E}|\tilde x|^4}\; \rho
W_2 \le \sqrt{2\sqrt{3} L_h c_1}\, L_F e^{-c_2 T/2} \left(5 + 2\mathbb{E}|x|^4 + 2\mathbb{E}|\tilde x|^4\right)^{1/4} \sqrt{\rho}
For the LNSDE, with constant σ_θ and |σ_θ|² > 2L_γ, and for the GSDE with |σ_θ|² > 2K_γ, for sufficiently large T §3.3:
W_1 \le L_F \exp\{-(|\sigma_\theta|^2 - 2L_\gamma) T/2\}(1+\rho)
The GSDE version replaces L_γ with K_γ. Both bounds hold for sufficiently large T, and the paper does not give the threshold §3.3 app. B.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Layers in vector field n_l | grid {1, 2, 3, 4} | app. E | grid search on validation; bold cells in the appendix tables mark the best per task |
| Hidden width n_h | grid {16, 32, 64, 128} | app. E | latent width equals n_h in the classification code; latent size is not stated in the text |
| Optimiser | Adam, learning rate 0.001 | app. E | Mortality, MuJoCo, Sepsis and Speech |
| Batch size | 64 (Mortality); 1024 (MuJoCo, Sepsis, Speech) | app. E | robustness: 16 to 128, chosen by data size app. E model_run.py |
| Epochs | 300 Mortality; 500 MuJoCo; 200 Sepsis and Speech | app. E | robustness: 100 epochs for all models app. E |
| Weight decay | not stated in the paper | common_sde.py | code sets it to 0.01 × learning rate |
| Dropout | 10% in the read-out network | app. D.3 | classification; the interpolation code also uses 0.1 in its encoder sde_interpolation.py |
| Early stopping | 10 epochs without validation-loss gain | app. D.3 | the code differs; see Paper vs code |
| Final-layer learning-rate scale | ×100 | app. D.2 | implemented in code as a gradient hook; see Paper vs code |
| Time encoding | sinusoidal positional encoding | app. D.2 | |
| Output nonlinearity | tanh on drift, diffusion and other vector fields | app. D.2 | |
| Noise structure | diagonal | app. D.2 | |
| Solver | explicit Euler | app. D.4 | step size not stated in the paper |
| Controlled path | Hermite cubic with backward differences | app. F common.py | the main text also names natural cubic splines §2.1 |
| Interpolation objective | normalised variational lower bound (ELBO) | app. E | the code uses a K = 5 importance-weighted bound; see Paper vs code |
| Missing rates | 0, 30, 50, 70% | §4.2 | robustness study |
| Data split | 70 / 15 / 15 | app. C | robustness study; the split code in torch-ists is partly commented out |
| Uniform scaling | series rescaled to the longest length | app. C | variable-length UEA/UCR datasets |
| Mortality data | 37 variables; 8000 instances; 50% to 90% observed | app. C §4.1 | interpolation |
| Sepsis data | 40,335 cases; 34 variables; AUROC | §4.1 app. C | classification, with and without observation intensity |
| Speech Commands | 10 of 35 words; 20-dimensional MFCC; length 161 | §4.1 app. C | classification |
| MuJoCo | 10,000 Hopper simulations; 14 dimensions; 100 steps | app. C | forecasting: 50 steps in, 10 out; 30, 50, 70% of values dropped |
Code-only settings, not in the paper: interpolation runs use latent width 32, decoder width 64, K = 5 importance samples, KL warm-up and quantisation 0.016 run.sh. The classification code also uses ReduceLROnPlateau with patience 5 [code: benchmark_classification/common_sde.py#L120-L145].
Results
Author-reported. The robustness study reruns the baselines with the authors' source code and a Ray-based search app. E. The MuJoCo, Sepsis and Speech baseline numbers are quoted from prior papers, and the Mortality setup follows the mTAN pipeline app. E. Hardware: an Ubuntu 22.04 server with a Xeon Gold 6242 and six NVIDIA A100 40 GB GPUs §4.1 app. D.
Interpolation, PhysioNet Mortality (MSE × 10⁻³ at each observed fraction) tab. 1
| Methods | 50% | 60% | 70% | 80% | 90% |
|---|---|---|---|---|---|
| RNN-VAE | 13.418 ± 0.008 | 12.594 ± 0.004 | 11.887 ± 0.005 | 11.133 ± 0.007 | 11.470 ± 0.006 |
| L-ODE-RNN | 8.132 ± 0.020 | 8.140 ± 0.018 | 8.171 ± 0.030 | 8.143 ± 0.025 | 8.402 ± 0.022 |
| L-ODE-ODE | 6.721 ± 0.109 | 6.816 ± 0.045 | 6.798 ± 0.143 | 6.850 ± 0.066 | 7.142 ± 0.066 |
| mTAND-Full | 4.139 ± 0.029 | 4.018 ± 0.048 | 4.157 ± 0.053 | 4.410 ± 0.149 | 4.798 ± 0.036 |
| LatentSDE | 8.862 ± 0.036 | 8.864 ± 0.058 | 8.686 ± 0.122 | 8.716 ± 0.032 | 8.435 ± 0.077 |
| Neural SDE | 8.592 ± 0.055 | 8.591 ± 0.052 | 8.540 ± 0.051 | 8.318 ± 0.010 | 8.252 ± 0.023 |
| Neural LSDE | 3.799 ± 0.055 | 3.584 ± 0.055 | 3.457 ± 0.078 | 3.262 ± 0.032 | 3.111 ± 0.076 |
| Neural LNSDE | 3.808 ± 0.078 | 3.617 ± 0.129 | 3.405 ± 0.089 | 3.269 ± 0.057 | 3.154 ± 0.084 |
| Neural GSDE | 3.824 ± 0.088 | 3.667 ± 0.079 | 3.493 ± 0.024 | 3.287 ± 0.070 | 3.118 ± 0.065 |
The proposed rows equal the grid cell n_l = 4, n_h = 128 at every level; see Pitfalls.
Classification, PhysioNet Sepsis (test AUROC and memory in MB, with and without observation intensity, OI) tab. 2
| Methods | AUROC OI | AUROC no OI | Memory OI | Memory no OI |
|---|---|---|---|---|
| GRU-Δt | 0.878 ± 0.006 | 0.840 ± 0.007 | 837 | 826 |
| GRU-D | 0.871 ± 0.022 | 0.850 ± 0.013 | 889 | 878 |
| GRU-ODE | 0.852 ± 0.010 | 0.771 ± 0.024 | 454 | 273 |
| ODE-RNN | 0.874 ± 0.016 | 0.833 ± 0.020 | 696 | 686 |
| Latent-ODE | 0.787 ± 0.011 | 0.495 ± 0.002 | 133 | 126 |
| ACE-NODE | 0.804 ± 0.010 | 0.514 ± 0.003 | 194 | 218 |
| Neural CDE | 0.880 ± 0.006 | 0.776 ± 0.009 | 244 | 122 |
| ANCDE | 0.900 ± 0.002 | 0.823 ± 0.003 | 285 | 129 |
| Neural SDE | 0.799 ± 0.007 | 0.796 ± 0.006 | 368 | 240 |
| Neural LSDE | 0.909 ± 0.004 | 0.879 ± 0.008 | 373 | 436 |
| Neural LNSDE | 0.911 ± 0.002 | 0.881 ± 0.002 | 341 | 445 |
| Neural GSDE | 0.909 ± 0.001 | 0.884 ± 0.002 | 588 | 280 |
Classification, Speech Commands (test accuracy and memory in MB) tab. 3. The naive Neural SDE is left out because of training instability §4.1.
| Methods | Accuracy | Memory |
|---|---|---|
| RNN | 0.197 ± 0.006 | 1905 |
| LSTM | 0.684 ± 0.034 | 4080 |
| GRU | 0.747 ± 0.050 | 4609 |
| GRU-Δt | 0.453 ± 0.313 | 1612 |
| GRU-D | 0.346 ± 0.286 | 1717 |
| GRU-ODE | 0.487 ± 0.018 | 171 |
| ODE-RNN | 0.678 ± 0.276 | 1472 |
| Latent-ODE | 0.912 ± 0.006 | 2668 |
| Augmented-ODE | 0.911 ± 0.008 | 2626 |
| ACE-NODE | 0.911 ± 0.003 | 3046 |
| Neural CDE | 0.898 ± 0.025 | 175 |
| ANCDE | 0.807 ± 0.075 | 180 |
| LEAP | 0.922 ± 0.002 | 391 |
| Neural LSDE | 0.927 ± 0.004 | 1187 |
| Neural LNSDE | 0.923 ± 0.001 | 1164 |
| Neural GSDE | 0.913 ± 0.001 | 1565 |
The LNSDE value here differs from the appendix; see Pitfalls.
Robustness to missing data, 30 UEA/UCR datasets (average accuracy with average standard deviation in parentheses, and average rank) tab. 4
| Methods | Regular | Rank | 30% | Rank | 50% | Rank | 70% | Rank |
|---|---|---|---|---|---|---|---|---|
| RNN | 0.582 (0.064) | 13.9 | 0.513 (0.087) | 15.3 | 0.485 (0.088) | 16.6 | 0.472 (0.072) | 15.6 |
| LSTM | 0.633 (0.053) | 11.2 | 0.595 (0.060) | 12.1 | 0.567 (0.061) | 12.9 | 0.558 (0.058) | 12.5 |
| GRU | 0.672 (0.059) | 8.3 | 0.621 (0.063) | 10.2 | 0.610 (0.055) | 10.3 | 0.597 (0.062) | 10.3 |
| GRU-Δt | 0.641 (0.070) | 10.3 | 0.636 (0.066) | 8.9 | 0.634 (0.056) | 8.7 | 0.618 (0.065) | 10.2 |
| GRU-D | 0.648 (0.071) | 10.3 | 0.624 (0.075) | 10.4 | 0.611 (0.073) | 11.1 | 0.604 (0.067) | 10.8 |
| MTAN | 0.648 (0.080) | 12.0 | 0.618 (0.099) | 10.7 | 0.618 (0.091) | 10.1 | 0.607 (0.078) | 9.8 |
| MIAM | 0.623 (0.048) | 11.0 | 0.603 (0.066) | 11.1 | 0.589 (0.063) | 12.2 | 0.569 (0.056) | 12.3 |
| GRU-ODE | 0.671 (0.067) | 9.8 | 0.663 (0.064) | 9.5 | 0.666 (0.059) | 8.3 | 0.655 (0.062) | 7.8 |
| ODE-RNN | 0.658 (0.063) | 9.1 | 0.635 (0.064) | 9.3 | 0.636 (0.067) | 8.2 | 0.630 (0.055) | 8.5 |
| ODE-LSTM | 0.619 (0.063) | 11.4 | 0.584 (0.064) | 12.1 | 0.561 (0.065) | 13.3 | 0.530 (0.085) | 12.9 |
| Neural CDE | 0.709 (0.061) | 8.2 | 0.706 (0.073) | 6.4 | 0.696 (0.064) | 6.5 | 0.665 (0.072) | 7.6 |
| Neural RDE | 0.607 (0.071) | 13.9 | 0.514 (0.064) | 14.9 | 0.468 (0.068) | 15.2 | 0.415 (0.077) | 16.3 |
| ANCDE | 0.693 (0.067) | 7.8 | 0.687 (0.068) | 7.2 | 0.683 (0.078) | 6.9 | 0.655 (0.067) | 7.1 |
| EXIT | 0.636 (0.073) | 11.1 | 0.633 (0.078) | 10.2 | 0.616 (0.075) | 10.6 | 0.599 (0.075) | 11.1 |
| LEAP | 0.444 (0.068) | 15.2 | 0.401 (0.078) | 16.3 | 0.425 (0.073) | 14.9 | 0.414 (0.070) | 14.7 |
| Latent SDE | 0.456 (0.073) | 16.6 | 0.455 (0.073) | 15.4 | 0.455 (0.069) | 15.0 | 0.446 (0.066) | 15.1 |
| Neural SDE | 0.526 (0.068) | 13.4 | 0.508 (0.066) | 13.1 | 0.517 (0.058) | 13.2 | 0.512 (0.066) | 12.9 |
| Neural LSDE | 0.717 (0.056) | 5.6 | 0.690 (0.050) | 6.4 | 0.686 (0.051) | 6.1 | 0.682 (0.067) | 5.2 |
| Neural LNSDE | 0.727 (0.047) | 5.4 | 0.723 (0.050) | 5.0 | 0.717 (0.054) | 4.3 | 0.703 (0.054) | 4.2 |
| Neural GSDE | 0.716 (0.065) | 5.7 | 0.707 (0.069) | 5.3 | 0.698 (0.063) | 6.1 | 0.689 (0.056) | 5.3 |
Ablation, averaged over the 30 datasets tab. 5. The caption does not define the symbols O and X. Read against the text §4.3, O means the controlled path is included in the drift and X means it is not; N means a nonlinear network for σ and L an affine σ. Values are accuracy with standard deviation, and cross-entropy loss with standard deviation.
| Method | ζ | σ | Acc. uni (15) | Loss uni | Acc. multi (15) | Loss multi | Acc. all (30) | Loss all |
|---|---|---|---|---|---|---|---|---|
| Neural CDE (l) | — | — | 0.593 (0.094) | 0.809 (0.107) | 0.770 (0.037) | 0.595 (0.086) | 0.682 (0.065) | 0.702 (0.097) |
| Neural CDE (r) | — | — | 0.550 (0.096) | 0.879 (0.090) | 0.716 (0.045) | 0.715 (0.105) | 0.633 (0.071) | 0.797 (0.098) |
| Neural CDE (n) | — | — | 0.597 (0.083) | 0.795 (0.084) | 0.773 (0.042) | 0.590 (0.095) | 0.685 (0.063) | 0.692 (0.089) |
| Neural CDE (h) | — | — | 0.611 (0.092) | 0.804 (0.119) | 0.777 (0.044) | 0.549 (0.101) | 0.694 (0.068) | 0.676 (0.110) |
| Neural SDE | O | N | 0.615 (0.090) | 0.736 (0.089) | 0.760 (0.036) | 0.618 (0.092) | 0.688 (0.063) | 0.677 (0.091) |
| Neural SDE | O | L | 0.535 (0.095) | 0.838 (0.094) | 0.752 (0.040) | 0.633 (0.087) | 0.643 (0.068) | 0.736 (0.090) |
| Neural SDE | X | N | 0.516 (0.084) | 0.914 (0.069) | 0.515 (0.045) | 1.248 (0.085) | 0.516 (0.064) | 1.081 (0.077) |
| Neural SDE | X | L | 0.498 (0.087) | 0.929 (0.078) | 0.514 (0.050) | 1.255 (0.081) | 0.506 (0.068) | 1.092 (0.080) |
| Neural LSDE | O | N | 0.604 (0.081) | 0.752 (0.084) | 0.783 (0.031) | 0.572 (0.080) | 0.694 (0.056) | 0.662 (0.082) |
| Neural LSDE | O | L | 0.533 (0.089) | 0.877 (0.086) | 0.745 (0.038) | 0.668 (0.085) | 0.639 (0.064) | 0.772 (0.085) |
| Neural LSDE | X | N | 0.530 (0.069) | 0.856 (0.060) | 0.527 (0.054) | 1.177 (0.082) | 0.528 (0.060) | 1.039 (0.074) |
| Neural LSDE | X | L | 0.505 (0.064) | 0.912 (0.061) | 0.518 (0.057) | 1.210 (0.101) | 0.512 (0.059) | 1.083 (0.088) |
| Neural LNSDE | O | N | 0.654 (0.073) | 0.701 (0.091) | 0.780 (0.029) | 0.577 (0.070) | 0.717 (0.051) | 0.639 (0.080) |
| Neural LNSDE | O | L | 0.586 (0.087) | 0.765 (0.083) | 0.764 (0.044) | 0.617 (0.108) | 0.675 (0.066) | 0.691 (0.096) |
| Neural LNSDE | X | N | 0.532 (0.077) | 0.878 (0.060) | 0.502 (0.051) | 1.254 (0.082) | 0.517 (0.064) | 1.066 (0.071) |
| Neural LNSDE | X | L | 0.528 (0.085) | 0.890 (0.069) | 0.510 (0.047) | 1.257 (0.089) | 0.519 (0.066) | 1.074 (0.079) |
| Neural GSDE | O | N | 0.633 (0.091) | 0.742 (0.086) | 0.772 (0.036) | 0.598 (0.077) | 0.703 (0.063) | 0.670 (0.081) |
| Neural GSDE | O | L | 0.572 (0.083) | 0.796 (0.084) | 0.748 (0.039) | 0.653 (0.094) | 0.660 (0.061) | 0.724 (0.089) |
| Neural GSDE | X | N | 0.531 (0.074) | 0.868 (0.052) | 0.510 (0.045) | 1.261 (0.093) | 0.520 (0.060) | 1.064 (0.072) |
| Neural GSDE | X | L | 0.525 (0.078) | 0.893 (0.072) | 0.509 (0.052) | 1.261 (0.093) | 0.517 (0.065) | 1.077 (0.083) |
Forecasting, MuJoCo (test MSE with 0, 30, 50 and 70% of values dropped; memory in MB) app. E. Proposed rows and the strongest baselines only; the full table is in the appendix.
| Methods | Regular | 30% | 50% | 70% | Memory |
|---|---|---|---|---|---|
| ANCDE | 0.026 ± 0.001 | 0.025 ± 0.001 | 0.025 ± 0.001 | 0.024 ± 0.001 | 79 |
| LEAP | 0.022 ± 0.002 | 0.022 ± 0.001 | 0.022 ± 0.002 | 0.022 ± 0.001 | 144 |
| Neural SDE | 0.028 ± 0.004 | 0.029 ± 0.001 | 0.029 ± 0.001 | 0.027 ± 0.000 | 234 |
| Neural LSDE | 0.013 ± 0.000 | 0.014 ± 0.001 | 0.014 ± 0.000 | 0.013 ± 0.001 | 249 |
| Neural LNSDE | 0.012 ± 0.001 | 0.014 ± 0.001 | 0.014 ± 0.001 | 0.014 ± 0.000 | 273 |
| Neural GSDE | 0.013 ± 0.001 | 0.013 ± 0.001 | 0.013 ± 0.000 | 0.014 ± 0.000 | 306 |
Cost: the authors report that their methods use more memory than the CDE-based methods, and that they need more compute than Neural CDE models §4.1 §5. In Table 2 the LNSDE uses 445 MB without observation intensity, against 122 MB for Neural CDE tab. 2. A runtime table on BasicMotions gives 89 to 96 seconds for 100 epochs for the proposed methods, against 102.5 seconds for Neural CDE app. D.4.
Figures: Figure 1 compares six diffusion functions on one dataset at a 50% missing rate fig. 1 §2.2. Figure 2 shows test loss over training for the proposed methods and the naive Neural SDE on four datasets, without early stopping fig. 2 §4.2. Figures 3 and 4 show per-dataset results for the 15 univariate and 15 multivariate datasets at four missing rates fig. 3 fig. 4 app. F.
Paper vs code
Official code at commit dbda6575f8e7150af357febf6ef00c9702183322, licensed MIT. The paper links the repository §4.1, and its owner is the first author's GitHub account. The README says the missingness-study pipeline has moved to the separate torch-ists repository and points to it as the up-to-date version [code: README.md].
- Drift and diffusion are clipped. The code applies tanh to the drift output and to the scaled diffusion [code: benchmark_classification/models_sde/neuralsde.py#L227-L231] [code: benchmark_classification/models_sde/neuralsde.py#L290-L307]. The text specifies tanh as a final operation app. D.2, so this clipping is in the paper too. The code's own docstring calls the time features and clipping stabilisers rather than the paper's parameterisation [code: benchmark_classification/models_sde/neuralsde.py#L1-L10].
- Diffusion is scaled by a learned sigmoid. The code multiplies the diffusion by sigmoid(θ), with θ initialised to 1.0 [code: benchmark_classification/models_sde/neuralsde.py#L163] [code: benchmark_classification/models_sde/neuralsde.py#L304-L307]. The paper has no such scale.
- Diffusion is non-negative. For the LSDE, LNSDE and GSDE, the code applies a ReLU to the output of the diffusion network [code: benchmark_classification/models_sde/neuralsde.py#L279-L282]. The paper's σ is an unrestricted network §3.1.
- Time features are fed to the diffusion. The paper's σ is a time network with sinusoidal encoding app. D.2. The code feeds sin(t) and cos(t) to every noise option that uses time [code: benchmark_classification/models_sde/neuralsde.py#L186-L193]. These features are periodic in t. The theory assumes σ(t) converges to a constant §3.3, which a periodic function can do only if it is constant.
- GSDE drift and diffusion are not the geometric form. The paper writes the GSDE as dz/z = γ dt + σ dW eq. 6. The code computes the drift as tanh(MLP(·) · tanh(y)), which replaces the factor z by tanh(y) and clips afterwards neuralsde.py. It computes the diffusion as tanh(sigmoid(θ) · ReLU(NN(t)) · y) [code: benchmark_classification/models_sde/neuralsde.py#L279-L282] [code: benchmark_classification/models_sde/neuralsde.py#L304-L307]. Nothing in the code enforces the positivity that the paper proves for the exact form §3.2.
- Read-out head has batch normalisation. The code's head is Linear, BatchNorm, ReLU, Dropout 0.1, Linear [code: benchmark_classification/models_sde/neuralsde.py#L58-L61]. The paper describes two fully connected layers with ReLU and 10% dropout app. D.3.
- Final-layer "×100 learning rate" is a gradient hook. The paper states layer-wise learning rates, for example ×100 app. D.2. The code multiplies the gradient of the final layer by 100 through a backward hook [code: benchmark_classification/sepsis-sde.py#L55-L56] [code: benchmark_classification/speech_commands-sde.py#L33-L34]. Under Adam a constant gradient scale mostly cancels in the update, so this is not a learning-rate multiplier. This is my reading of the code and Adam's update rule; I did not run it.
- Regulariser and weight decay. The code adds 0.01 times the norm of each vector-field parameter tensor to the loss [code: benchmark_classification/common_sde.py#L16-L22] [code: benchmark_classification/common_sde.py#L310], and uses Adam with weight decay 0.01 × learning rate common_sde.py. Neither is in the paper. Evaluation uses the same wrapped loss [code: benchmark_classification/common_sde.py#L268-L273].
- Learning-rate schedule. The classification code uses ReduceLROnPlateau with patience 5 common_sde.py. The paper gives a learning rate of 0.001 and no schedule app. E.
- Model selection differs. The paper keeps the parameters with the lowest validation loss alg. 1. The code keeps the checkpoint with the highest validation accuracy common_sde.py and restores it at the end common_sde.py.
- Early stopping differs. The paper stops after 10 epochs without a validation-loss gain app. D.3. The code stops after 50 epochs without a gain in training loss or in training accuracy [code: benchmark_classification/common_sde.py#L205-L212]. The training-loss criterion uses the loss that includes the regulariser.
- Sepsis positive-class weight. The code uses a positive weight of 10 in the binary cross-entropy sepsis-sde.py sepsis-sde.py. The paper does not state one §4.1.
- Initial state for Sepsis and Speech. The paper's z(0) is an affine map of the observations §3.1. The Sepsis code replaces it with a two-layer network over the static features, with 256 hidden units [code: benchmark_classification/sepsis-sde.py#L15-L29]. The Speech code uses the affine initial network applied to the spline value at the first time [code: benchmark_classification/models_sde/neuralsde.py#L63-L69] [code: benchmark_classification/models_sde/neuralsde.py#L146].
- Interpolation objective. The paper says ELBO app. E. The code draws K = 5 samples and optimises an importance-weighted bound with KL warm-up [code: benchmark_interpolation/sde_interpolation.py#L196-L209] [code: benchmark_interpolation/sde_interpolation.py#L149-L156] [code: benchmark_interpolation/run.sh#L10]. The optimiser has a cosine schedule with T_max 10 sde_interpolation.py. The run fixes the seed to 0 run.sh.
- Naive baseline is configured differently. The README maps the paper's "Neural SDE" to neuralsde_3_18 README.md. The classification code builds naivesde with input option 1 and noise option 18 [code: benchmark_classification/common_sde.py#L311-L317]. The interpolation default is neuralsde_0_18 sde_interpolation.py, and run.sh uses neuralsde_1_18 run.sh.
- Learning-rate range for the robustness study. The paper's grid is 10⁻⁴ to 10⁻¹ on a log scale app. E. The search code uses 10⁻⁴ to 10⁻² [code: torch-ists/param_search.py#L231].
- Robustness split is partly commented out. The paper gives 70/15/15 [app. C]. In torch-ists/model_run.py the split lines are commented out [code: torch-ists/model_run.py#L140-L142]. I did not verify which split runs.
- Scheduler for the robustness code. The training scheduler is StepLR with step 10 and gamma 0.5 model_run.py. The search uses step 5 param_search.py. The paper does not state a schedule.
No discrepancies were found in the MuJoCo settings: the paper's 500 epochs, learning rate 0.001 and batch 1024 app. E match the script [code: benchmark_forecasting/mujoco.sh#L11] mujoco-sde.py.
Pitfalls
- Positivity is not carried over. The non-negativity proof for the geometric design uses an exponential change of variables §3.2 app. A. The code does not implement that transform, and item 5 of Paper vs code changes the drift and diffusion. Non-negativity is therefore not guaranteed by the code. I did not test it numerically.
- Saturation of the diffusion. Because the code applies tanh after multiplying by y, the LNSDE and GSDE diffusions saturate to ±1 when |y| is large neuralsde.py. The multiplicative σz and z·σ structure of the paper is lost in that regime.
- Bounded case versus the code. The paper's bounded case eq. 7 covers a tanh last layer, so a bounded drift is inside the theory. The code, however, squashes the drift and the scaled diffusion after the multiplicative terms neuralsde.py [code: benchmark_classification/models_sde/neuralsde.py#L290-L293]. The constant K_γ the paper defines for the network does not describe the code's effective drift without computing it from the code.
- Theorem constants are missing. The bounds in §3.3 use L_F, L_h, L_γ, K_γ, c_1 and c_2, none of which are given numerically. The threshold on T is not stated. The bounds cannot be evaluated for a given model without estimating these constants.
- Time scale. The periodic time features depend on the scale of the observation times neuralsde.py. The paper does not say how times are normalised for Sepsis or Speech. Check the dataset loaders before reusing the code.
- Euler step size. The step is the minimum gap between observation times, floored at 10⁻³ neuralsde.py. The paper's own solver study ranks Euler below Milstein and SRK on convergence order app. D.4. Results can change with the grid of observation times.
- Mortality grid versus Table 1. The proposed rows of Table 1 equal the grid cell n_l = 4, n_h = 128 at every level. The grid's bold cell at 50% for LSDE is 3.793 at n_l = 3, n_h = 128, lower than the 3.799 reported tab. 1 app. E. The text does not say which configuration Table 1 reports.
- Speech LNSDE value. Table 3 gives 0.923 ± 0.001 tab. 3. The appendix gives 0.924 ± 0.000 for the same configuration (n_l = 1, n_h = 128, memory 1164 MB) app. E.
- Sepsis no-OI Neural SDE value. Table 2 gives 0.796 ± 0.006 tab. 2. The appendix gives 0.797 ± 0.006 for n_l = 3, n_h = 32, with the same memory of 240 MB app. E.
- Some baselines are quoted, not rerun. The MuJoCo, Sepsis and Speech baseline numbers come from prior papers app. E. Comparisons with numbers from other sources need care.
- Memory and compute. The proposed methods use more memory than the CDE baselines in the sepsis test (445 MB against 122 MB) tab. 2 and need more compute overall §5.
- Naive Neural SDE excluded from Speech. The paper leaves the naive Neural SDE out of the Speech table because of training instability §4.1, so that comparison is missing.
- Tuning budget. The classification models are tuned on a grid of 16 cells. The robustness code draws 20 samples with ASHA and a 10-epoch budget [code: torch-ists/param_search.py#L225-L236]. The paper does not report the number of trials for the classification baselines app. E.
- Table 5 symbols. The caption does not define O and X. The meaning used in Table 5 here is inferred from the text §4.3.
Lineage
- Builds on: neural ODEs eq. 1 §2.1; neural CDEs with a spline-driven controlled path eq. 2 §2.1; neural SDEs with the Itô integral eq. 3 §2.1. The paper names earlier Neural SDE work by Tzen and Raginsky (2019), Liu et al. (2019) and Li et al. (2020, Latent SDE) §1. The Langevin and linear-noise designs with their stability analysis are the paper's own contribution §3.1 §3.3.
- Compared against: RNN models, attention models (mTAN, MIAM), neural ODE variants (GRU-ODE, ODE-RNN, Latent-ODE, Augmented-ODE, ACE-NODE), neural CDE variants (ANCDE, EXIT, LEAP, Neural RDE) and Latent SDE §4 app. D.1.
- Superseded by: not known from wiki content.
Source and license
Adapted from “Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data” by YongKyung Oh, Dong-Young Lim, Sungil Kim (arXiv:2402.14989), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/yongkyung-oh/Stable-Neural-SDEs at commit dbda6575f8e7 (MIT). Any quoted code is shown under that license.