wikillm

Paper · 2024-02-22 · cs.LG, cs.AI

Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data

Three Neural SDE classes (Langevin-type, linear-noise, geometric) with a spline-driven controlled path in the drift, analysed for stability and for output-distribution shift under missing data. Author-reported: lowest interpolation MSE on PhysioNet Mortality, best Speech Commands accuracy and Sepsis AUROC with observation intensity among the compared models, and robustness across 30 datasets.

Neural SDENeural ODETime series classificationTime-series imputationneural-sdeirregular-time-seriesstochastic-differential-equations

arXivPDFCodeMarkdown21 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many real measurements arrive at uneven times and with gaps. An ICU chart is updated when a nurse writes something down, and a sensor can drop readings. Recurrent networks expect equal steps, so they cope poorly. Neural differential equations take a different route: they learn a hidden state that changes continuously in time, so the gaps become a question of when to read the state.

A neural SDE adds random noise to that continuous change. The noise can help, but its form is easy to get wrong: a naive choice can stop training from converging or let the hidden state blow up. The paper picks the drift and noise terms from three stochastic equations it calls the Langevin-type, the linear-noise and the geometric design. For these it argues that a small change in the input, such as a missing value, causes only a small change in the predicted output distribution. The geometric design also keeps the hidden state non-negative. The dynamics are fed a spline through the observed values, so the observations shape the path between readings.

The authors test the designs on interpolation, on classification of clinical and speech series, and on a study that removes values from 30 public datasets. They report the best numbers on several of these tests. Those numbers are the authors' own, and the official code differs from the text in places.

TL;DR

The paper proposes three Neural SDE classes (LSDE, LNSDE, GSDE) whose drift and diffusion are chosen for stability, combined with a controlled path in the drift §3.1 §3.4. It proves that solutions exist and are unique under Lipschitz and growth conditions, and bounds how far missing-data perturbations move the output distribution §3.2 §3.3. The authors report the best results among the compared models on PhysioNet interpolation, Sepsis and Speech Commands classification, and a 30-dataset missingness study §4. The official code departs from the text in several places: it scales the diffusion by a learned sigmoid, applies a ReLU to the diffusion network, adds batch normalisation and an L2-type regulariser, replaces the geometric drift factor, and implements the "×100 learning rate" as a gradient hook. Implement from the paper's equations, and check each item under Paper vs code before relying on the code.

Problem

  • Irregularly sampled, partly observed series break the fixed-step assumptions of RNNs §1. Neural ODEs and neural CDEs model continuous time, and a neural CDE is driven by a spline through the data §2.1 eq. 2.
  • A naive Neural SDE can lack a unique strong solution, destabilise, or give an Euler discretisation that explodes in finite time §2.2. Its performance depends strongly on the diffusion choice: in the paper's test on one dataset at a 50% missing rate, a non-Lipschitz diffusion and the naive network diffusion both fail to stabilise training §2.2 fig. 1.
  • Distribution shift caused by irregularity and missingness is a robustness problem. The authors state that this had not been explored for Neural SDEs §1.
  • The guarantees rest on Lipschitz continuity, linear growth or boundedness of the networks, and a dissipativity condition on the drift §3.2.

Method

Latent dynamics

All three classes start from a state set by an affine map of the input, z(0) = h(x) §3.1. Drift and diffusion are neural networks with parameters θ_γ and θ_σ §3.1. The latent state is a d_z-dimensional Brownian-driven process, and the stochastic integral is Itô §2.1 eq. 3.

ClassEquationDriftDiffusionCite
LSDE (Langevin-type)dz = γ(z̄) dt + σ(t) dWno explicit time dependenceσ(t), a network of timeeq. 4 §3.1
LNSDE (linear noise)dz = γ(t, z̄) dt + σ(t) z dWdepends on t and z̄σ(t) multiplied by zeq. 5 §3.1
GSDE (geometric)dz/z = γ(t, z̄) dt + σ(t) dWdepends on t and z̄σ(t) in the dz/z form, so z·σ(t) in dzeq. 6 §3.1

For the LSDE the paper assumes a potential U with ∇U = γ and σ(t) tending to a positive constant. Under mild conditions the process then has a unique invariant (Gibbs) measure §3.1. The robustness argument for the LSDE uses a contraction result for Langevin SDEs §3.3 app. B.2.

Controlled path

The drift receives a controlled map of the state and the path X(t): z̄(t) = ζ(t, z(t), X(t)) eq. 10 §3.4. ζ is a network, and X(t) is a cubic spline through the data. The main text names a natural cubic spline or a Hermite cubic with backward differences §2.1; the appendix reports the Hermite form was selected app. F. The authors show that the controlled versions keep unique strong solutions §3.4 app. A.1.

Read-out

The final state is passed to a two-layer fully connected network with ReLU app. D.3:

Pseudocode
y = F(z(T))   [eq. 9]

For classification the loss is cross-entropy on that output, with 10% dropout app. D.3. The Sepsis code uses binary cross-entropy [code: benchmark_classification/sepsis-sde.py#L32].

Numerical scheme

The experiments use the explicit Euler method with diagonal noise app. D.2 app. D.4. The appendix compares it with the Milstein and stochastic Runge-Kutta schemes, which the paper says have higher convergence orders but cost more time app. D.4. The step size is not stated in the paper.

Forward pass for the proposed classifier

Pseudocode
# Forward pass of a proposed Neural SDE classifier  [§3.1] [§3.4] [eq. 9]
# Inputs: times t_0 .. t_n, observed values x_0 .. x_n, controlled path X(t)
# Output: class scores for one series

def classify(times, controlled_path, x0):
    # Initial latent state is an affine map of the first observation  [§3.1]
    z = h(x0)  # (d_z,)
    # Step from t_0 to T with explicit Euler and diagonal noise  [app. D.4]
    for t, dt, dW in brownian_steps(times):
        # Controlled map, used in place of z inside the drift  [eq. 10]
        z_bar = zeta(t, z, controlled_path(t))  # (d_z,)
        # Drift and diffusion of the chosen class  [eq. 4] [§3.1]
        drift = gamma(z_bar)
        diffusion = sigma(t)
        # Explicit Euler-Maruyama step  [app. D.4]
        z = z + drift * dt + diffusion * dW
    # Two-layer read-out of the terminal state  [eq. 9] [app. D.3]
    return F(z)

Training procedure

The paper's Algorithm 1 trains all four parameter groups jointly with cross-entropy and keeps the parameters with the lowest validation loss alg. 1. The text adds early stopping after 10 successive epochs without a validation-loss gain app. D.3.

Pseudocode
# Training loop for classification  [alg. 1] [app. D.3]
# Inputs: train set, validation set, max epochs
# Output: parameters with the lowest validation loss

def train(train_set, val_set, max_epochs):
    # Initialise the controlled map, drift, diffusion and read-out  [alg. 1]
    params = initialise_all_groups()
    best_params = params
    best_val_loss = infinity
    epochs_without_gain = 0
    for epoch in range(max_epochs):
        # Minimise cross-entropy on the training split  [alg. 1]
        for batch in train_set:
            loss = cross_entropy(model(batch), batch.labels)
            params = adam_step(params, loss)
        # Track the lowest validation loss and keep those parameters  [alg. 1]
        val_loss = evaluate(model, val_set)
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            best_params = params
            epochs_without_gain = 0
        else:
            epochs_without_gain = epochs_without_gain + 1
        # Early stopping after 10 epochs without a gain  [app. D.3]
        if epochs_without_gain >= 10:
            break
    return best_params

Assumptions and what is proved

  • Lipschitz continuity of each network, with constant L_s §3.2.
  • Linear growth, |s(t,x)| ≤ K_s(1 + |x|), when the last layer is ReLU or linear. A tanh or sigmoid last layer gives the bounded case |s| ≤ K_s instead §3.2 eq. 7.
  • Dissipativity: ⟨s(t,x), x⟩ ≤ −m|x|² + b for positive m and b §3.2. The paper states that convexity is not required §3.2.
  • Under the first two, the LSDE and LNSDE have unique strong solutions. For the GSDE with a tanh or sigmoid activation in γ, the solution is unique, almost surely non-negative, and state 0 is absorbing §3.2 app. A.
  • With ρ bounding the input perturbation, √E|x − x̃|² ≤ ρ eq. 8, and F Lipschitz with constant L_F, the paper gives bounds on the output distribution §3.3. The constants L_F, L_h (initial map), L_γ and K_γ are not given numerically. c_1 and c_2 are positive and independent of T, but no values are stated.

For the LSDE, with W_1 and W_2 the Wasserstein distances between the output distributions for x and x̃ §3.3:

W_1 \le \sqrt{3}\, L_F L_h c_1 e^{-c_2 T} \sqrt{5 + 2\mathbb{E}|x|^4 + 2\mathbb{E}|\tilde x|^4}\; \rho

W_2 \le \sqrt{2\sqrt{3} L_h c_1}\, L_F e^{-c_2 T/2} \left(5 + 2\mathbb{E}|x|^4 + 2\mathbb{E}|\tilde x|^4\right)^{1/4} \sqrt{\rho}

For the LNSDE, with constant σ_θ and |σ_θ|² > 2L_γ, and for the GSDE with |σ_θ|² > 2K_γ, for sufficiently large T §3.3:

W_1 \le L_F \exp\{-(|\sigma_\theta|^2 - 2L_\gamma) T/2\}(1+\rho)

The GSDE version replaces L_γ with K_γ. Both bounds hold for sufficiently large T, and the paper does not give the threshold §3.3 app. B.

Hyperparameters

NameValueWhereNotes
Layers in vector field n_lgrid {1, 2, 3, 4}app. Egrid search on validation; bold cells in the appendix tables mark the best per task
Hidden width n_hgrid {16, 32, 64, 128}app. Elatent width equals n_h in the classification code; latent size is not stated in the text
OptimiserAdam, learning rate 0.001app. EMortality, MuJoCo, Sepsis and Speech
Batch size64 (Mortality); 1024 (MuJoCo, Sepsis, Speech)app. Erobustness: 16 to 128, chosen by data size app. E model_run.py
Epochs300 Mortality; 500 MuJoCo; 200 Sepsis and Speechapp. Erobustness: 100 epochs for all models app. E
Weight decaynot stated in the papercommon_sde.pycode sets it to 0.01 × learning rate
Dropout10% in the read-out networkapp. D.3classification; the interpolation code also uses 0.1 in its encoder sde_interpolation.py
Early stopping10 epochs without validation-loss gainapp. D.3the code differs; see Paper vs code
Final-layer learning-rate scale×100app. D.2implemented in code as a gradient hook; see Paper vs code
Time encodingsinusoidal positional encodingapp. D.2
Output nonlinearitytanh on drift, diffusion and other vector fieldsapp. D.2
Noise structurediagonalapp. D.2
Solverexplicit Eulerapp. D.4step size not stated in the paper
Controlled pathHermite cubic with backward differencesapp. F common.pythe main text also names natural cubic splines §2.1
Interpolation objectivenormalised variational lower bound (ELBO)app. Ethe code uses a K = 5 importance-weighted bound; see Paper vs code
Missing rates0, 30, 50, 70%§4.2robustness study
Data split70 / 15 / 15app. Crobustness study; the split code in torch-ists is partly commented out
Uniform scalingseries rescaled to the longest lengthapp. Cvariable-length UEA/UCR datasets
Mortality data37 variables; 8000 instances; 50% to 90% observedapp. C §4.1interpolation
Sepsis data40,335 cases; 34 variables; AUROC§4.1 app. Cclassification, with and without observation intensity
Speech Commands10 of 35 words; 20-dimensional MFCC; length 161§4.1 app. Cclassification
MuJoCo10,000 Hopper simulations; 14 dimensions; 100 stepsapp. Cforecasting: 50 steps in, 10 out; 30, 50, 70% of values dropped

Code-only settings, not in the paper: interpolation runs use latent width 32, decoder width 64, K = 5 importance samples, KL warm-up and quantisation 0.016 run.sh. The classification code also uses ReduceLROnPlateau with patience 5 [code: benchmark_classification/common_sde.py#L120-L145].

Results

Author-reported. The robustness study reruns the baselines with the authors' source code and a Ray-based search app. E. The MuJoCo, Sepsis and Speech baseline numbers are quoted from prior papers, and the Mortality setup follows the mTAN pipeline app. E. Hardware: an Ubuntu 22.04 server with a Xeon Gold 6242 and six NVIDIA A100 40 GB GPUs §4.1 app. D.

Interpolation, PhysioNet Mortality (MSE × 10⁻³ at each observed fraction) tab. 1

Methods50%60%70%80%90%
RNN-VAE13.418 ± 0.00812.594 ± 0.00411.887 ± 0.00511.133 ± 0.00711.470 ± 0.006
L-ODE-RNN8.132 ± 0.0208.140 ± 0.0188.171 ± 0.0308.143 ± 0.0258.402 ± 0.022
L-ODE-ODE6.721 ± 0.1096.816 ± 0.0456.798 ± 0.1436.850 ± 0.0667.142 ± 0.066
mTAND-Full4.139 ± 0.0294.018 ± 0.0484.157 ± 0.0534.410 ± 0.1494.798 ± 0.036
LatentSDE8.862 ± 0.0368.864 ± 0.0588.686 ± 0.1228.716 ± 0.0328.435 ± 0.077
Neural SDE8.592 ± 0.0558.591 ± 0.0528.540 ± 0.0518.318 ± 0.0108.252 ± 0.023
Neural LSDE3.799 ± 0.0553.584 ± 0.0553.457 ± 0.0783.262 ± 0.0323.111 ± 0.076
Neural LNSDE3.808 ± 0.0783.617 ± 0.1293.405 ± 0.0893.269 ± 0.0573.154 ± 0.084
Neural GSDE3.824 ± 0.0883.667 ± 0.0793.493 ± 0.0243.287 ± 0.0703.118 ± 0.065

The proposed rows equal the grid cell n_l = 4, n_h = 128 at every level; see Pitfalls.

Classification, PhysioNet Sepsis (test AUROC and memory in MB, with and without observation intensity, OI) tab. 2

MethodsAUROC OIAUROC no OIMemory OIMemory no OI
GRU-Δt0.878 ± 0.0060.840 ± 0.007837826
GRU-D0.871 ± 0.0220.850 ± 0.013889878
GRU-ODE0.852 ± 0.0100.771 ± 0.024454273
ODE-RNN0.874 ± 0.0160.833 ± 0.020696686
Latent-ODE0.787 ± 0.0110.495 ± 0.002133126
ACE-NODE0.804 ± 0.0100.514 ± 0.003194218
Neural CDE0.880 ± 0.0060.776 ± 0.009244122
ANCDE0.900 ± 0.0020.823 ± 0.003285129
Neural SDE0.799 ± 0.0070.796 ± 0.006368240
Neural LSDE0.909 ± 0.0040.879 ± 0.008373436
Neural LNSDE0.911 ± 0.0020.881 ± 0.002341445
Neural GSDE0.909 ± 0.0010.884 ± 0.002588280

Classification, Speech Commands (test accuracy and memory in MB) tab. 3. The naive Neural SDE is left out because of training instability §4.1.

MethodsAccuracyMemory
RNN0.197 ± 0.0061905
LSTM0.684 ± 0.0344080
GRU0.747 ± 0.0504609
GRU-Δt0.453 ± 0.3131612
GRU-D0.346 ± 0.2861717
GRU-ODE0.487 ± 0.018171
ODE-RNN0.678 ± 0.2761472
Latent-ODE0.912 ± 0.0062668
Augmented-ODE0.911 ± 0.0082626
ACE-NODE0.911 ± 0.0033046
Neural CDE0.898 ± 0.025175
ANCDE0.807 ± 0.075180
LEAP0.922 ± 0.002391
Neural LSDE0.927 ± 0.0041187
Neural LNSDE0.923 ± 0.0011164
Neural GSDE0.913 ± 0.0011565

The LNSDE value here differs from the appendix; see Pitfalls.

Robustness to missing data, 30 UEA/UCR datasets (average accuracy with average standard deviation in parentheses, and average rank) tab. 4

MethodsRegularRank30%Rank50%Rank70%Rank
RNN0.582 (0.064)13.90.513 (0.087)15.30.485 (0.088)16.60.472 (0.072)15.6
LSTM0.633 (0.053)11.20.595 (0.060)12.10.567 (0.061)12.90.558 (0.058)12.5
GRU0.672 (0.059)8.30.621 (0.063)10.20.610 (0.055)10.30.597 (0.062)10.3
GRU-Δt0.641 (0.070)10.30.636 (0.066)8.90.634 (0.056)8.70.618 (0.065)10.2
GRU-D0.648 (0.071)10.30.624 (0.075)10.40.611 (0.073)11.10.604 (0.067)10.8
MTAN0.648 (0.080)12.00.618 (0.099)10.70.618 (0.091)10.10.607 (0.078)9.8
MIAM0.623 (0.048)11.00.603 (0.066)11.10.589 (0.063)12.20.569 (0.056)12.3
GRU-ODE0.671 (0.067)9.80.663 (0.064)9.50.666 (0.059)8.30.655 (0.062)7.8
ODE-RNN0.658 (0.063)9.10.635 (0.064)9.30.636 (0.067)8.20.630 (0.055)8.5
ODE-LSTM0.619 (0.063)11.40.584 (0.064)12.10.561 (0.065)13.30.530 (0.085)12.9
Neural CDE0.709 (0.061)8.20.706 (0.073)6.40.696 (0.064)6.50.665 (0.072)7.6
Neural RDE0.607 (0.071)13.90.514 (0.064)14.90.468 (0.068)15.20.415 (0.077)16.3
ANCDE0.693 (0.067)7.80.687 (0.068)7.20.683 (0.078)6.90.655 (0.067)7.1
EXIT0.636 (0.073)11.10.633 (0.078)10.20.616 (0.075)10.60.599 (0.075)11.1
LEAP0.444 (0.068)15.20.401 (0.078)16.30.425 (0.073)14.90.414 (0.070)14.7
Latent SDE0.456 (0.073)16.60.455 (0.073)15.40.455 (0.069)15.00.446 (0.066)15.1
Neural SDE0.526 (0.068)13.40.508 (0.066)13.10.517 (0.058)13.20.512 (0.066)12.9
Neural LSDE0.717 (0.056)5.60.690 (0.050)6.40.686 (0.051)6.10.682 (0.067)5.2
Neural LNSDE0.727 (0.047)5.40.723 (0.050)5.00.717 (0.054)4.30.703 (0.054)4.2
Neural GSDE0.716 (0.065)5.70.707 (0.069)5.30.698 (0.063)6.10.689 (0.056)5.3

Ablation, averaged over the 30 datasets tab. 5. The caption does not define the symbols O and X. Read against the text §4.3, O means the controlled path is included in the drift and X means it is not; N means a nonlinear network for σ and L an affine σ. Values are accuracy with standard deviation, and cross-entropy loss with standard deviation.

MethodζσAcc. uni (15)Loss uniAcc. multi (15)Loss multiAcc. all (30)Loss all
Neural CDE (l)——0.593 (0.094)0.809 (0.107)0.770 (0.037)0.595 (0.086)0.682 (0.065)0.702 (0.097)
Neural CDE (r)——0.550 (0.096)0.879 (0.090)0.716 (0.045)0.715 (0.105)0.633 (0.071)0.797 (0.098)
Neural CDE (n)——0.597 (0.083)0.795 (0.084)0.773 (0.042)0.590 (0.095)0.685 (0.063)0.692 (0.089)
Neural CDE (h)——0.611 (0.092)0.804 (0.119)0.777 (0.044)0.549 (0.101)0.694 (0.068)0.676 (0.110)
Neural SDEON0.615 (0.090)0.736 (0.089)0.760 (0.036)0.618 (0.092)0.688 (0.063)0.677 (0.091)
Neural SDEOL0.535 (0.095)0.838 (0.094)0.752 (0.040)0.633 (0.087)0.643 (0.068)0.736 (0.090)
Neural SDEXN0.516 (0.084)0.914 (0.069)0.515 (0.045)1.248 (0.085)0.516 (0.064)1.081 (0.077)
Neural SDEXL0.498 (0.087)0.929 (0.078)0.514 (0.050)1.255 (0.081)0.506 (0.068)1.092 (0.080)
Neural LSDEON0.604 (0.081)0.752 (0.084)0.783 (0.031)0.572 (0.080)0.694 (0.056)0.662 (0.082)
Neural LSDEOL0.533 (0.089)0.877 (0.086)0.745 (0.038)0.668 (0.085)0.639 (0.064)0.772 (0.085)
Neural LSDEXN0.530 (0.069)0.856 (0.060)0.527 (0.054)1.177 (0.082)0.528 (0.060)1.039 (0.074)
Neural LSDEXL0.505 (0.064)0.912 (0.061)0.518 (0.057)1.210 (0.101)0.512 (0.059)1.083 (0.088)
Neural LNSDEON0.654 (0.073)0.701 (0.091)0.780 (0.029)0.577 (0.070)0.717 (0.051)0.639 (0.080)
Neural LNSDEOL0.586 (0.087)0.765 (0.083)0.764 (0.044)0.617 (0.108)0.675 (0.066)0.691 (0.096)
Neural LNSDEXN0.532 (0.077)0.878 (0.060)0.502 (0.051)1.254 (0.082)0.517 (0.064)1.066 (0.071)
Neural LNSDEXL0.528 (0.085)0.890 (0.069)0.510 (0.047)1.257 (0.089)0.519 (0.066)1.074 (0.079)
Neural GSDEON0.633 (0.091)0.742 (0.086)0.772 (0.036)0.598 (0.077)0.703 (0.063)0.670 (0.081)
Neural GSDEOL0.572 (0.083)0.796 (0.084)0.748 (0.039)0.653 (0.094)0.660 (0.061)0.724 (0.089)
Neural GSDEXN0.531 (0.074)0.868 (0.052)0.510 (0.045)1.261 (0.093)0.520 (0.060)1.064 (0.072)
Neural GSDEXL0.525 (0.078)0.893 (0.072)0.509 (0.052)1.261 (0.093)0.517 (0.065)1.077 (0.083)

Forecasting, MuJoCo (test MSE with 0, 30, 50 and 70% of values dropped; memory in MB) app. E. Proposed rows and the strongest baselines only; the full table is in the appendix.

MethodsRegular30%50%70%Memory
ANCDE0.026 ± 0.0010.025 ± 0.0010.025 ± 0.0010.024 ± 0.00179
LEAP0.022 ± 0.0020.022 ± 0.0010.022 ± 0.0020.022 ± 0.001144
Neural SDE0.028 ± 0.0040.029 ± 0.0010.029 ± 0.0010.027 ± 0.000234
Neural LSDE0.013 ± 0.0000.014 ± 0.0010.014 ± 0.0000.013 ± 0.001249
Neural LNSDE0.012 ± 0.0010.014 ± 0.0010.014 ± 0.0010.014 ± 0.000273
Neural GSDE0.013 ± 0.0010.013 ± 0.0010.013 ± 0.0000.014 ± 0.000306

Cost: the authors report that their methods use more memory than the CDE-based methods, and that they need more compute than Neural CDE models §4.1 §5. In Table 2 the LNSDE uses 445 MB without observation intensity, against 122 MB for Neural CDE tab. 2. A runtime table on BasicMotions gives 89 to 96 seconds for 100 epochs for the proposed methods, against 102.5 seconds for Neural CDE app. D.4.

Figures: Figure 1 compares six diffusion functions on one dataset at a 50% missing rate fig. 1 §2.2. Figure 2 shows test loss over training for the proposed methods and the naive Neural SDE on four datasets, without early stopping fig. 2 §4.2. Figures 3 and 4 show per-dataset results for the 15 univariate and 15 multivariate datasets at four missing rates fig. 3 fig. 4 app. F.

Paper vs code

Official code at commit dbda6575f8e7150af357febf6ef00c9702183322, licensed MIT. The paper links the repository §4.1, and its owner is the first author's GitHub account. The README says the missingness-study pipeline has moved to the separate torch-ists repository and points to it as the up-to-date version [code: README.md].

  1. Drift and diffusion are clipped. The code applies tanh to the drift output and to the scaled diffusion [code: benchmark_classification/models_sde/neuralsde.py#L227-L231] [code: benchmark_classification/models_sde/neuralsde.py#L290-L307]. The text specifies tanh as a final operation app. D.2, so this clipping is in the paper too. The code's own docstring calls the time features and clipping stabilisers rather than the paper's parameterisation [code: benchmark_classification/models_sde/neuralsde.py#L1-L10].
  2. Diffusion is scaled by a learned sigmoid. The code multiplies the diffusion by sigmoid(θ), with θ initialised to 1.0 [code: benchmark_classification/models_sde/neuralsde.py#L163] [code: benchmark_classification/models_sde/neuralsde.py#L304-L307]. The paper has no such scale.
  3. Diffusion is non-negative. For the LSDE, LNSDE and GSDE, the code applies a ReLU to the output of the diffusion network [code: benchmark_classification/models_sde/neuralsde.py#L279-L282]. The paper's σ is an unrestricted network §3.1.
  4. Time features are fed to the diffusion. The paper's σ is a time network with sinusoidal encoding app. D.2. The code feeds sin(t) and cos(t) to every noise option that uses time [code: benchmark_classification/models_sde/neuralsde.py#L186-L193]. These features are periodic in t. The theory assumes σ(t) converges to a constant §3.3, which a periodic function can do only if it is constant.
  5. GSDE drift and diffusion are not the geometric form. The paper writes the GSDE as dz/z = γ dt + σ dW eq. 6. The code computes the drift as tanh(MLP(·) · tanh(y)), which replaces the factor z by tanh(y) and clips afterwards neuralsde.py. It computes the diffusion as tanh(sigmoid(θ) · ReLU(NN(t)) · y) [code: benchmark_classification/models_sde/neuralsde.py#L279-L282] [code: benchmark_classification/models_sde/neuralsde.py#L304-L307]. Nothing in the code enforces the positivity that the paper proves for the exact form §3.2.
  6. Read-out head has batch normalisation. The code's head is Linear, BatchNorm, ReLU, Dropout 0.1, Linear [code: benchmark_classification/models_sde/neuralsde.py#L58-L61]. The paper describes two fully connected layers with ReLU and 10% dropout app. D.3.
  7. Final-layer "×100 learning rate" is a gradient hook. The paper states layer-wise learning rates, for example ×100 app. D.2. The code multiplies the gradient of the final layer by 100 through a backward hook [code: benchmark_classification/sepsis-sde.py#L55-L56] [code: benchmark_classification/speech_commands-sde.py#L33-L34]. Under Adam a constant gradient scale mostly cancels in the update, so this is not a learning-rate multiplier. This is my reading of the code and Adam's update rule; I did not run it.
  8. Regulariser and weight decay. The code adds 0.01 times the norm of each vector-field parameter tensor to the loss [code: benchmark_classification/common_sde.py#L16-L22] [code: benchmark_classification/common_sde.py#L310], and uses Adam with weight decay 0.01 × learning rate common_sde.py. Neither is in the paper. Evaluation uses the same wrapped loss [code: benchmark_classification/common_sde.py#L268-L273].
  9. Learning-rate schedule. The classification code uses ReduceLROnPlateau with patience 5 common_sde.py. The paper gives a learning rate of 0.001 and no schedule app. E.
  10. Model selection differs. The paper keeps the parameters with the lowest validation loss alg. 1. The code keeps the checkpoint with the highest validation accuracy common_sde.py and restores it at the end common_sde.py.
  11. Early stopping differs. The paper stops after 10 epochs without a validation-loss gain app. D.3. The code stops after 50 epochs without a gain in training loss or in training accuracy [code: benchmark_classification/common_sde.py#L205-L212]. The training-loss criterion uses the loss that includes the regulariser.
  12. Sepsis positive-class weight. The code uses a positive weight of 10 in the binary cross-entropy sepsis-sde.py sepsis-sde.py. The paper does not state one §4.1.
  13. Initial state for Sepsis and Speech. The paper's z(0) is an affine map of the observations §3.1. The Sepsis code replaces it with a two-layer network over the static features, with 256 hidden units [code: benchmark_classification/sepsis-sde.py#L15-L29]. The Speech code uses the affine initial network applied to the spline value at the first time [code: benchmark_classification/models_sde/neuralsde.py#L63-L69] [code: benchmark_classification/models_sde/neuralsde.py#L146].
  14. Interpolation objective. The paper says ELBO app. E. The code draws K = 5 samples and optimises an importance-weighted bound with KL warm-up [code: benchmark_interpolation/sde_interpolation.py#L196-L209] [code: benchmark_interpolation/sde_interpolation.py#L149-L156] [code: benchmark_interpolation/run.sh#L10]. The optimiser has a cosine schedule with T_max 10 sde_interpolation.py. The run fixes the seed to 0 run.sh.
  15. Naive baseline is configured differently. The README maps the paper's "Neural SDE" to neuralsde_3_18 README.md. The classification code builds naivesde with input option 1 and noise option 18 [code: benchmark_classification/common_sde.py#L311-L317]. The interpolation default is neuralsde_0_18 sde_interpolation.py, and run.sh uses neuralsde_1_18 run.sh.
  16. Learning-rate range for the robustness study. The paper's grid is 10⁻⁴ to 10⁻¹ on a log scale app. E. The search code uses 10⁻⁴ to 10⁻² [code: torch-ists/param_search.py#L231].
  17. Robustness split is partly commented out. The paper gives 70/15/15 [app. C]. In torch-ists/model_run.py the split lines are commented out [code: torch-ists/model_run.py#L140-L142]. I did not verify which split runs.
  18. Scheduler for the robustness code. The training scheduler is StepLR with step 10 and gamma 0.5 model_run.py. The search uses step 5 param_search.py. The paper does not state a schedule.

No discrepancies were found in the MuJoCo settings: the paper's 500 epochs, learning rate 0.001 and batch 1024 app. E match the script [code: benchmark_forecasting/mujoco.sh#L11] mujoco-sde.py.

Pitfalls

  • Positivity is not carried over. The non-negativity proof for the geometric design uses an exponential change of variables §3.2 app. A. The code does not implement that transform, and item 5 of Paper vs code changes the drift and diffusion. Non-negativity is therefore not guaranteed by the code. I did not test it numerically.
  • Saturation of the diffusion. Because the code applies tanh after multiplying by y, the LNSDE and GSDE diffusions saturate to ±1 when |y| is large neuralsde.py. The multiplicative σz and z·σ structure of the paper is lost in that regime.
  • Bounded case versus the code. The paper's bounded case eq. 7 covers a tanh last layer, so a bounded drift is inside the theory. The code, however, squashes the drift and the scaled diffusion after the multiplicative terms neuralsde.py [code: benchmark_classification/models_sde/neuralsde.py#L290-L293]. The constant K_γ the paper defines for the network does not describe the code's effective drift without computing it from the code.
  • Theorem constants are missing. The bounds in §3.3 use L_F, L_h, L_γ, K_γ, c_1 and c_2, none of which are given numerically. The threshold on T is not stated. The bounds cannot be evaluated for a given model without estimating these constants.
  • Time scale. The periodic time features depend on the scale of the observation times neuralsde.py. The paper does not say how times are normalised for Sepsis or Speech. Check the dataset loaders before reusing the code.
  • Euler step size. The step is the minimum gap between observation times, floored at 10⁻³ neuralsde.py. The paper's own solver study ranks Euler below Milstein and SRK on convergence order app. D.4. Results can change with the grid of observation times.
  • Mortality grid versus Table 1. The proposed rows of Table 1 equal the grid cell n_l = 4, n_h = 128 at every level. The grid's bold cell at 50% for LSDE is 3.793 at n_l = 3, n_h = 128, lower than the 3.799 reported tab. 1 app. E. The text does not say which configuration Table 1 reports.
  • Speech LNSDE value. Table 3 gives 0.923 ± 0.001 tab. 3. The appendix gives 0.924 ± 0.000 for the same configuration (n_l = 1, n_h = 128, memory 1164 MB) app. E.
  • Sepsis no-OI Neural SDE value. Table 2 gives 0.796 ± 0.006 tab. 2. The appendix gives 0.797 ± 0.006 for n_l = 3, n_h = 32, with the same memory of 240 MB app. E.
  • Some baselines are quoted, not rerun. The MuJoCo, Sepsis and Speech baseline numbers come from prior papers app. E. Comparisons with numbers from other sources need care.
  • Memory and compute. The proposed methods use more memory than the CDE baselines in the sepsis test (445 MB against 122 MB) tab. 2 and need more compute overall §5.
  • Naive Neural SDE excluded from Speech. The paper leaves the naive Neural SDE out of the Speech table because of training instability §4.1, so that comparison is missing.
  • Tuning budget. The classification models are tuned on a grid of 16 cells. The robustness code draws 20 samples with ASHA and a 10-epoch budget [code: torch-ists/param_search.py#L225-L236]. The paper does not report the number of trials for the classification baselines app. E.
  • Table 5 symbols. The caption does not define O and X. The meaning used in Table 5 here is inferred from the text §4.3.

Lineage

  • Builds on: neural ODEs eq. 1 §2.1; neural CDEs with a spline-driven controlled path eq. 2 §2.1; neural SDEs with the Itô integral eq. 3 §2.1. The paper names earlier Neural SDE work by Tzen and Raginsky (2019), Liu et al. (2019) and Li et al. (2020, Latent SDE) §1. The Langevin and linear-noise designs with their stability analysis are the paper's own contribution §3.1 §3.3.
  • Compared against: RNN models, attention models (mTAN, MIAM), neural ODE variants (GRU-ODE, ODE-RNN, Latent-ODE, Augmented-ODE, ACE-NODE), neural CDE variants (ANCDE, EXIT, LEAP, Neural RDE) and Latent SDE §4 app. D.1.
  • Superseded by: not known from wiki content.

Source and license

Adapted from “Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data” by YongKyung Oh, Dong-Young Lim, Sungil Kim (arXiv:2402.14989), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/yongkyung-oh/Stable-Neural-SDEs at commit dbda6575f8e7 (MIT). Any quoted code is shown under that license.