Paper · 2021-09-29 · cs.RO, eess.SY
Lyapunov-stable neural-network control
Trains a neural-network controller and a neural-network Lyapunov function together. A mixed-integer program certifies the Lyapunov conditions on a box or returns worst-case counterexamples for retraining. Tested on a pendulum and 2D/3D quadrotors; the 2D controller stabilises 9,996 of 10,000 sampled starts versus 8,078 for LQR (sums from the authors' table).
Lyapunov functionNeural-network verificationBranch-and-boundcontrolverificationrobotics
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A neural-network controller has no built-in promise that a robot settles into its goal. This paper asks for one. It trains a second neural network, a "Lyapunov function", that acts like an energy measure: it is zero at the goal, positive nearby, and must drop at every step. If a controller and such a function are found together, stability is certified on the region checked.
The two networks are trained together. A solver checks the conditions over a small region of states. When it finds a state that breaks a condition, that worst case is fed back to adjust both networks. The method is shown on an inverted pendulum and on 2D and 3D quadrotors, and it is compared with the classical LQR controller.
TL;DR
Synthesises a neural-network controller and a neural-network Lyapunov function together, so that the closed loop is provably stable on a chosen region. Each check is a mixed-integer linear program, which is exact for ReLU-type networks but slow to solve as the network grows. Results are reported on an inverted pendulum and on 2D and 3D quadrotors (author-reported), with the 2D controller beating LQR on the share of sampled starts that converge.
Problem
Classical design gives stability guarantees for linear controllers (LQR) and for polynomial ones (sum-of-squares), but a neural-network controller usually comes with none §I. The paper assumes the system's forward dynamics are given as a leaky-ReLU network, and aims to find a controller and a Lyapunov function that satisfy the Lyapunov conditions on a compact set eq. 1a eq. 2a eq. 2b eq. 2c §II. The conditions must hold for all states in the region, not only on samples §I. The approach only applies to systems that a regular, locally Lipschitz controller can stabilise; the paper cites a unicycle as a case that needs a non-regular controller §IV-A.
Method
The method has four parts: a network-based forward model, a Lyapunov function with a 1-norm term, an exact verifier, and two ways to train the networks from the verifier's counterexamples. Each part is described below and all symbols follow the paper.
-
Forward model. The dynamics are approximated by a leaky-ReLU network φ_dyn with the update x[t+1] = φ_dyn(x, u) − φ_dyn(x*, u*) + x* and input bounds u_min ≤ u ≤ u_max eq. 1a eq. 1b §II. The paper states that this form covers arbitrary smooth systems by the universal approximation theorem §II. The official code uses a different update for second-order systems; see Paper vs code.
-
Lyapunov conditions. The target is V(x*) = 0 eq. 2c; V(x) > 0 on the set S for x ≠ x* eq. 2a; and V(x[t+1]) − V(x[t]) ≤ −ε2·V(x[t]) on S eq. 2b. Here S = {x : V(x) ≤ ρ} is a sublevel set and ε2 > 0 is given §II. The paper notes that the derivative condition constrains both the controller and V, because x[t+1] depends on π §II.
-
Lyapunov function. V(x) = φ_V(x) − φ_V(x*) + |R(x − x*)|₁ eq. 5. The subtraction makes V(x*) = 0 §IV-A. The 1-norm term is needed because a network-only V is linear near x* and can go negative there §IV-A fig. 4. R must have full column rank §IV-A. It is parameterised as R = U(Σ + diag(r₁², …, rₙ²))Vᵀ, with given orthonormal U and V, a given diagonal Σ with strictly positive entries, and free scalars rᵢ eq. 6. The smallest singular value of R is then strictly positive §IV-A.
-
Controller. The controller is u = π(x) = clamp(φ_π(x) − φ_π(x*) + u*, u_min, u_max) eq. 7 eq. 8 §IV-A. The shift by φ_π(x*) and u* makes the control at x* equal to u* §IV-A. Because clamp is piecewise affine, the closed loop is piecewise affine and can be encoded in a MIP app. B.
-
From strict to non-strict. The verifier handles only non-strict inequalities, so V(x) > 0 is replaced by V(x) ≥ ε1·|R(x − x*)|₁ with 0 < ε1 < 1 eq. 9 §IV-A. The paper proves in a lemma that this is also a necessary condition, not only a sufficient one app. C eq. 20 eq. 21.
-
Verifier. Over a bounded polytope B around x*, the verifier maximises the two violations: ε1·|R(x − x*)|₁ − V(x) eq. 10a, and V(x[t+1]) − V(x[t]) + ε2·V(x[t]) eq. 10b §IV-A. If both maxima equal 0, and are attained at x*, the Lyapunov condition is certified on B §IV-A. Otherwise the maximiser is a counterexample, and the worst one (largest violation) is taken §IV-A. The leaky-ReLU network is encoded with big-M constraints and a binary β per unit eq. 4a eq. 4b eq. 4c §III. The 1-norm and the clamp are encoded in app. B eq. 17 eq. 19. Branch-and-cut can return several counterexamples along the search tree §IV-A.
-
Trainer, approach 1 (counterexample sets). Losses are η1(x) = max(ε1·|R(x − x*)|₁ − V(x), 0) eq. 11a and η2(x) = max(V(f(x, π(x))) − V(x) + ε2·V(x), 0) eq. 11b. The surrogate loss is the p-norm of η over two growing sample sets X1 and X2 eq. 12. The paper names the 1-norm (mean), the ∞-norm (maximum) and the 4-norm as options §IV-B. Each MIP solve appends its counterexamples to X1 or X2, and batched gradient descent continues until the loss is zero or a maximum number of epochs is reached alg. 1 §IV-B.
-
Trainer, approach 2 (min-max). The method minimises the sum of the two MIP optima over θ, the parameters of π and V and the r-variables of R eq. 13 §IV-B. The gradient is taken after the inner MIP is solved: fix the binaries and keep only the active constraints, giving max c_θᵀs + d_θ subject to A_θ s = b_θ eq. 14a eq. 14b. The optimum has the closed form c_θᵀ A_θ⁻¹ b_θ + d_θ, which is differentiated by back-propagation §IV-B. The gradient is defined only if a small change in θ moves the continuous solution without changing the active set or the binaries §IV-B.
-
Region of attraction. After V is found, the verified box B is not itself a region of attraction, because it is not invariant §IV-C. The paper takes the largest sublevel set inside B, S = {x : V(x) ≤ ρ}, with ρ = min over the boundary ∂B of V(x) eq. 15 §IV-C. V is fixed, so only ρ is searched. The paper says the resulting set can be small, and suggests searching V and ρ jointly as future work §IV-C §VI.
Algorithm 1 (approach 1) and Algorithm 2 (approach 2), as described in the paper:
# Algorithm 1: counterexample-guided training [alg. 1]
# Input: candidate controller pi, candidate Lyapunov V, sets X1 and X2
# Output: pi and V, or "not converged"
def train_on_counterexamples(pi, V, X1, X2):
while True:
# Solve both verifier MIPs over the box B [eq. 10a] [eq. 10b]
positivity_obj, positivity_cex = solve_positivity_mip(pi, V)
derivative_obj, derivative_cex = solve_derivative_mip(pi, V)
if positivity_obj <= 0 and derivative_obj <= 0:
# Both maxima are 0: the Lyapunov conditions are certified
# [§IV-A]
return pi, V
if positivity_obj > 0:
# Add the positivity counterexamples to X1 [alg. 1]
X1 = X1 + positivity_cex
if derivative_obj > 0:
# Add the derivative counterexamples to X2 [alg. 1]
X2 = X2 + derivative_cex
# Batched gradient descent on the surrogate loss [eq. 12]
# until the loss is zero or the epoch budget is spent [alg. 1]
pi, V = descend_on_loss(pi, V, X1, X2)# Algorithm 2: min-max training through the MIP [alg. 2]
# Input: candidate controller pi, candidate Lyapunov V
# Output: updated pi and V
def train_min_max(pi, V):
while not converged(pi, V):
# Solve both verifier MIPs over the box B [eq. 10a] [eq. 10b]
positivity_obj, derivative_obj = solve_mips(pi, V)
if positivity_obj > 0 or derivative_obj > 0:
# Gradient of the MIP optima with respect to theta, with
# binaries and active constraints fixed [eq. 14a] [eq. 14b]
gradient = d_optimum_d_theta(pi, V)
# Gradient step on the min-max objective [eq. 13] [alg. 2]
theta = theta - step_size * gradientClosed loop and Lyapunov function as written in the paper:
# Controller with input clamp [eq. 7] [eq. 8]
def control(x):
# Network output shifted so that u(x*) = u* [eq. 7]
u_pre_clamp = phi_pi(x) - phi_pi(x_star) + u_star
# Clamp each input element to [u_min, u_max] [eq. 8]
return clamp(u_pre_clamp, u_min, u_max)
# Lyapunov function with 1-norm term [eq. 5] [eq. 6]
def lyapunov(x):
# Network term shifted to V(x*) = 0 [eq. 5]
network_term = phi_V(x) - phi_V(x_star)
# Full-column-rank R from the SVD-style parameterisation [eq. 6]
R = U @ (Sigma + diag(r ** 2)) @ V_mat.T
return network_term + norm_1(R @ (x - x_star))Hyperparameters
The paper states few numbers. Values in the code are given with their location, and they are not in the paper unless marked. The code's demo scripts set most of them; each demo runs in float64 and uses a time step of 0.01 train_pendulum_demo.py train_quadrotor_2d_demo.py train_quadrotor_demo.py. The time step is not stated in the paper.
| Name | Value | Where | Notes |
|---|---|---|---|
| ε1 (positivity margin) | 0 < ε1 < 1; no value stated | eq. 9 §IV-A | Pendulum 0.5 train_pendulum_demo.py; 2D 0.1 train_quadrotor_2d_demo.py; 3D 0.1 train_quadrotor_demo.py |
| ε2 (derivative decay) | ε2 > 0; no value stated | §II eq. 2b | Code 0.001 with ExpLower in all three demos train_pendulum_demo.py. Code's lyapunov_positivity_epsilon is the paper's ε1 and lyapunov_derivative_epsilon is ε2 train_lyapunov_barrier.py |
| λ (weight on the 1-norm in V) | not in paper; Eq. 5 has weight 1 | eq. 5 | Code: 0.8 pendulum train_pendulum_demo.py; 0.9 2D train_quadrotor_2d_demo.py; 0.6 3D train_quadrotor_demo.py |
| R | Full column rank; no value stated | eq. 5 eq. 6 §IV-A | Pendulum: fixed 4×2 stack of two rotation matrices, unless --search_R train_pendulum_demo.py. 2D: LQR Riccati matrix S plus 0.01·I train_quadrotor_2d_demo.py. 3D: fixed 15×12 matrix train_quadrotor_demo.py |
| Verified box B | Pendulum: 0 ≤ θ ≤ 2π, −5 ≤ θ̇ ≤ 5 §V-A. 2D: [−0.75, −0.75, −0.5π, −4, −4, −2.75] to the negated upper bound §V-B. 3D: not stated | §V-A §V-B | The demo boxes differ; see Paper vs code |
| Input limits u_min, u_max | Not stated | eq. 1b | Pendulum [−20, 20] train_pendulum_demo.py; 2D [0, 8] train_quadrotor_2d_demo.py; 3D [0, 3 × hover thrust] train_quadrotor_demo.py |
| Dynamics network (hidden layers) | Pendulum (5, 5); 2D (7, 7); 3D (10, 10) | tab. III | Code: pendulum (5, 5) train_pendulum_demo.py; 2D (6, 6) train_quadrotor_2d_demo.py; 3D width is a command-line argument with no default train_quadrotor_demo.py |
| Controller network (hidden) | Pendulum (2, 2); 2D (6, 4); 3D (16, 8) | tab. III | Code: pendulum (3, 2) train_pendulum_demo.py; 2D (6, 4) train_quadrotor_2d_demo.py; 3D (8, 6) train_quadrotor_demo.py |
| Lyapunov network (hidden) | Pendulum (8, 4, 4); 2D (10, 10, 4); 3D (16, 12, 8) | tab. III | Code: pendulum (8, 8, 6) train_pendulum_demo.py; 2D (10, 10, 4) train_quadrotor_2d_demo.py; 3D (14, 8, 4) train_quadrotor_demo.py |
| Leaky slope c | 0 ≤ c < 1; no value stated | §III | Dynamics 0.01 (pendulum, 2D), 0.1 (3D). Controller 0.1 (pendulum), 0.01 (2D, 3D). Lyapunov 0.1 in all three demos |
| Optimiser and learning rate | Not stated | alg. 1 alg. 2 | Code default Adam, learning rate 0.003 train_lyapunov_barrier.py. Pre-training uses learning rate 0.001 train_pendulum_demo.py |
| Maximum iterations | Not stated | alg. 1 alg. 2 | Code default 1000 train_lyapunov_barrier.py; demos 5000 (pendulum, 2D) and 10000 (3D) train_quadrotor_demo.py |
| Convergence tolerances | Order 1e-6 app. E | app. E | Code default positivity 1e-6, derivative 3e-5 train_lyapunov_barrier.py. Demos use derivative 1e-5 train_pendulum_demo.py and positivity 5e-6 for 2D and 3D train_quadrotor_2d_demo.py |
| Dynamics fit error | MSE about 1e-6 for most examples and about 1e-4 for 3D | §V-C | Pendulum dynamics trained for 100 epochs, batch 20, learning rate 0.001 train_pendulum_demo.py |
| MIP solver | Gurobi | §V | Version not stated |
| Sample margins and weights | Not in paper | — | Code: sample weight 1, MIP weight 10, sample margin 0, sample pool 500 train_lyapunov_barrier.py train_lyapunov_barrier.py |
Results
Author-reported. Setup: a Gurobi MIP solver, run on an Intel Xeon CPU §V. Pendulum and 2D and 3D quadrotors are simulated with the Lagrangian dynamics for checks against the learned model §V-A §V-B §V-C.
| Setting | Metric | Value | Source |
|---|---|---|---|
| Inverted pendulum, small box 0.8π ≤ θ ≤ 1.2π, −1 ≤ θ̇ ≤ 1 | Convergence time | A few minutes, both algorithms | §V-A |
| Inverted pendulum, large box 0 ≤ θ ≤ 2π, −5 ≤ θ̇ ≤ 5 | Convergence time | Within 3 hours, both algorithms | §V-A |
| Inverted pendulum | Behaviour | Controller swings up and stabilises on the true Lagrangian model; many starts outside the verified region also converge | §V-A fig. 8 |
| 2D quadrotor, 10,000 sampled initial states, Lagrangian simulation | States stabilised (sums derived from the table) | LQR 8,078; neural network 9,996 (8,078 + 1,918) | tab. I |
| 2D quadrotor | States where LQR fails and the network succeeds | 1,918 | tab. I |
| 2D quadrotor | States where the network fails and LQR succeeds | 0 | tab. I |
| 2D quadrotor, small box | Convergence time | 20 minutes, both algorithms | §V-B |
| 2D quadrotor, larger box | Convergence time | 1 day, both algorithms | §V-B |
| 3D quadrotor | Training time | 3 days | §V-C |
| Average time over 10 runs | Algorithm 1 (pendulum, 2D, 3D) | 8.4 s, 948.3 s, timed out after 5 days | tab. II |
| Average time over 10 runs | Algorithm 2 (pendulum, 2D, 3D) | 224 s, 1004.7 s, 65.7 h | tab. II |
| 3D quadrotor | Lyapunov decrease | Decreases along network-dynamics trajectories; can increase along Lagrangian ones | fig. 12 app. D |
The table reports averages over 10 runs. Two Algorithm 1 runs on the 2D quadrotor timed out after 6 hours and are excluded. For the 3D quadrotor only 3 runs were made tab. II. The authors say Algorithm 1 fluctuates more than Algorithm 2 on the 2D loss curves, but that the comparison is not conclusive fig. 13 §V-C. The authors report the 2D controller stabilising more states than LQR, and attribute this to piecewise-linear control and to input limits that LQR ignores §V-B tab. I.
Paper vs code
The official repository is StanfordASL/neural-network-lyapunov, MIT licence,
at the pinned code_commit. Code paths below are relative to its root. The
paper and code differ in several places, and the code's demos are the source of
the values in the Hyperparameters table.
-
Dynamics form. The paper's update is x[t+1] = φ(x, u) − φ(x*, u*) + x* for the full state eq. 1a §II. For the pendulum the code uses a second-order update. The network gives the velocity v[t+1] = φ(q, v, u) − φ(q*, v*, u*), and the position uses the midpoint rule relu_system.py relu_system.py. The 2D demo uses a residual form, v[t+1] − v[t] = φ(x̄, u) − φ(x̄*, u*), with a partial state x̄ relu_system.py. The 3D quadrotor network takes (rpy, angular velocity, four rotor inputs), so 10 inputs, and returns 9 outputs train_quadrotor_demo.py. The class docstring describes a thrust input instead quadrotor.py. Check this before you reuse the model.
-
Network sizes. Table III disagrees with the demo scripts. Pendulum: controller (2, 2) in the paper and (3, 2) in code train_pendulum_demo.py, Lyapunov (8, 4, 4) in the paper and (8, 8, 6) in code train_pendulum_demo.py. 2D: dynamics (7, 7) in the paper and (6, 6) in code train_quadrotor_2d_demo.py. 3D: controller (16, 8) in the paper and (8, 6) in code train_quadrotor_demo.py, Lyapunov (16, 12, 8) in the paper and (14, 8, 4) in code train_quadrotor_demo.py. The 2D controller (6, 4) and Lyapunov (10, 10, 4) match tab. III.
-
Verified boxes. The paper's pendulum box is 0 ≤ θ ≤ 2π, −5 ≤ θ̇ ≤ 5 §V-A. The pendulum demo uses π ± 0.1π and ±0.5 train_pendulum_demo.py. The paper's 2D box is ±0.75, ±0.75, ±0.5π, ±4, ±4, ±2.75 §V-B. The 2D demo uses ±0.7, ±0.7, ±0.5π, ±3.75, ±3.75, ±2.5 train_quadrotor_2d_demo.py. The paper does not say which box produced Table I or Figure 10.
-
Trainer modes. The default demo path is one trainer,
train. It sums sample hinge losses with weight 1 and MIP objectives with weight 10 train_lyapunov_barrier.py train_lyapunov_barrier.py. The pendulum demo passes an empty initial sample set train_pendulum_demo.py. Counterexample pools, which resemble the paper's approach 1, are switched on only in the--train_adversarialbranch train_pendulum_demo.py train_quadrotor_2d_demo.py. That branch adds its own schedule: batches per epoch, epochs per MIP and pool sizes train_lyapunov_barrier.py. It is not the paper's Algorithm 1 as written. -
Optimiser. Algorithm 2 is written as gradient descent alg. 2. The code uses Adam by default train_lyapunov_barrier.py, and differentiates the MIP objective with fixed binaries and active constraints through
compute_objective_from_mip_data_and_solutionwithpenalty=1e-13train_lyapunov_barrier.py. -
MIP encoding. The paper's leaky-ReLU encoding is the plain big-M form eq. 4a eq. 4b eq. 4c. The code adds strengthened big-M cuts, chosen by an index set and added when violated relu_to_optimization.py. The 3D demo exposes options for extra strengthening points train_quadrotor_demo.py.
-
Region of attraction. The paper's ρ = min over ∂B of V eq. 15. The code's
compute_region_of_attractioninstead minimises V at states inside the box whose one-step successor leaves it, and at states outside whose successor enters it. It also takes a V upper bound lyapunov.py. No demo calls it outside the tests. The demos therefore do not compute ρ as the paper defines it. -
Scale of V. The paper's Eq. 5 has coefficient 1 on the 1-norm, with the scale absorbed into R eq. 5. The code adds λ·|R(x − x*)|₁ with λ between 0.6 and 0.9 lyapunov.py.
-
Parameterisation of R. The paper's SVD-style parameterisation eq. 6 matches
SearchRwithSVDOptionsr_options.py. The pendulum's--search_Rflag uses a different SPD parameterisation instead r_options.py. The demos use a fixed R unless--search_Ris set train_quadrotor_2d_demo.py. -
Termination. The paper says the tolerance is of order 1e-6 app. E. The demos use 1e-5 on the derivative MIP and 5e-6 on the positivity MIP for 2D and 3D train_quadrotor_2d_demo.py.
-
LQR baseline. The paper does not give the LQR weights used in the comparison. The 2D demo uses Q = diag(10, 10, 10, 1, 1, length/(2π)) and R = [[0.1, 0.05], [0.05, 0.1]] train_quadrotor_2d_demo.py. The 3D demo uses Q = diag(10 × 6, 1 × 6) and R = I₄ train_quadrotor_demo.py. Whether the paper's baseline used these is not stated.
-
Code-only pre-training. The demos add supervised pre-training, which the paper does not describe. For the 2D demo, LQR control and value targets are sampled and fitted, with 100000 samples train_quadrotor_2d_demo.py. The pendulum demo uses energy-shaping plus LQR trajectories as controller data train_pendulum_demo.py.
-
Model constants. The paper cites Mellinger for the 3D model and gives no constants. The code uses mass 0.468, gravity 9.81 and inertia (4.9e-3, 4.9e-3, 8.8e-3) quadrotor.py.
Pitfalls
-
Strict versus non-strict. The verifier only certifies V(x) ≥ ε1·|R(x − x*)|₁, which is a non-strict condition. It implies the strict one only when R has full column rank and 0 < ε1 < 1 §IV-A app. C. With a rank-deficient R, the 1-norm is zero on a subspace through x*, so the certificate says nothing there. Check the rank of R, not only its shape.
-
Certification depends on tolerances. The certificate is "both MIP optima ≤ tolerance", not an exact zero alg. 1 app. E. The paper gives no solver tolerances. The code's convergence thresholds are 1e-6 to 3e-5 train_lyapunov_barrier.py. Optima near the threshold depend on the Gurobi version and settings, which the paper does not give.
-
Guarantee is for the learned model. The proof is for the network dynamics, not the true system. The 3D model has a larger fit error, about 1e-4 MSE versus about 1e-6 for the other examples §V-C. The authors note that the Lyapunov function can increase along the true Lagrangian trajectories app. D fig. 12. Re-verify on the true dynamics before deploying.
-
Verifier cost. The number of binaries grows linearly with the number of ReLU units. Worst-case MIP time grows exponentially in the number of binaries §VI. Table II shows Algorithm 1 timing out on 3D, and Algorithm 2 needing 65.7 hours tab. II. The authors note the verified region was kept small to speed this up tab. II.
-
Input limits matter. The controller is clamped inside the verification, so changing u_min or u_max changes the result. The paper gives no numbers for the quadrotors eq. 1b. Take them from the code, and check the units train_quadrotor_demo.py.
-
Gradient for approach 2. The gradient is undefined where a small change in θ changes the active set or the binaries §IV-B. The paper does not say what the training loop does at such points.
-
Verified box is not the basin. The verified box B is not the region of attraction §IV-C. The usable region is the sublevel set inside B, which the paper says can be small. Do not report the box as the basin.
-
Hidden choices. The paper does not state the optimiser, learning rates, batch sizes, dynamics-fit epochs, the time step, or the R used for the 2D and 3D runs. The code fixes all of these (see Hyperparameters).
-
Naming trap in the code. In the code
lyapunov_positivity_epsilonis ε1 and its comment reads ε₂ train_lyapunov_barrier.py. Map the parameters by their role, not by name. -
Numerics. The demos run in float64 train_pendulum_demo.py. V is a difference of network outputs, so values near x* are small differences of larger numbers. This is our inference; the paper has no precision analysis. Keep float64 when porting, or test float32 against the 1e-6 tolerances first.
Lineage
- Builds on: the Lyapunov-function and neural-network-verification lines the paper cites. Chang et al. learn neural Lyapunov functions with tanh activations §I. Abate et al. use quadratic activations §I. MIP verification of ReLU networks is cited to Tjeng et al. and to Dai et al. §I §III. None of these are compiled here.
- Compared against: an LQR controller on the 2D quadrotor §V-B tab. I.
- Related future work named by the authors: searching V and ρ jointly §IV-C, and tighter MIP formulations §VI.
Source and license
Adapted from “Lyapunov-stable neural-network control” by Hongkai Dai, Benoit Landry, Lujie Yang, Marco Pavone, Russ Tedrake (arXiv:2109.14152), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/StanfordASL/neural-network-lyapunov at commit 15de7cd47d09 (MIT). Any quoted code is shown under that license.