---
kind: concept
slug: differentiable-simulation
title: Differentiable simulation
summary: >-
  A simulator whose gradients drive a fit to data or a training loss. PhysDreamer fits a per-particle Young's modulus to a generated video. Physics-aware training takes gradients from a fitted digital model of a real device. TrafficSim unrolls a learned traffic policy in closed loop. Choose by whether you can simulate the system or must measure it. Results are author-reported, from three cards.
tags: [physics-simulation, inverse-problems, material-estimation, differentiable-rendering, physical-neural-networks, traffic-simulation]
papers: ["2101.06557", "2104.13386", "2404.13026"]
related: [gaussian-splatting, diffusion-models, learned-simulator, robot-simulation, hardware-aware-training, surrogate-modeling, motion-prediction, imitation-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

A differentiable simulator is a model of a process whose final state can be
differentiated with respect to its parameters, such as stiffness, mass or an
initial velocity. The gradient can then come from any loss computed on the
result. The shared idea is that the simulator sits inside the optimisation
loop, so its derivatives drive the update, not only its forward output
[arxiv:2404.13026 §4] [arxiv:2104.13386 app. S1] [arxiv:2101.06557 §3.4].

The three cards in this corpus differ in where the forward pass comes from.
PhysDreamer simulates it and differentiates through the simulator. The loss
is computed in image space: the simulated particles are rendered from a camera
and compared with a reference clip [arxiv:2404.13026 §4] [arxiv:2404.13026
eq. 10]. Physics-aware training (PAT) measures the forward pass on a real
device, and uses a differentiable digital model only for the backward pass.
Its loss is computed from the measured outputs [arxiv:2104.13386 app. S1].
TrafficSim runs a learned policy forward in time and trains that policy
through the rollout. Its "simulator" is therefore a learned model over vehicle
boxes, not a physics engine [arxiv:2101.06557 §3.3]. Because PhysDreamer fits
the material rather than setting it by hand, its variant also answers "how do
I estimate the physical parameters of an object from observations?"
[arxiv:2404.13026 §4].

## Variants

### Material-point simulation with a video-fitted stiffness field (PhysDreamer)

The object is a set of 3D Gaussians, each treated as a simulation particle.
The particles follow a fixed corotated elastic model and are advanced with the
material point method (MPM) [arxiv:2404.13026 §4.1] [arxiv:2404.13026 eq. 1].
Young's modulus is a learned field over the particles, and the initial
velocity is a second learned field [arxiv:2404.13026 §4.2]. The reference is
a 14-frame clip from an image-to-video diffusion model, started from one still
render of the object [arxiv:2404.13026 §5.2].

The fit runs in two stages. The first fits the initial velocity with the
stiffness frozen. The second fits the stiffness with the velocity frozen, and
its gradients reach only the previous frame, to limit unstable gradients
[arxiv:2404.13026 §4.2]. The loss mixes an L1 term and a D-SSIM term, weighted
0.1 and 0.9, and adds a total-variation penalty to keep the fields smooth
[arxiv:2404.13026 eq. 10] [arxiv:2404.13026 eq. 11].

To make this affordable, MPM runs on a smaller set of driving particles, found
by K-Means. Each Gaussian follows the rigid transform fitted to its 8 nearest
driving particles [arxiv:2404.13026 §4.3] [arxiv:2404.13026 alg. 1]. After the
fit, a new force or initial velocity can be applied and the scene rendered from
any viewpoint [arxiv:2404.13026 §5.1].

Reported setup: eight real static scenes, five of them flowers, with a
two-alternative forced-choice study of 100 participants [arxiv:2404.13026 §5.1].
Against PhysGaussian, the authors report 80.8% of judgements prefer their
motion [arxiv:2404.13026 tab. 1]. Card:
[PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](/papers/2404.13026).

### Physics-aware training with a fitted digital backward model (PAT)

The forward pass runs on the physical system. Each physical stage is wrapped
as an autodiff function, and its backward pass uses the derivatives of a
differentiable digital model, evaluated at the inputs that were actually
measured [arxiv:2104.13386 app. S1]. The error vector at the output is
therefore exact, and the Jacobians are taken at the true intermediate signals
[arxiv:2104.13386 app. S1]. The authors chain several stages into one network
and train the chain with the same update [arxiv:2104.13386 app. S1].

The digital model is a fitted neural network. Its training data are
input-output pairs measured on the device, usually about 10⁴ samples, and its
architecture is chosen by a search [arxiv:2104.13386 app. S2]. The authors also
fit a noise model to repeated measurements [arxiv:2104.13386 app. S2].

Reported setups, author-reported: three physical systems (an optical crystal,
a metal plate and an electronic circuit) and a hybrid network that combines
optical stages with trainable digital layers [arxiv:2104.13386 fig. 2]
[arxiv:2104.13386 fig. 4] [arxiv:2104.13386 app. S2]. Vowel classification
with five optical stages reaches 93% test accuracy, against about 40% at best
for training in simulation [arxiv:2104.13386 fig. 2C-D] [arxiv:2104.13386
fig. 3B-C]. On MNIST, the metal plate reaches 87%, the electronic network 93%
and the hybrid network 97%, where digital operations alone reach about 90%
[arxiv:2104.13386 fig. 4D] [arxiv:2104.13386 fig. 4H] [arxiv:2104.13386
fig. 4L]. Card:
[Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](/papers/2104.13386).

The paper's experiments used Adadelta, and its learning-rate schedules are
only partly given [arxiv:2104.13386 app. S2]. The authors report that penalty
terms on the controls and inputs were needed to keep them in range
[arxiv:2104.13386 app. S2].

### Closed-loop unrolling of a learned traffic policy (TrafficSim)

The simulator is a learned policy rolled forward in time. At each tick it reads
the map and the recent boxes of every actor, samples one scene-level latent
variable, and applies the first few steps of a joint plan for all actors
[arxiv:2101.06557 §3.3]. No physics is modelled. The gradient that matters
flows through the sampled actions, using the reparameterisation trick, so the
policy learns how an early decision changes the states that come after it
[arxiv:2101.06557 §3.4].

Training runs this policy in closed loop for 12 s and computes a loss at every
simulated step. Labels cover only the first 8 s. In that part the imitation
term is the CVAE evidence lower bound, with posterior samples conditioned on
the recorded future. After that point, and on prior samples throughout, a
collision penalty is used instead [arxiv:2101.06557 §3.4]. The collision term
is multiplied by 0.01 to match the scale of the imitation term [arxiv:2101.06557
app. A]. The two terms are mixed with a weight that changes over time
[arxiv:2101.06557 eq. 3]. The Huber δ, the KL weight β and the number of
future steps P are not stated [arxiv:2101.06557 §3.4].

Reported setup, author-reported: ATG4D, 15 samples per scenario, and the
scenario collision rate and traffic-rule violation rate as the main metrics
[arxiv:2101.06557 tab. 1] [arxiv:2101.06557 §4]. Scenario collision rate is
0.50%, against 2.90% for the next-best model, while traffic-rule violation is
2.77%, against 0.25% for IDM [arxiv:2101.06557 tab. 1]. Training open-loop
instead raises the collision rate to 5.92% [arxiv:2101.06557 tab. 2]. The
authors call closed-loop training with backpropagation through the simulation
the most important component [arxiv:2101.06557 §4.2]. Card:
[TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](/papers/2101.06557).
No official code is known for it, and the paper does not say whether ATG4D is
public [arxiv:2101.06557 §4].

## When to use which

The three variants answer different questions. The table compares them with
each other and with the routes the papers describe, and names the constraint
that decides between them.

| Situation | Use | Why |
|---|---|---|
| Elastic, collision-free object, no measured material data, new forces needed | PhysDreamer | Material is fitted from video. Paper's stated scope is elastic objects without collisions [arxiv:2404.13026 §6]. |
| Material values known or set by hand | MPM on Gaussians with hand-set values, as in PhysGaussian | The paper describes that baseline as needing hand-set material values [arxiv:2404.13026 §5.1]. This corpus has no card for it, so it is not evaluated here. |
| Replay a generated motion, no new forces | Motion replay without physics, as in DreamGaussian4D | The paper says it cannot extrapolate to new forces [arxiv:2404.13026 §5.3]. |
| Real-time or interactive use | Not PhysDreamer | About one minute per second of video on one V100 GPU [arxiv:2404.13026 §6]. |
| Single camera view only | PhysDreamer with one view, the paper's default setup | In one scene (alocasia), two-view supervision was preferred 81.0% for visual quality and 86.0% for motion realism [arxiv:2404.13026 §5.4]. That is one scene, so it is not a general gain. |
| Need a physically validated material | Not PhysDreamer on its own | The 2AFC study measures perceived realism, not physical accuracy [arxiv:2404.13026 §5.1]. |
| Real device you can run, but no accurate model of it, and the parameters must work on the device | PAT | The loss uses measured outputs, and the digital model only supplies gradients. The paper argues that errors in a digital model compound through the layers and fail on the device [arxiv:2104.13386 app. S1]. |
| Device runs cannot be bought in volume | Not PAT, or a smaller run | The SHG operating rate is 200 Hz, and training needs millions of executions. One million executions take about 5,000 seconds of machine time [arxiv:2104.13386 app. S2]. |
| Need to fit the digital model only from about 10⁴ device measurements | PAT, as the paper does | The paper's usual digital-model data size is about 10⁴ uniform samples [arxiv:2104.13386 app. S2]. The paper does not test smaller sets. |
| Labelled trajectories cover only part of the rollout you need | TrafficSim-style unrolling: imitation where labels exist, collision penalty after | The paper's labels cover 8 s of a 12 s rollout, and the loss switches term at the label horizon [arxiv:2101.06557 §4] [arxiv:2101.06557 §3.4]. |
| Compounding error in a learned rollout is the main failure | Unroll in training, not open-loop | Open-loop training gives a 5.92% collision rate against 0.50% with unrolling. This ablation is on one dataset [arxiv:2101.06557 tab. 2]. |
| Rollout must be cheap per step | Coarser inference frequency, with a cost | At 2 Hz one run takes 0.83 s on a GTX 1080 Ti; at 0.5 Hz it takes 0.24 s, and the collision rate rises from 0.50% to 0.96% [arxiv:2101.06557 tab. 4]. |
| Hard collision avoidance needed at run time | Rejection sampling, or gradient steps on the latent | Gradient steps lower the collision rate to 0.12%, but traffic-rule violation rises from 2.77% to 3.00% [arxiv:2101.06557 tab. 5]. |
| Physical forces, materials or contact dynamics matter | Not TrafficSim | Its rollout moves boxes under a learned policy, with no physics model [arxiv:2101.06557 §3.3]. Use PhysDreamer or PAT. |

## Implementation sketch

The first loop is the PhysDreamer variant, written from the equations and the
algorithm in the paper. The second is the update the variants share: the
forward pass decides what the loss sees, and the gradient comes from a model of
the process. Where the paper gives no value, the comment says so.

```pseudocode
# Fit a per-particle stiffness field to one reference clip  [arxiv:2404.13026 §4]
def fit_stiffness(particles, reference_frames):
    # Stage 1: fit the initial velocity with stiffness frozen  [arxiv:2404.13026 §4.2]
    stiffness = random_per_particle_stiffness(particles)
    velocity = init_velocity_field()
    for step in range(n1):  # count not stated  [arxiv:2404.13026 §4.2]
        frames = simulate(particles, velocity, stiffness)  # MPM  [arxiv:2404.13026 §4.1]
        # first 3 frames only  [arxiv:2404.13026 §4.2]
        loss = frame_loss(frames[:3], reference_frames[:3])
        velocity = optimizer_step(velocity, loss)  # optimiser not stated
    # Stage 2: fit stiffness with velocity frozen  [arxiv:2404.13026 §4.2]
    for step in range(n2):  # count not stated  [arxiv:2404.13026 §4.2]
        frames = simulate(particles, velocity, stiffness)
        loss = frame_loss(frames, reference_frames)
        loss = loss + total_variation(stiffness)  # weight not stated  [arxiv:2404.13026 eq. 11]
        stiffness = optimizer_step(stiffness, loss)
    return stiffness, velocity
```

```pseudocode
# Shared update: the forward pass sets the loss, the model sets the gradient
# [arxiv:2404.13026 §4] [arxiv:2104.13386 app. S1] [arxiv:2101.06557 §3.4]
def physics_update(inputs, targets, params, forward, model, eta):
    # PhysDreamer: forward is an MPM simulation. PAT: forward is the device.
    # TrafficSim: forward is a learned policy rollout.
    outputs = forward(inputs, params)
    grad_out = loss_gradient(outputs, targets)
    # PhysDreamer: model is the simulator itself. PAT: a fitted network.
    # TrafficSim: the policy's sampled actions, via reparameterisation.
    # Jacobians at the measured inputs  [arxiv:2104.13386 app. S1]
    grad_params = vjp(model, inputs, params, grad_out)
    return params - eta * grad_params
```

```pseudocode
# Per-frame rendering loss  [arxiv:2404.13026 eq. 10]
def frame_loss(rendered, reference):
    # L1 weight 0.1, D-SSIM weight 0.9  [arxiv:2404.13026 eq. 10]
    return 0.1 * l1(rendered, reference) + 0.9 * dssim(rendered, reference)
```

```pseudocode
# Lamé parameters from Young's modulus and Poisson's ratio  [arxiv:2404.13026 eq. 2]
def lame(E, nu):
    mu = E / (2.0 * (1.0 + nu))
    lam = E * nu / ((1.0 + nu) * (1.0 - 2.0 * nu))
    return mu, lam
```

## Contradictions

Within the PhysDreamer paper, the sub-step size is stated two ways. §4.2 gives
a sub-step of about 1e-4 for most experiments, and §5.2 gives 4.34e-5 s per
sub-step with 768 sub-steps per frame [arxiv:2404.13026 §4.2]
[arxiv:2404.13026 §5.2]. At 768 sub-steps these give about 77 ms and 33 ms per
frame, so the paper does not say which one the reported runs used
[arxiv:2404.13026 §5.2].

On the hat scene, PhysGaussian has a lower Fréchet Video Distance than the
authors' method (50 against 54) [arxiv:2404.13026 app. A]. The authors'
averages across scenes favour their method, so the per-scene result is worth
checking before you rely on the average.

The PhysDreamer paper's realism results compare against real captured video.
Against real video, the authors report 53.7% for motion realism, averaged over
seven scenes, because no real video exists for the red rose
[arxiv:2404.13026 tab. 1] [arxiv:2404.13026 §5.3]. For one scene, 86% of users
judged the simulated motion more realistic than the real capture, and the
authors explain this by MPM producing slower motion [arxiv:2404.13026 §5.3].

Within the PAT paper, the in silico baseline is described two ways. The
caption of its depth figure calls in silico training "barely distinguishable
from random guessing" for final accuracy at every depth [arxiv:2104.13386
fig. 3C]. The text says it reaches about 40% at its best [arxiv:2104.13386
§Results]. This page does not resolve the difference.

Within the TrafficSim paper, the imitation weight is described two ways. The
text says it must fall to zero at the end of the labelled part [arxiv:2101.06557
app. A]. The printed formula, min((T_label − t)/T_label, 0), is zero up to
that point and negative after it, so it does not give that schedule
[arxiv:2101.06557 app. A]. The paper's collision loss is also printed as
max(1, Σ L_pair). Read literally, that cannot depend on the actors' positions,
while the description calls for a loss between 0 and 1 [arxiv:2101.06557 §3.4].
This page does not resolve these. An implementation has to pick a schedule and
a collision loss, and should record the choice.

Across the three papers, the metrics differ: perceived motion realism and
Fréchet Video Distance for PhysDreamer, classification accuracy for PAT, and
collision and rule-violation rates for TrafficSim. No cross-paper contradiction
is known, and their numbers should not be compared directly.

## Papers

- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](/papers/2101.06557): trains a learned multi-actor traffic policy by unrolling it in closed loop, with an imitation loss on labelled steps and a collision penalty.
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](/papers/2404.13026): fits a per-particle Young's modulus to a generated video by differentiating an MPM simulation through differentiable rendering.
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](/papers/2104.13386): runs the forward pass on physical hardware and takes gradients from a differentiable digital model of each stage.