PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

In plain words

A 3D model of an object, such as a flower, shows how it looks but not how it would bend or sway when pushed. PhysDreamer fills that gap. It takes one still picture of the object, asks a video generator to animate it, and treats that clip as a guess about how real things of this kind move.

It then searches for a stiffness value for each small piece of the object so that a physics simulation of the object moves like the guessed clip. Once that search has run, the object can be pushed or hit in ways the video never showed, and the simulation reacts in a plausible way. The authors judge the result mainly by asking people which of two clips looks more realistic.

TL;DR

• Makes a static 3D Gaussian object interactive by estimating a spatially varying Young's modulus from a generated video §4 fig. 3.
• Uses a fixed corotated elastic model, MPM simulation of the Gaussians, and differentiable rendering to fit the material and an initial velocity §4.1 §4.2.
• Author-reported 2AFC study on eight real scenes: 80.8% prefer it over PhysGaussian on motion realism, 63.5% over DreamGaussian4D tab. 1.
• Relevant if you need elastic, force-responsive 3D objects without measured material data. Limited to elastic objects without collisions, and it is slow §6.

Problem

Synthesising how an object moves after a new interaction needs its physical material properties, such as stiffness. These are hard to measure for real objects, and the material can vary across one object §1. Dynamics that are generated without interaction, or that are text-conditioned, do not solve this §1.

Two earlier routes fall short. PhysGaussian runs MPM on 3D Gaussians but needs material values set by hand §5.1. DreamGaussian4D animates Gaussians from a generated video, but it does not simulate physics, so it cannot extrapolate to new forces §5.3.

The method assumes elastic materials and excludes collisions §6. It assumes a single constant Poisson's ratio §3.

Method

The idea is to render the static scene from one viewpoint, animate that image with an image-to-video model to get a reference clip, and then optimise a material field and an initial velocity field so that a differentiable simulation, rendered from the same viewpoint, matches the clip §4 fig. 3.

1. Reference video. Render the Gaussian scene, background included, from one viewpoint to get I_0. Stable Video Diffusion animates it into 14 frames with a small motion bucket, so the motion is mostly the object's and little of it is camera motion §4 §5.2.

2. Particles and material model. Treat each Gaussian as a simulation particle with volume, mass, position, velocity, deformation gradient and velocity gradient. Use the fixed corotated hyperelastic energy §4.1 eq. 1.

3. Lamé parameters. Compute \mu and \lambda from Young's modulus E and Poisson's ratio \nu §4.1.

   \mu = \frac{E}{2(1+\nu)}, \quad \lambda = \frac{E\nu}{(1+\nu)(1-2\nu)} eq. 2

4. MPM transfers. Scatter mass and momentum from particles to a grid eq. 4. Gather grid velocity back to particles and update the velocity gradient and the deformation gradient eq. 5 eq. 6. The particle mass is m_p = \rho V_p, with V_p from the background cell volume divided by the particle count in that cell §3.

5. Fields. Model E(\mathbf{x}) and the initial velocity \mathbf{v}_0(\mathbf{x}) as two triplanes, each followed by a three-layer MLP §4.2 §5.2.

6. Two-stage fit. Stage 1 freezes random per-particle E and fits the initial velocity on the first three frames of the reference. Stage 2 freezes the velocity and fits E. In stage 2, gradients reach only the previous frame, to avoid exploding or vanishing gradients §4.2.

7. Loss. Compare each rendered frame with the reference frame, using a weighted sum of an L1 term and a D-SSIM term with \lambda = 0.1 eq. 10. Add a total-variation penalty on the triplane features to encourage smooth fields eq. 11.

   L^t = \lambda L_1(\hat I^t, I^t) + (1-\lambda) L_{\text{D-SSIM}}(\hat I^t, I^t) eq. 10

   L_{\text{tv}} = \sum_{i,j} \|\mathbf{u}_{i+1,j} - \mathbf{u}_{i,j}\|_2^2 + \|\mathbf{u}_{i,j+1} - \mathbf{u}_{i,j}\|_2^2 eq. 11

8. Subsampling for speed. Run MPM on K-Means driving particles, not on all Gaussians §4.3. For each Gaussian, take its 8 nearest driving particles at t=0, fit a rigid transform from their t=0 positions to their current positions, and apply it to the Gaussian's position and rotation §4.3 alg. 1 app. C.

9. Interaction. After the fit, apply a new external force or initial velocity and simulate, then render from any viewpoint §5.1 fig. 5.

# Fit material and initial velocity to one reference video  [§4]
# Inputs:  gaussians; reference frames I_0 .. I_13
# Output:  Young's modulus field E; initial velocity field v0

def fit_material(gaussians, reference_frames):
    # Stage 1: E is random per particle and frozen; fit v0  [§4.2]
    youngs = random_initial_youngs(gaussians)
    velocity_field = init_triplane_mlp()
    for step in range(num_stage1_steps):  # count not stated  [§4.2]
        simulated = simulate(gaussians, velocity_field, youngs)
        loss = frame_loss(simulated[:3], reference_frames[:3])
        velocity_field = optimizer_step(velocity_field, loss)  # not stated
    # Stage 2: v0 frozen; fit E, gradients reach the previous frame  [§4.2]
    for step in range(num_stage2_steps):  # count not stated  [§4.2]
        simulated = simulate(gaussians, velocity_field, youngs_field)
        loss = frame_loss(simulated, reference_frames)
        youngs_field = optimizer_step(youngs_field, loss + tv_term)  # [eq. 11]
    return youngs_field, velocity_field

# Per-frame rendering loss  [eq. 10]
def frame_loss(simulated_frame, reference_frame):
    # L1 term on rendered pixels, weight lambda = 0.1  [eq. 10]
    l1 = mean(abs(simulated_frame - reference_frame))
    # D-SSIM term, weight 1 - lambda = 0.9; exact definition not stated
    d_ssim = dssim(simulated_frame, reference_frame)  # [eq. 10]
    return 0.1 * l1 + 0.9 * d_ssim

# One MPM sub-step on the simulated particles  [§4.1] [eq. 4] [eq. 5]
def mpm_substep(x, v, F, C, mass, volume, E, nu, dt):
    # Lame parameters from E and nu  [eq. 2]
    mu = E / (2.0 * (1.0 + nu))
    lam = E * nu / ((1.0 + nu) * (1.0 - 2.0 * nu))
    # Fixed corotated stress from F  [eq. 1]
    stress = fixed_corotated_stress(F, mu, lam)
    # Particle-to-grid: mass and momentum, including stress  [eq. 4]
    grid_mass, grid_momentum = scatter_p2g(
        x, v, C, mass, volume, stress, dt,
    )
    # Grid velocity is momentum over mass; apply stationary cells  [§5.2]
    grid_v = grid_momentum / grid_mass
    grid_v = apply_dirichlet_mask(grid_v)
    # Grid-to-particle: velocity and velocity gradient  [eq. 5]
    v, C = gather_g2p(grid_v, x)
    # Deformation gradient update  [eq. 6]
    F = (identity + dt * grid_gradient(grid_v, x)) @ F
    x = x + dt * v
    return x, v, F, C

# Render Gaussians from K-Means driving particles  [§4.3] [alg. 1]
def render_frame(gaussians, drive_t0, drive_t, k=8):
    # k nearest driving particles of each Gaussian at t = 0  [§4.3]
    neighbours = knn_indices(gaussians.x_t0, drive_t0, k)
    # Rigid transform from the t = 0 neighbours to their current positions
    rotation, translation = fit_rigid(
        drive_t0[neighbours],
        drive_t[neighbours],
    )  # [§4.3] [alg. 1]
    # Move each Gaussian and rotate its covariance  [alg. 1]
    x = gaussians.x_t0 + translation
    covariance = rotation @ gaussians.covariance @ rotation.T
    return rasterize(x, gaussians.alpha, covariance, gaussians.color)

Key settings are in the Hyperparameters table below. The three pseudocode blocks above were written from the equations and alg. 1. Where the paper is silent they say so.

Hyperparameters

Results

Author-reported. Setup: eight real static scenes (five flowers, an alocasia plant, a telephone cord and a beanie hat) §5.1. Baselines are PhysGaussian and DreamGaussian4D, with real captured videos as a reference §5.1. Judgements are two-alternative forced choice (2AFC) by 100 Prolific participants §5.1. Compute cost is about one minute on an NVIDIA V100 per second of generated video §6.

Qualitative results are in fig. 5 (material fields and force responses) and fig. 6 (space-time slices against baselines and real video). Two-view supervision is shown in fig. 7. The two-view numbers appear only in the text §5.4.

Paper vs code

The paper is arXiv v2 (2024-10-07) LaTeX source. The code is the official repository linked from the project page, at commit code_commit. Its training code sits in projects/uncleaned_train/, which the authors' own folder name marks as uncleaned. The inference demo is in projects/inference/.

• Rendering loss. Paper: L1 term weighted 0.1 plus D-SSIM weighted 0.9 §4.2 eq. 10. Code: 0.5 \times MSE weighted (1 - \text{ssim}), plus \text{ssim} \times (1 - \text{SSIM}), with ssim defaulting to 0.9 train_material.py train_material.py. The weights match. The pixel term is MSE, not L1.
• Regularisation. Paper: total-variation term on both triplanes, weight not stated §4.2 eq. 11. Code: plane TV of both fields weighted by tv_loss_weight (default 1e-4), velocity smoothness multiplied by 10, and an entropy term that is off by default train_material.py train_material.py train_material.py train_material.py.
• Stage 1 length. Paper: initial velocity fitted on the first three frames §4.2. Code: the window starts at 6 frames by default and is scheduled over training, with a warm-up of 800 steps in one branch train_material.py train_material.py. These are not the same schedule.
• Gradient flow. Paper: in stage 2, gradients reach only the previous frame §4.2. Code: compute_window defaults to 1 and grad_window to 14, with loss_decay 1.0 (no decay) train_material.py train_material.py. The mechanism differs and was not traced in detail.
• Young's modulus. Paper: per-particle random initial value, frozen in stage 1, value not stated §4.2. Code: a global scalar starts at 1e5 train_material.py (a log-uniform draw in [1e-3, 1e3] appears at #L369; this card did not trace which branch runs); a per-particle network output is multiplied by 1000 train_material.py; the result is clamped to [1e3, 5e8] train_material.py. The scalar's learning rate is scaled by 1e-10, so it is effectively frozen train_material.py. The clamp is not in the paper.
• Poisson's ratio. Paper: one constant, value not stated §3. Code: the setup builds \nu = 0.3 train_material.py. Initialisation also samples \nu uniformly from [0.1, 0.4] and makes it trainable train_material.py train_material.py. This card did not trace which value reaches the solver.
• Density. Paper: constant \rho, value not stated §3. Code: 2000 kg/m³ fast_train_velocity.py; stem points are set to 2000 train_material.py.
• Velocity scale. Paper: nothing stated §4.2. Code: the velocity field output is multiplied by 0.1 train_material.py.
• Sub-steps. Paper: 768 sub-steps between frames §5.2. Code: configs use 768 for alocasia and carnation, 384 for the hat and 256 for the telephone cord alocasia.py carnation.py hat.py telephone.py.
• Driving-particle ratio. Paper: 10 to 50 times fewer driving particles §5.2. Code: downsample_scale defaults to 0.04 in training train_material.py and is 0.1 in the carnation config carnation.py. This card did not trace how the scale maps to the count.
• Interpolation. Paper: a rigid transform is fitted over the 8 nearest driving particles §4.3. Code (inference): one function averages the neighbours' displacements for translation only local_utils.py; another fits a rigid transform local_utils.py. This card did not determine which one the demo calls.
• Checkpointing. Paper: state checkpointing with recomputation is used §5.2. Code: checkpoint_steps defaults to -1, meaning none train_material.py.
• Triplane and grid sizes. Consistent: 8^3 material field train_material.py, 24^3 velocity field train_material.py, 64^3 grid train_material.py.

Pitfalls

• Unstated training values. The paper gives no optimiser, learning rate, iteration counts, stage lengths, density, Poisson value, TV weight or E range. The training code defaults to a learning rate of 1e-3 train_material.py and 200 iterations train_material.py, with AdamW and weight decay 1e-4 and a linear warm-up schedule train_material.py. These are code defaults, not values the paper reports.
• Hidden scaling and clamps. The factor of 1000 on the network output, the clamp to [1e3, 5e8] and the 0.1 velocity scale are not in the paper. Each one changes the effective range that the optimiser sees. Units for E and \rho are not stated in the paper; the code's comment for density reads kg/m³ fast_train_velocity.py.
• Sub-step size conflict inside the paper. §4.2 gives \Delta t \approx 1 \times 10^{-4} for most experiments. §5.2 gives 4.34e-5 s per sub-step at 768 sub-steps per frame. At 768 sub-steps, the first value gives about 77 ms per frame and the second about 33 ms (derived). The paper does not say which applies to the reported runs. The code's configs use 768, 384 and 256 sub-steps carnation.py, so check the sub-step size against the code before implementing.
• Algorithm 1 is ambiguous. It updates x in place and applies x + t_sim without showing whether x is the rest or current position. §4.3 says the transform is applied to the initial position and rotation. The card's pseudocode reads it as the rest position.
• Stage 1 window conflict. §4.2 says three frames. The code starts at six frames and changes the window over training. An implementation should use the paper's three frames unless it deliberately matches the code.
• Cost. About one minute on a V100 per second of video §6. Practical use needs the subsampling in §4.3, and it still does not run in real time. Gradient checkpointing is needed for memory §5.2, but the code's default turns it off.
• Manual setup. The object must be picked out from the background and stationary parts fixed by hand §6. Collisions are excluded §6.
• Selection of generated video. The compiled text says a small motion bucket is used §5.2 but does not describe how many videos were generated or how one was chosen.
• What the evaluation measures. The 2AFC study measures perceived realism, not physical accuracy §5.1. Against real captured videos, the average is 53.7%, taken over seven scenes because no real video exists for the red rose tab. 1. For the alocasia scene, 86% of users rated the simulated motion more realistic than the real capture §5.3. The paper's explanation is that MPM tends to produce slower, lower-frequency motion on thin leaves, and that people may prefer slow motion §5.3. Treat this as a perception result, not a validation of material values.
• FVD is not uniformly better. On the hat, PhysGaussian has a lower FVD (50 vs 54) app. A. The FVD and FID tables use seven scenes and omit the red rose.
• Two-view numbers only in text. The 81.0% and 86.0% figures for two-view supervision appear in §5.4 and not in any table in the compiled paper §5.4.
• Licence and provenance of the code. The repository has no LICENSE file, so its licence is not stated. The README says parts come from three other projects, graphdeco-inria/gaussian-splatting, zeshunzong/warp-mpm and PingchuanMa/NCLaw README.md. Their licences were not checked here. No code is quoted in this card; it is cited by path and line only.
• No physical ground truth. Material values are never compared with measured data in the paper. Evaluation rests on user judgements and on FVD and FID against captured videos §5.1 app. A.

Lineage

• Builds on: 3D Gaussian Splatting for the representation (cited as gaussiansplat2023 in the paper; the bibliography gives no arXiv id) §4. PhysGaussian, arXiv:2311.12198, for MPM on Gaussians and the internal filling step §4.1 §5.2. Stable Video Diffusion, arXiv:2311.15127, to produce the reference video §5.2.
• Compared against: PhysGaussian, arXiv:2311.12198, with material values set by hand §5.1. DreamGaussian4D, arXiv:2312.17142, which replays a generated motion without physics §5.1 §5.3.
• Superseded by: not stated.