Learning to Generate Rigid Body Interactions with Video Diffusion Models

In plain words

Video generators can turn one photo into a short clip, but they often get physics wrong: objects fly off, vanish, or never touch what they should hit. KineMask lets you say how fast one object should start moving. You mark that object with a coloured mask, and the colours give the direction and speed. The model then works out what happens next, for example a second object being knocked over or liquid spilling after a collision.

The model learns this from simple computer-made scenes of boxes and cylinders that move and collide. Training first gives it the mask for every frame. Then the masks for most later frames are removed, so that at the end the model only has to work from the first push. A text description of the expected outcome can be added too. A vision-language model writes these descriptions for the training clips, and a large language model writes one for a new photo.

The authors report that the same method, attached to three different video models, makes their motion and collisions look more plausible than the original models on their tests.

TL;DR

KineMask is a ControlNet-style branch that conditions an image-to-video diffusion model on per-object velocity masks and an optional text description §3.1 §3.2. It is trained in two stages on synthetic Blender scenes, the second one with most mask frames dropped §3.2 §3.3. The authors report gains over CogVideoX-5B, Wan2.2-5B and Cosmos2.5-2B on their synthetic test sets and in a 30-person user study §4.2 §4.3 tab. 4. No code was released at compile time (see Paper vs code). Resolution, sampler, step count and guidance scale are not stated.

Problem

Video diffusion models produce plausible frames but often miss rigid-body physics: objects fly, disappear, or fail to collide §1.

Earlier controls each fall short for this task:

• Physics-simulator pipelines (WonderPlay, PhysGen, PhysGen3D) need explicit scene reconstruction and a simulator §2.3.
• Drag-based methods (for example TORA and MotionI2V) need a full target trajectory, which already states where the object ends up §1 §2.2.
• Force Prompting uses a force and a point, but does not model object interactions §2.3.

The paper asks two questions: can a video diffusion model produce realistic collisions between rigid bodies from initial dynamic conditions alone, and how do the training data and text conditioning affect causal effects §1? The setting is one input image plus one user-given velocity for the target object at the first frame §3.3.

Method

Backbone and control branch. The video model p_θ is an image-to-video diffusion model conditioned on a text description c and a reference image y §3.1 eq. 1 eq. 2. KineMask adds a ControlNet branch ψ_φ that encodes a dense control tensor u; the backbone parameters θ stay frozen and only φ is trained §3.1 eq. 3 eq. 4.

Control signal. The control is a tensor m of shape F × H × W × 3 §3.2. Each object has a segmentation mask. Its three colour channels encode the instantaneous velocity along x, y and z, painted over the object's pixels §3.2. Only objects that move in the first frame are annotated; objects that are moved by collisions get a blank mask §3.2. The paper uses the velocity of each frame, v_t, rather than the initial velocity v_0 for every frame [app. Velocity encoding].

Stage one. Train φ' on the full per-frame masks m, minimising the control loss L_ctrl over videos, first frames, masks and captions §3.2 eq. 4 eq. 5. The paper describes the loss as a KL divergence and says it is implemented as noise prediction §3.1.

Stage two. Fine-tune φ' into φ'' with a truncated mask m_⊙. Frames whose index is in the kept set f* keep their mask; every other frame is set to zero §3.2 eq. 6 eq. 7. The kept frames are sampled non-uniformly, biased toward earlier frames, because collisions in the simulated clips happen mostly at the start §4.1 app. Velocity encoding.

Training data. Blender scenes contain boxes and cylinders on textured surfaces. Each controlled object gets a random direction and magnitude. The rendered video is x_0, its first frame is y, and the per-frame masks are m §3.3. The caption c comes from the Tarsier vision-language model, prompted with "Describe the video in a concise way, covering all motions and collisions between objects." app. Used prompts. Training uses 10,000 clips per synthetic set §4.1.

Inference. From one unseen image y:

• SAM2 gives a mask of the target object §3.3.
• The user draws an arrow, which is turned into RGB masks automatically §3.3.
• Only the first frame keeps its mask, so m_⊙ = {m_0, 0, ..., 0} §3.2 §3.3.
• GPT-5 writes an effect description c_infer from in-context examples §3.3 app. Used prompts.
• Start from x_T drawn from N(0, 1) and generate 49 frames §3.3 §4.1.

Ablation settings. For the simple-motion and low-level control experiments, training and inference both use the fixed prompt "An object moving on a surface" §4.3. Text conditioning is then switched on and off in the table below tab. 3.

Figures. fig. 2 is the pipeline: stage one and two on the left, inference on the right. fig. 1 is the teaser. fig. 3 compares with CogVideoX on real scenes. fig. 4 shows direction, speed and object choice. fig. 6 shows the effect of training on Simple Motion versus Interactions. fig. 7 shows the mass-dependent outcome. fig. 8 shows multi-object control. fig. 9 shows transfer to hand-object scenes. fig. 11 shows the effect of text. fig. 13 shows the same method on Wan and Cosmos. The paper's appendix has more examples app..

# Stage one: fit the ControlNet on full per-frame velocity masks  [§3.2] [eq. 5]
# The backbone stays frozen; only phi is updated  [§3.1] [eq. 4]
def train_stage_one(phi, backbone, dataset):
    for video, first_frame, masks, caption in dataset:
        # Draw a noise level and a noisy copy of the video  [§3.1]
        t, noisy = add_noise(video)
        # Control branch reads all mask frames  [§3.2]
        control = phi(masks)
        # Noise-prediction loss on the controlled backbone  [§3.1] [eq. 4]
        loss = noise_loss(backbone(noisy, t, first_frame, caption, control))
        phi = update(phi, loss)
    return phi

# Stage two: keep only some mask frames, then fine-tune  [§3.2] [eq. 6] [eq. 7]
# Initialised from the stage-one model; kept set f* is non-uniform  [§4.1]
def train_stage_two(phi, backbone, dataset):
    for video, first_frame, masks, caption in dataset:
        # Choose the frames whose masks are kept (f*)  [§3.2] [app. Dropout]
        kept = sample_kept_frames(num_frames=len(masks))
        # Zero the masks of all other frames  [eq. 6]
        truncated = [
            masks[f] if f in kept else zeros_like(masks[f])
            for f in range(len(masks))
        ]
        t, noisy = add_noise(video)
        loss = noise_loss(
            backbone(noisy, t, first_frame, caption, phi(truncated))
        )
        phi = update(phi, loss)
    return phi

# Inference from one image and one user arrow  [§3.3]
# Only the first frame carries a velocity  [§3.2] [eq. 6]
def generate(image, arrow, num_frames=49):
    # Segment the target object with SAM2  [§3.3]
    object_mask = segment_object(image)
    # Turn the arrow into an RGB velocity mask  [§3.3]; mapping not stated
    first_mask = arrow_to_rgb_mask(object_mask, arrow)
    truncated = [first_mask] + [zeros_like(first_mask)] * (num_frames - 1)
    # Predict the outcome in words with GPT-5  [§3.3] [app. Used prompts]
    caption = describe_effects(image, arrow)
    # Sample from N(0, 1) and denoise conditioned on everything above  [§3.3]
    noise = sample_gaussian(shape_of_video(num_frames))
    return sample_video(noise, image, truncated, caption)

Hyperparameters

Results

All numbers are author-reported. Metrics are MSE, FVD, FVMD (called FMVD in the tables) and IoU of object masks from SAM2 §4.1. The Real World image set has no ground-truth motion §4.1.

Two-stage training, Simple Motion tab. 1 §4.3

Training data, Interactions test set tab. 2 §4.3

Text conditioning, Interactions test set tab. 3 §4.3

Other backbones, Interactions test set tab. 4 app. Test on different model backbones

Real-world videos tab. 5 app. Additional Quantitative Comparison. The text says these PhysicsIQ and lab videos give the quantitative comparison; the number of clips is not stated.

Encoding and sampling tab. 6 app. Velocity encoding app. Dropout density. Interactions test set.

User study, real-world scenes §4.2 fig. 5 fig. 12. 30 participants compared pairs of videos on motion fidelity, interaction realism and overall physical consistency. The text says KineMask is preferred on all three questions against all baselines. The per-question percentages appear only in the figures, not in the text. A user study on Wan and Cosmos reports the same ordering against each original model fig. 12 §4.3.

Qualitative. The paper shows collisions with other objects, liquid spilling after a collision, steam, and glass breaking after a fall §1 §4.2 fig. 3 fig. 13 fig. 22. It reports failures for low objects that do not collide and for scenes with many similar objects, which can cause duplication or disappearance [app. Failure Cases] fig. 21.

Paper vs code

No official code released. The paper's project page links github.com/Daromog/KineMask, which I treated as the authors' repository. It is not verified against the author list. The repository at commit e5faa135a611dbdb0aae5ea1aabdd3a45452cc8f holds only a README and an index page. The README's heading reads "Code Comming Soon", and the repository has no LICENSE file and no model, training or inference code. The README points to a Hugging Face checkpoint repository and a dataset; I did not inspect either. Its badge links to arXiv 2402.10698, a different id from this paper. Nothing in the repository can be compared with the paper's values.

Searches: the paper text, the project page, the GitHub repository search for "KineMask" (which also returned bermudezarii/KineMaskPage and bermudezarii/KineMaskWebsite; both appear to be project sites, not code, and were not cloned). The arXiv metadata has no comments field.

Pitfalls

• No code to check against. Every value below comes from the paper text or appendix. Anything missing is "not stated" in Hyperparameters.
• Stage-two wording. The text calls f* the "masked frame indices", but eq. 6 keeps the mask on frames in f* and zeroes the rest §3.2. Read f* as the kept frames.
• Frame 0 during training. Inference always keeps the first-frame mask §3.2 §3.3. The paper does not say whether frame 0 is always in f* during training.
• Kept-frame distribution. The dropout ratio and the non-uniform distribution are not given. The paper says only that frames near the start are favoured §4.1 app. Dropout density.
• Stage-two equation. Eq. 7 minimises over φ'' but writes the loss with φ' §3.2. The sensible reading is: initialise φ'' from φ' and fine-tune it.
• Loss form. The objective is written as a KL divergence eq. 2 eq. 4 and implemented as noise prediction §3.1. The loss is not written in its noise-prediction form, and no per-timestep weighting is stated.
• Steps per stage. The 1000 steps are given once in the appendix. They are not split between the two stages, and the checkpoint used for evaluation is not stated.
• Velocity encoding. The paper does not define how a velocity maps to RGB values: the scale, normalisation, and the sign convention for negative velocity are all unstated. The arrow-to-mask mapping is also unstated §3.2 §3.3.
• ControlNet downscaling factor 8. The appendix gives the value but not what it downsamples app. Training Hyperparameters.
• Wan checkpoint. The baseline is "Wan2.2-I2V-5B" in §4.1, the appendix names "Wan-AI/Wan2.2-TI2V-5B", and §4.3 writes "Wan2.2-5B". Check which checkpoint you use.
• Claims about "all" baselines. §4.2 says KineMask "outperforms all baselines". In tab. 5, Force Prompting has the best Physics-IQ MSE, and the fine-tuned CogVideoX has the best FVMD on both real sets. KineMask has the best FVD and IoU on both sets, and the best MSE on the authors' own data. Treat the claim as limited to some metrics.
• Metric names. The text says FVMD and the tables say FMVD. The paper treats them as the same metric §4.1 tab. 2.
• IoU sensitivity. The paper notes IoU changes a lot with small displacements, while the other metrics do not §4.3. Compare it with care.
• Data limits. Captions in training are tied to the synthetic objects, so their vocabulary is narrow §4.3. Simple Motion training fails to produce collisions §4.3 fig. 6.
• Hidden costs. Each generation calls SAM2 and GPT-5, and the paper's own runtime estimate appears only in a commented-out line of the source, so it is not counted here.
• Limits stated by the authors. Only velocity is controlled (no mass, friction, shape or air drag). Soft bodies and part-level actuation are out of scope. Only static cameras are used §5.

Lineage

• Builds on: CogVideoX-I2V-5B as backbone, and the ControlNet branch design §3.1 §4.1.
• Compared against: CogVideoX (zero-shot and fine-tuned), Wan2.2 image-to-video, TORA, MotionI2V, Force Prompting §4.2 app. Baseline implementation details; PhysGen3D in qualitative comparison app. Additional Qualitative Comparison.
• Other work the paper names as related: WonderPlay, PhysGen, PhysGen3D, C-Drag, InterDyn, Goal Force §2.3.
• Superseded by: not stated.