Real-time Deep Dynamic Characters

In plain words

The paper builds a digital copy of a real person that you can steer with a skeleton, the stick-figure skeleton used in animation. Its clothes move the way the real clothes would: a skirt swings when the person spins. The model is learned from many synchronised videos of the person moving, and it needs no 3D scans of the moving body during training.

The model has two parts. One part predicts how the surface bends and folds, including loose fabric, from the recent movement of the skeleton. The other part predicts the colour pattern on the surface, which changes with the motion and with the viewing angle, so shiny spots and wrinkles look right. Because the whole thing is differentiable, training can compare rendered images with the real video frames and adjust the model directly. The authors report that this runs fast enough for interactive use.

TL;DR

• Person-specific character: a scanned, rigged template plus two learned networks.
• Geometry is predicted coarse-to-fine: an embedded graph first, then per-vertex displacements, both from a short window of skeleton motion.
• Training uses only multi-view silhouettes and images, with a differentiable renderer and a lighting fit; no 3D ground truth §3.1 §3.4.
• A UNet predicts a 1024×1024 dynamic texture from normal maps and camera encoding §3.6.
• Author-reported: 38 fps inference §4; 90.70% AMVIoU on DeepCap S4 test, above MVBL's 88.14% tab. 2.

Problem

Classic skinning moves a mesh with its skeleton but cannot produce clothing dynamics such as a swinging skirt; physics simulation can, but needs expert setup and is costly §1. Learned models that regress clothing from motion usually need registered 3D geometry, which means expensive capture and template registration §1. Learned monocular neural rendering gives limited resolution and loses consistency across pose and view §1. The authors assume a studio with a dense multi-view rig (green screen, calibrated cameras), a textured template scan per actor, and a markerless skeleton tracker §3.1 §3.2 §4.1.

Method

The character is a template mesh of about 5000 vertices, rigged to a skeleton with 57 joint angles plus global rotation and translation, skinned with dual quaternion skinning §3.1. Non-rigid deformation is applied in canonical (T-)pose before skinning, so the networks predict deformation without the ambiguity that pose changes add §3.1.

Steps of the forward model, for one frame t:

1. Normalise the motion window for geometry. Take F = 2 past frames plus the current one. Remove global translation (relative to the current frame) and the rotation about the vertical axis §3.2.
2. Pose the embedded graph (about 500 nodes) with the normalised motion, giving a per-node feature of size 6(F+1). EGNet maps these features to per-node Euler rotations A and translations T in canonical pose §3.3.1.
3. Apply the embedded deformation to the canonical template, with per-vertex displacements set to zero at this stage eq. 4 eq. 2.
4. Add the DeltaNet vertex displacements in canonical pose §3.5.1 eq. 14.
5. Pose the deformed canonical mesh with the current skeleton, using the weighted dual-quaternion blend eq. 3.
6. Render the posed mesh into a view with per-camera spherical-harmonic lighting, which is fitted beforehand §3.4 eq. 10-eq. 13.
7. For texture, use the motion window with only global translation removed, pose the template for each pose, render global normals per pose, stack them with a 6-channel camera encoding, and let TexNet output a 1024×1024×3 texture §3.2 §3.6.

# Forward pass of the deep dynamic character  [§3.1] [§3.3] [§3.5] [§3.6]
# Input:  motion window (F+1 poses), camera pose
# Output: posed, deformed vertices (N, 3); dynamic texture (1024, 1024, 3)

def character_forward(motion_window, camera_pose):
    # Normalise the window for geometry: no global translation, no
    # rotation about the vertical axis  [§3.2]
    geo_motion = normalise_for_geometry(motion_window)
    # Pose the embedded graph with the normalised motion  [§3.3.1]
    graph_features = pose_embedded_graph(geo_motion)  # (K, 6*(F+1))
    # Coarse deformation: per-node rotations and translations  [§3.3.1]
    node_rot, node_trans = egnet(graph_features)      # (K, 3), (K, 3)
    # Blend node motion into each canonical vertex  [eq. 2] [§3.1]
    canonical = embedded_deform(template, node_rot, node_trans)
    # Fine deformation: per-vertex displacements  [§3.5.1] [eq. 14]
    posed_pose_positions = pose_template(geo_motion)  # (N, 3*(F+1))
    displacements = deltanet(posed_pose_positions)     # (N, 3)
    canonical = canonical + displacements
    # Pose the deformed canonical mesh with the skeleton  [eq. 3]
    vertices = skin_with_dual_quaternions(canonical, current_pose)
    # Texture input: stacked normal maps plus camera encoding  [§3.6]
    tex_input = stack_normal_maps(motion_window) + camera_encoding()
    texture = texnet(tex_input)                        # (1024, 1024, 3)
    return vertices, texture

Training is weakly supervised in four stages, each with its own losses:

1. EGNet: silhouette loss plus an as-rigid-as-possible (ARAP) regulariser on the node parameters §3.3.4 eq. 7 eq. 8 eq. 9.
2. Lighting: fit per-camera spherical-harmonic lighting with the static texture and the coarse geometry, rendering with Lambertian shading §3.4 eq. 13.
3. DeltaNet: dense chroma loss in YUV (U and V channels only) plus silhouette and a Laplacian regulariser, using the fixed lighting §3.5.2 eq. 15 eq. 16 eq. 17.
4. TexNet: photometric L1 loss on the eroded foreground of the conditioning view only §3.6 eq. 18.

Key equations (the paper's own notation, reproduced for implementers):

\mathbf{y}_i = \mathbf{d}_i + \sum_{k \in \mathcal{N}_\mathrm{vn}(i)} w_{i,k}\big(R(\mathbf{a}_k)(\hat{\mathbf{v}}_i - \mathbf{g}_k) + \mathbf{g}_k + \mathbf{t}_k\big) eq. 2

\mathbf{v}_i = \mathbf{z} + \sum_{k \in \mathcal{N}_\mathrm{vn}(i)} w_{i,k}\big(R_{\mathrm{sk},k}\,\mathbf{y}_i + t_{\mathrm{sk},k}\big) eq. 3

\mathcal{L}_\mathrm{arap} = \sum_{k=1}^{K} \sum_{l \in \mathcal{N}_\mathrm{n}(k)} u_{k,l}\,\lVert d_{k,l}(\mathbf{A},\mathbf{T}) \rVert_1 eq. 9

\mathcal{L}_\mathrm{chroma} = \sum_{c=1}^{C}\sum_{\mathbf{u}} \lVert g(\Phi_{c,\mathbf{u}}(\mathbf{V}_\mathrm{fi}, \mathcal{T}_\mathrm{st}, \mathbf{l}^*_{c})) - g(\mathcal{I}_{c,\mathbf{u}}) \rVert^2 eq. 16

\mathcal{L}_\mathrm{texture} = \sum_{\mathbf{u}} \big\lvert \hat{\mathcal{F}}_{c',\mathbf{u}}\big(\Phi_{c',\mathbf{u}}(\mathbf{V}_\mathrm{fi}, \mathcal{T}_\mathrm{dyn}, \mathbf{I}_\mathrm{sh}) - \mathcal{I}_{c',\mathbf{u}}\big)\big\rvert eq. 18

Character model and skinning §3.1

• Template: a scan of the actor in T-pose, reconstructed with commercial multi-view stereo to a static texture and downsampled to around 5000 vertices. The mesh is segmented manually by human-parsing labels, and each vertex gets a rigidity weight s_i (low for cloth, high for skin) §3.1.
• Skeleton: θ ∈ R^57 joint angles, global rotation α ∈ R^3, global translation z ∈ R^3. Skinning weights are computed automatically in Blender §3.1.
• Embedded graph: about 500 nodes, with per-node Euler rotation A and translation T. Vertex weights w_{i,k} fall off with geodesic distance and are zero beyond a threshold; the threshold value is not stated §3.1.
• Equations 2 and 3 are differentiable in pose, graph and displacements eq. 2 eq. 3.

Data capture and motion normalisation §3.2

• Studio: C = 120 synchronised, calibrated 4K cameras at 25 fps, green screen colour keying §3.2.
• Images I, distance transform D, and foreground masks F per camera and frame §3.2.
• Skeleton tracking with a multi-view markerless system §3.2.
• Window: F = 2 §3.2.
• Geometry normalisation: ẑ_t = 0, ẑ_t' = z_t' − z_t; the rotation about the vertical axis is normalised the same way. Texture normalisation removes only global translation, keeping the vertical-axis rotation because lighting depends on it §3.2.

Embedded deformation regression (EGNet) §3.3

• EGNet is a structure-aware graph convolutional network (SAGCN) on the embedded graph. Input width 6(F+1) = 18, output width 6 (rotation and translation per node). It uses L = 8 residual blocks, H_1 = 16, ring radius R = 3, and no dense block, since the graph is small §3.3.3.
• Output is the node parameters in canonical pose for the current frame §3.3.1.
• Structure-aware graph convolution (SAGC) §3.3.2: each node has its own kernel K_l and bias b_k. The node output is a bias plus a sum over the R-ring neighbours of a_{k,l} K_l f_l. The weights a_{k,l} are the inverse ring value divided by the sum over the neighbourhood, so the ring value falls off linearly eq. 5 eq. 6.
• Residual block: two chains of batch norm, ELU and SAGC, with a skip connection. A dense block, where all a_{k,l} = 1, is used in DeltaNet but not EGNet §3.3.3 §3.5.1.
• Losses §3.3.4: L_eg = L_sil + L_arap eq. 7. The silhouette term is bi-sided: model boundary vertices are pushed onto the zero contour of the distance transform, scaled by a normal-matching weight ρ, and each silhouette pixel is pulled to its nearest projected vertex eq. 8. ARAP is an L1 penalty on the rotation-translation mismatch across graph edges, weighted by u_{k,l}, the average of the rigidity weights of the vertices attached to the two nodes eq. 9.

Lighting estimation §3.4

• Lambertian surface and distant light. Per camera, 9 spherical-harmonic coefficients per colour channel, so 27C in total §3.4.
• Pixel colour is albedo times shading. Albedo is a visibility indicator times a barycentric texture lookup; shading is the sum over the nine SH basis functions of the coefficients times the basis at the screen-space normal eq. 10 eq. 11 eq. 12.
• Fit with the static texture and the coarse posed geometry V_co, minimising squared colour error over all frames with the same lighting per camera eq. 13. Each SGD step samples four frames; the number of steps is 30,000 §3.4 §3.7.

Vertex displacement regression (DeltaNet) §3.5

• Same SAGCN as EGNet, but each vertex is a graph node and the graph is the template mesh. Input width 3(F+1) = 9 (posed vertex positions over the window), output width 3 (one displacement per vertex). L = 8, H_1 = 16, R = 3, with a dense block because the mesh is large §3.5.1.
• Displacements are in canonical pose and go through the same two-stage pipeline as EGNet eq. 14.
• Losses §3.5.2:
  • Chroma: render with the fixed lighting, convert rendered and ground-truth RGB to YUV, and compare only U and V with squared error eq. 16. This discounts self-shadows the renderer cannot model.
  • Silhouette: the same bi-sided loss as EGNet, applied after displacements eq. 8.
  • Laplacian: per-vertex weight s_i times the difference between the Laplacian of the displaced and coarse meshes, with the one-ring neighbourhood N_i eq. 17.
  • Total: L_Delta = L_chroma + L_sil + L_lap eq. 15.

Dynamic texture (TexNet) §3.6

• Input: normal maps of the template posed at each window pose, stacked (3(F+1) channels), plus a 6-channel camera position and orientation map. Total 15 channels at 1024×1024 §3.6.
• Network: a UNet adapted to 1024×1024 inputs. Output 1024×1024×3 §3.6.
• Loss: L1 photometric error on the conditioning camera only, with the foreground mask eroded so background pixels do not leak into the texture eq. 18. The identity lighting I_sh is used when rendering §3.6.

Implementation details §3.7

• Optimiser: Adam for all experiments §3.7.
• Multi-view losses sample 40 cameras per step, if available §3.7.
• Distance transforms at 350×350; renderer at 512×512 (643×470 original aspect) for DeltaNet and lighting, and 1024×1024 (1285×940) for TexNet §3.7.

Hyperparameters

Results

Author-reported. All numbers are on DynaCap (the authors' new dataset, 5 sequences, 4 subjects, 50 to 101 cameras at 1285×940) unless noted §4.1. DeepCap S4 (11,000 test frames, 11 to 14 cameras at 1024×1024) is the external benchmark §4.1 §4.4.1.

Qualitative results cover eight sequences with different apparel, including unseen fast motion such as jumping jacks, and show view-dependent reflections and augmented empty-background renders fig. 7 fig. 8 §4.2. Applications are motion retargeting, free-viewpoint synthesis from a driving motion, and an interactive pose-and-camera editor running in real time fig. 10 §4.6.

Conceptual comparison tab. 1: prior multi-view work is compared on dynamic geometry, dynamic texture, view-dependent effects, controllability, real time, unseen motions and loose clothing. The authors' method is marked as having all of them; the comparison is qualitative, not run on the baselines §4.3.

Paper vs code

No official code.

Searched, in order (2026-11-08):

• The paper, the LaTeX source and the arXiv metadata contain no code, GitHub or project link. The arXiv JSON has no comment field with a link.
• The project page URL gvv.mpi-inf.mpg.de/projects/DynamicCharacters/ redirects to vcai.mpi-inf.mpg.de/projects/DynamicCharacters/, which returned HTTP 404.
• GitHub repository search for "dynamic characters", "DynaCap" and "deep dynamic characters" found three unrelated DynaCap repositories and one third-party PyTorch port (kv2000/DeepCharacters_Pytorch, no licence). None is by the authors, so none is used.

The paper says the DynaCap dataset will be released §1 §4.1. No release was found in the search above, and the release status is not verified here.

Pitfalls

• Equation and figure numbers here are counted in source order and checked against the LaTeX; the arXiv v2 build could differ. If citing in new text, check against the PDF.
• Two different window normalisations. Geometry removes global translation and vertical-axis rotation; texture keeps the vertical-axis rotation because lighting depends on it §3.2. Mixing these up gives the wrong input space for one of the two networks.
• Two-stage displacement input. DeltaNet consumes the template posed over the window (3(F+1) per vertex), not the regressed EGNet output §3.5.1. The displacements are added in canonical pose, before the global skinning eq. 14 eq. 3.
• Lighting is fitted, then frozen. The lighting uses the static scan texture, not the dynamic one. Its purpose is to make the chroma loss fair, not to give the final appearance §3.4 fig. 4.
• Chroma loss compares colour only. It compares only the U and V channels, so it does not directly constrain luma; the silhouette and Laplacian terms supply the other constraints on the displaced geometry §3.5.2.
• Erosion and identity lighting. TexNet uses an eroded foreground mask and the identity lighting. The erosion size is not stated §3.6.
• Unstated details to fill in. Weight decay, LR schedule, optimiser betas, the vertex-weight distance threshold, the exact ARAP weight averaging rule and the exact normal-matching formula for ρ are not stated §3.1 §3.3.4 §3.7.
• Sampling noise. Multi-view losses sample 40 random cameras per step, so the per-step loss depends on the random camera choice; the seed is not stated §3.7.
• Lighting uses every frame, but each step only four. Thirty thousand SGD steps with four frames per step is a different schedule from full-batch optimisation §3.4.
• Preprocessing is manual. Template segmentation is manual, the rigging is done by hand, and the static texture comes from a commercial MVS package. Each new actor needs this work §3.1.
• Training cost and hardware. EGNet took 20 h on four 48 GB GPUs, lighting about 7 h, DeltaNet about 2 days, TexNet about 4 days. Hardware for the last three stages is not stated §3.7.
• Failure modes the authors name. Topology changes such as removing a garment, face and hand motion, multi-person scenes, and poses far from the training motion distribution (for example a hand stand) §4.7.
• Comparison against MVBL is not like for like. MVBL optimises on the test images, while this method sees only skeleton motion at test time. The authors say this is why the comparison is favourable §4.4.1.
• Texture numbers are on a held-out camera. MSE and SSIM are averaged every 100th frame with the background masked out, so they do not measure full-image quality §4.4.2.
• Test-time start-up. For the first test frame the pose is copied to fill the past window §4.
• Third-party tools in the pipeline. The paper uses a commercial multi-view stereo package for the template, a markerless motion tracker and Blender for skinning weights. None of these is part of the released material (there is none), so an implementation has to replace them §3.1 §3.2.

Lineage

• Builds on: the same group's DeepCap work (MVBL baseline, rigidity weights and material-aware weighting) §3.1 §3.3.4 §4.4.1; embedded deformation of Sorkine and Sumner §3.3.1; dual quaternion skinning §3.1.
• Modifies: graph convolution, with a per-node kernel and a linear ring weight, instead of the shared-filter graph convolution of Defferrard et al., which the authors use as the ablation baseline §3.3.2 §4.5.
• Compared against: Xu et al. (2011) and Casas et al. (2014), both retrieval-based and not learning geometry §4.3; Textured Neural Avatars (Shysheya et al., 2019), which uses a DensePose proxy and a static texture §4.3.
• Superseded by: not known from wiki content.