Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition

In plain words

The goal is a 3D model of one person, with their body shape and clothing, built from a normal video of them moving in a messy real scene. The model should be able to show the person from new angles and in new poses. The hard part is working out which pixels show the person and which show the background, without anyone telling the computer where the person is. Earlier methods usually first ran a separate tool that cuts the person out of each frame, and their results could be no better than that cut-out.

The authors avoid the cut-out step. They model the person and the background as two separate 3D fields and learn both at the same time from the video. The person is stored in a standard rest pose and bent to match each frame's body pose, using a common body model called SMPL. Rays from the camera pass through both fields. Extra training rules push each ray towards being either fully empty or fully solid, and force rays that miss the body to stay empty. The body shape, the appearance, the background and the pose of every frame are all optimised together.

The result is a posable avatar from a single video, without 3D scans or a segmentation network. The authors report better person masks and surface reconstructions than the baselines they tested, though their tests are on a few datasets and one of them is a synthetic set they built themselves.

TL;DR

• Learns a posable 3D human and a separate background from one monocular in-the-wild video, with no ground-truth supervision and no segmentation network §1 §3.
• The human is a signed distance field in SMPL-normalised canonical space, bent to each frame by linear blend skinning §3.1. The background is a NeRF++-style inverted-sphere field §3.2.
• Two scene-decomposition losses (a sparsity term on rays that miss the body and a binary cross-entropy term on ray opacity) sharpen the human-background boundary §3.3.
• Author-reported gains over baselines for segmentation on MonoPerfCap, rendering on NeuMan, and surfaces on 3DPW and the authors' SynWild set tab. 1 tab. 2 tab. 3 tab. 4.
• The official code is MIT-licensed but has extras the paper does not describe: an initial SDF checkpoint (assets/smpl_init.pth), a projected-mask ray sampler, and loss schedules. See Paper vs code.

Problem

Monocular, in-the-wild video makes both parts of the task difficult: separating the human from an arbitrary background, and recovering a detailed 3D surface from a short sequence §1. Template-based methods need a pre-scanned template and rigging and do not generalise §1. Explicit mesh methods are tied to a fixed topology and resolution §1. Fully supervised regressors trained on clothed scans struggle with unusual poses and shapes and are not always temporally consistent §1. Earlier neural-implicit video methods depend on pre-segmented inputs, so their quality is capped by the segmentation step §1.

Assumptions: the method needs reasonable initial pose estimates for each frame §5. Loose, fast-moving clothing such as skirts is a stated weak point §5. The main text does not say how the pose estimates or the cameras are obtained; the released pipeline uses external estimators (see Paper vs code).

Method

The model has two parts, a foreground human and a background, learned jointly and composited per pixel §3 fig. 2. The cameras are normalised so that they sit within a sphere of radius 3 run_preprocessing.sh, and the model uses the same radius v2a.py. (Neither value is stated in the paper.)

Canonical human shape §3.1. A network f_sdf^H takes a point in canonical space and the body pose θ, and returns a signed distance plus a 256-dimensional feature vector z eq. 1. The human surface is the zero level set of the signed distance eq. 2. Pose-dependent wrinkles are captured by feeding θ to the network §3.1.

Skinning §3.1. A canonical point x_c maps to a deformed point x_d by linear blend skinning over the bone transforms B_i with weights w_i eq. 3. The inverse map, which takes a deformed sample back to canonical space, uses the inverse of the blended transform eq. 4. The skinning weights of a point are taken from the nearest SMPL vertices, weighted by distance §3.1.

Canonical texture §3.1. A second network f_rgb^H predicts colour at a canonical point from its normal, the pose and the 256-dimensional feature z eq. 5. The normal is the gradient of the signed distance with respect to deformed-space position, obtained by the chain rule through the inverse skinning map eq. 6.

Background §3.2. The background uses an inverted-sphere parametrisation borrowed from NeRF++ §3.2. A background point is written as a unit direction plus its inverse distance, which gives better numerical stability and gives distant points lower resolution §3.2. A background network f^B takes this 4-D point, the view direction and a learnable per-frame latent code of size n_t, and returns a density and a view-dependent colour eq. 7. The per-frame code absorbs changes such as lighting §3.2.

Foreground density and rendering §3.2. The signed distance is converted to density with the cumulative distribution function of a scaled Laplace distribution applied to the negated signed distance eq. 8. The scale α and width β are both described as learnable and positive eq. 8, §3.2. Samples along each ray are drawn in two stages, uniform then inverse-CDF §3.2. The foreground colour is the transmittance-weighted sum of canonical colours eq. 9 with weights τ_i eq. 10. The accumulated weight gives the ray opacity α^H §3.2.

Composition §3.2. The final colour is the foreground colour plus the background colour scaled by one minus the foreground opacity eq. 11.

Scene decomposition losses §3.3. Two terms make the foreground and background separate cleanly.

• Sparsity. Points of a ray are warped to canonical space and their signed distance to the current human shape is computed. Rays that never come near the human form a set R_off, and their opacity is penalised with an L1 term eq. 12. The human shape is refreshed during training §3.3.
• Binary cross-entropy. The opacity of every sampled ray is pushed towards 0 or 1 by a binary cross-entropy term eq. 13. The combined decomposition loss is a weighted sum of these two terms eq. 14.

Global optimisation §3.4. Training is a single optimisation over all frames. It updates the human shape and texture networks, the background networks and the per-frame pose parameters θ_i §3.4. An eikonal term from IGR enforces unit gradient norm of the signed distance over canonical space eq. 15. The reconstruction term is an L1 loss between rendered and observed pixel colours eq. 16. The full objective sums these terms over the F frames eq. 17.

Full objective eq. 17, as the paper writes it, with F frames:

\mathcal{L}(\Theta) = \sum_{i=1}^{F}\left( \mathcal{L}^{i}_{rgb} + \lambda_{dec}\,\mathcal{L}^{i}_{dec} + \lambda_{eik}\,\mathcal{L}^{i}_{eik} \right) eq. 17

The paper gives no numeric values for λ_dec, λ_eik or any other weight §3.4. The values used in the official code are in Hyperparameters.

The forward pass for one frame, reconstructed from the paper and checked against the code, is:

# Forward pass for one frame  [§3.1] [§3.2]
# Input:  rays (R rays, origin and direction), body pose theta, frame code
# Output: pixel colours (R, 3), ray opacity (R,)

def render_frame(rays, theta, frame_code):
    # Choose depths on each ray, coarse-to-fine  [§3.2]
    # [code: code/lib/model/ray_sampler.py#L66-L70]
    depths_fg, depths_bg = sample_depths(rays)   # (R, N), (R, M)
    # Map each foreground sample back to canonical space  [§3.1] [eq. 4]
    points_c = inverse_skin(points_along(rays, depths_fg), theta)
    # Query the canonical signed distance and its feature  [§3.1] [eq. 1]
    sdf, feature = sdf_net(points_c, theta)     # (R, N, 1), (R, N, 256)
    # Convert the signed distance to density  [§3.2] [eq. 8]
    sigma = laplace_density(sdf, beta)
    # Normal: gradient in deformed space, then normalised  [§3.1] [eq. 6]
    normals = normalise(deformed_gradient(sdf, points_c, theta))
    # Foreground colour from the canonical texture network  [§3.1] [eq. 5]
    colour_fg = texture_net(points_c, normals, theta, feature)
    # Foreground rendering: weights, colour and opacity  [§3.2] [eq. 9]
    weights = render_weights(sigma, depths_fg)
    fg_rgb = sum(weights * colour_fg, axis=samples)
    opacity = sum(weights, axis=samples)
    # Background rendering over inverse-depth samples  [§3.2] [eq. 7]
    bg_rgb = render_background(depths_bg, rays, frame_code)
    # Composite foreground over background  [§3.2] [eq. 11]
    return fg_rgb + (1 - opacity) * bg_rgb, opacity

The per-frame loss, with its sparsity and cross-entropy terms, is:

# Loss for one frame  [§3.3] [§3.4]
# Input:  predicted and target colours, ray opacity, two ray masks,
#         gradient of the signed distance at surface-near canonical points
# Output: scalar loss

def frame_loss(pred_rgb, target_rgb, opacity, off_ray, grad_norm):
    # Reconstruction: L1 between rendered and observed colours  [eq. 16]
    rgb_term = mean(abs(pred_rgb - target_rgb))
    # Eikonal: unit gradient norm of the canonical signed distance  [eq. 15]
    eik_term = mean((grad_norm - 1) ** 2)
    # Binary cross-entropy pushes opacity towards 0 or 1  [eq. 13]
    bce_term = mean(binary_cross_entropy(opacity))
    # Sparsity: rays that miss the human must have zero opacity  [eq. 12]
    sparse_term = mean(abs(opacity[off_ray]))
    # Decomposition term, then the full per-frame loss  [eq. 14] [eq. 17]
    dec_term = bce_term_weight * bce_term + sparse_weight * sparse_term
    return rgb_term + eik_weight * eik_term + dec_term

The pseudocode omits the epoch-dependent weights and the extra in-shape term that the official code adds. Those are in Paper vs code.

Hyperparameters

The paper's main text gives almost no numeric hyperparameters. It refers to a supplement for more details §3.2 §5. That supplement is not in the source bundle, so the values below come from the official code. They have not been checked against the supplement.

Results

Author-reported. Setups below come from the paper's text §4. Tables are in the compiled paper's order (see Pitfalls for the numbering caveat).

Human segmentation on MonoPerfCap, with ground-truth masks §4.1 §4.3 tab. 1:

Novel-view rendering on NeuMan, test views §4.4 tab. 2:

Surface reconstruction on 3DPW, with pose-based quasi ground truth §4.5 tab. 3:

Surface reconstruction on SynWild, the authors' semi-synthetic set §4.1 §4.5 tab. 4:

Metrics: precision, F1 and mask IoU for segmentation; volumetric IoU, Chamfer distance in cm and normal consistency for surfaces; SSIM and PSNR for rendering §4.1.

Paper vs code

Official code: MoyGcc/vid2avatar. The authors' project page links it, and the MIT LICENSE names Chen Guo, an author LICENSE. The project page was checked with WebFetch; the code is not cited from the paper itself. Cloned at commit 0d026bae. The copyright line reads 2024, so the snapshot is later than the 2023 paper.

Discrepancies and undocumented additions, each with both citations:

1. Density scale. The paper treats the Laplace scale α and width β as two learnable parameters eq. 8 §3.2. The code learns only β and sets α = 1/β density.py. This is the usual VolSDF form.
2. Extra in-shape loss. The code adds a term that pulls ray opacity to 1 for rays that pass through the canonical mesh, weighted down to zero by epoch 200 loss.py loss.py. The paper has no such term.
3. Sparsity weight schedule. The code increases the sparsity weight with epoch, up to epoch 200 loss.py. The paper uses a fixed weight λ_sparse eq. 14.
4. Factor of 2 in BCE. The code multiplies the cross-entropy term by 2 loss.py. The paper's eq. 13 has no factor.
5. Skinning neighbours. The paper weights the nearest SMPL vertices by distance §3.1. The code uses K = 1 during training deformer.py deformer.py. With one neighbour the weighting is trivial, so this is the nearest-vertex weight. The test-time mesh export uses K = 7 v2a_model.py.
6. Normals. The paper's normal eq. 6 is an unnormalised gradient. The code normalises it before the colour network v2a.py.
7. Background colour input. Eq. 7 has the background network output density and colour directly. The code's background network outputs density and a 256-dimensional feature, and a separate small colour network turns that feature into colour, with the frame code and a view encoding v2a.py model_w_bg.yaml.
8. Pose conditioning dimension. The paper writes the pose as n_θ §3.1. The code uses the 69 body-pose values with global orientation excluded, scaled by 1/π, and passes them through a linear layer to 8 values in the colour network v2a.py networks.py.
9. SMPL-based initialisation. The code loads a checkpoint assets/smpl_init.pth into the SDF network at start-up v2a.py. The paper does not mention any initialisation of this kind. The README does not describe where the checkpoint comes from.
10. Off-surface test. The paper says sparsity uses the distance to "the human shape", refreshed throughout training §3.3. The code measures the distance to the canonical mesh, which starts as the SMPL template and is re-extracted from the learned SDF every 20 epochs v2a.py v2a_model.py. A ray is off-surface when the smallest signed distance of its samples is above 0.05 v2a.py.
11. Sampling. The paper describes two stages, uniform and inverse-CDF §3.2. The code runs up to five proposal rounds, choosing β per ray by bisection against a bound ray_sampler.py ray_sampler.py.
12. Eikonal samples. The paper takes the expectation over canonical space eq. 15. The code samples points near canonical SMPL vertices with σ = 0.01 jitter and no uniform samples v2a.py sampler.py.
13. Body parameters. The paper optimises per-frame pose parameters θ_i §3.4. The code also optimises the global orientation, the translation and the shape vector, all at 0.1 × the network learning rate v2a_model.py v2a_model.py.
14. Test-time outliers. Samples more than 0.1 from the SMPL vertices get an SDF of 4 at inference deformer.py v2a.py. The paper does not mention this.
15. Ray sampling uses masks. Training rays are drawn with 90% inside the box of a projected SMPL mask, dilated by a 20 × 20 kernel dataset.py utils.py preprocessing.py. The mask is only used to choose rays, not in the loss. The paper says it uses no external segmentation modules §1 §3. The mask pipeline needs ROMP and OpenPose README.md run_preprocessing.sh.
16. Evaluation code. The repository has a PSNR helper utils.py but no script for IoU, Chamfer, normal consistency, F1 or SSIM. The test step writes masks, renderings, normals and meshes v2a_model.py. The paper's metrics cannot be reproduced from the release alone.
17. Data. The SynWild data is not in the repository. The README links an external site README.md. The demo sequence is a Google Drive link README.md. The paper says code and data "will be made available" §1. The card does not cover the dataset licence.

Items 1, 5, 6, 7 and 8 change the forward-pass equations eq. 1 to eq. 11. Items 2, 3, 4, 10, 11 and 12 change the losses and sampling. Items 9, 13 to 17 concern initialisation, data and evaluation.

Pitfalls

• The supplement is missing. The paper refers to a supplement for the architecture, sampling and training details §3.2 §5. It is not in the source bundle. Every numeric value in Hyperparameters is from the code.
• Extra inputs are required. The method needs an SMPL shape and per-frame poses. The released pipeline gets these from ROMP and refines the poses offline with OpenPose keypoints README.md run_preprocessing.sh. The paper says it needs only reasonable pose estimates §5. Pose quality is therefore the main upstream risk.
• SMPL model files are not included. The user must download the SMPL models under their own licence and place them in the code tree README.md. Kaolin 0.10.0 is required README.md. The code imports kaolin v2a.py and pytorch3d deformer.py.
• Numerics. The Laplace density divides by β, with a floor of 1e-4 density.py. The skinning Jacobian is inverted per sample, with no damping v2a.py. The ray-proposal bound uses exponentials of error integrals, clamped at 10^6 ray_sampler.py. Near-singular Jacobians or very small β can yield NaNs; the loss filters NaN rays loss.py.
• Costs. Training runs up to 8000 epochs on one GPU, which the README puts at 24 to 48 hours train.py README.md. The canonical mesh is re-extracted every 20 epochs v2a_model.py. Each step renders 512 rays for one frame video.yaml.
• Comparisons are not like-for-like. ICON and SelfRecon use extra human masks as input §4.5. The 2D baselines PointRend and RVM are trained on large annotated data §4.3. The paper's segmentation test uses ground-truth masks for MonoPerfCap §4.1.
• Small margins on two sets. On SynWild the IoU lead over SelfRecon is 0.008 tab. 4. On NeuMan the SSIM lead over HumanNeRF is 0.001 tab. 2. The NeuMan sequence is one person walking §4.1, so the test is narrow.
• Table and figure numbers here follow the source order of the LaTeX and are not checked against the rendered PDF. The equation numbers follow the \begin{equation} order in sec/method.tex.
• Unused tables. tables.tex defines a background-ablation table and a CAPE table that the body never places [tables.tex]. They are not used in this card.
• Loose clothing and fast motion are stated weak points §5. The card has no independent check of these limits.

Lineage

• Builds on, as cited in the paper: NeRF++ for the inverted-sphere background §3.2, VolSDF for the SDF-to-density conversion and sampling §3.2, NeRF for the volume-rendering integral §3.2, IGR for the eikonal term §3.4, and IMavatar for the normal through the deformation §3.1. The README also acknowledges SMPL-X, Anim-NeRF and SNARF README.md. None of these has a card here, so none is linked.
• Compared against: ICON and SelfRecon for reconstruction §4.5, HumanNeRF and NeuMan for novel views §4.4, and PointRend, RVM and Ye et al. for segmentation §4.3.
• Superseded by: not known from the content of this wiki.