PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing

In plain words

Given one photo of a person, the back and sides are hidden from the camera, so a computer has to guess them. PSHuman does this in two stages. First, an image generator draws the same person from several angles, plus a close-up of the face. It is steered by a parametric body model (SMPL-X), so the poses and body proportions stay plausible even when the person is bent or partly hidden. Second, the method takes that body model as a starting shape and reshapes its surface, including its topology, until it matches the drawn views. The output is a textured 3D mesh. The authors report that this takes about a minute.

TL;DR

• Single full-body RGB photo in, textured 3D human mesh out, no per-subject optimisation.
• Stage 1 fine-tunes Stable Diffusion 2.1-unclip to produce six body views plus a face view, as color and normal maps, with SMPL-X renders as an extra condition §3.1.
• Stage 2 deforms and remeshes an SMPL-X mesh against the generated normal and silhouette maps, then projects colors onto it §3.2.
• Author-reported gains over nine baselines on CAPE and THuman2.1, for geometry and appearance §4, tab. 1, tab. 3.
• Code is official and MIT-licensed, but the texturing step differs from the paper, so treat the paper's texture description as unreliable reconstruct.py.

Problem

Implicit-function methods regress a surface from one image and lose fine detail. Methods that add a generated back view still rely on implicit functions and gain little stereo information §1. Multiview diffusion on a single human image gives distorted poses for bent or self-occluded bodies, and faces are often distorted or changed, because the face covers only a small region of a full-body image §1 §3.1.1. Optimisation-based SDS methods take hours; the authors cite about six hours for TeCH §1.

Method

3.1 Body-face multiview diffusion

Views. The model generates six views, front, front-left, left, back, right and front-right, as color and normal maps, following Era3D and Wonder3D rather than front and back views only §3. The paper argues that six views balance cost and quality §3. Supplement §2 says that two or four views produce incomplete or unnatural shapes [supp. §2].

Body-face cross-scale diffusion (§3.1.1). The data distribution is written as a joint over the body image x^B and the face image x^F eq. 1. The forward process is approximated by noising body and face separately eq. 4; the reverse process lets the body denoiser depend on the face denoising result eq. 3. The reverse process is implemented by noise blending, inside every self-attention block of the UNet. The face tokens are resized by a scale s and added to the face region of the front body view, weighted by w eq. 5. Other body views are not blended eq. 5. Training minimises a face denoising loss plus a body loss conditioned on the face latent, with shared weights eq. 6.

SMPL-X conditioning (§3.1.2). SMPL-X is estimated from the input image and rendered from the six target viewpoints. A pretrained VAE encodes these renders and the reference image; the latents are concatenated with the noise latent as UNet input §3.1.2. The paper says this constrains pose and self-occlusion hallucinations §3.1.2 fig. 1.

Training data. 2,385 THuman2.1 scans and 647 CustomHumans scans, all with SMPL-X parameters §4. Each scan is rendered at 8 yaw angles, 45° apart, at 768 × 768; a face detector keeps only views with a clear face [supp. §2].

3.2 SMPL-X-initialised explicit human carving

Carving has three stages: SMPL-X initialisation, differentiable remeshing, and appearance fusion §3.2.

Initialisation eq. 7. SMPL-X translation, shape and pose are optimised so that its normal and silhouette renders match the generated views, with view confidences as weights §3.2.

Remeshing eq. 8. Vertices and face connectivity are both optimised, using the optimiser of Palfinger et al. §3.2. The loss is the weighted normal and silhouette mismatch plus a Laplacian-style smoothness term with weight λ = 0.02 §3.2. After optimisation, Poisson reconstruction fills small invisible areas such as the chin. Hands can optionally be replaced with the SMPL-X hands §3.2.

Appearance eq. 9. Per-vertex colors minimise the weighted difference to the generated color views §3.2. Vertices not seen from the six views are then filled by topology-aware KD-tree interpolation §3.2. The code uses a different texturing method (see Paper vs code).

# Joint noise blending, inside each self-attention block  [§3.1.1] [eq. 5]
# hidden_body: (N, H*W, D) body tokens; hidden_face: (1, H*W, D) face tokens

def noise_blend(hidden_body, hidden_face, side):
    # Face region size s = side // 3, i.e. 32 for a 96 x 96 latent
    # [code: mvdiffusion/models_unclip/attn_processors.py#L175]
    s = side // 3
    # Shrink the face tokens to the size of the face region  [§3.1.1]
    face_small = resize(hidden_face, (s, s))
    # Blend into the front view only, rows 0..s and columns s..2s  [eq. 5]
    # The paper's form is additive with a mask weight w; the code uses
    # a fixed convex blend 0.5 / 0.5  [code: attn_processors.py#L261]
    hidden_body[0, :s, s:2s] = (
        0.5 * hidden_body[0, :s, s:2s] + 0.5 * face_small
    )
    return hidden_body

# SMPL-X-initialised remeshing  [§3.2] [eq. 7] [eq. 8]
# normal_maps, masks: 6 views; view_weights: per-view confidence  [§3.2]

def carve(normal_maps, masks, view_weights, smplx):
    # Fit translation, shape and pose to the views  [§3.2] [eq. 7]
    # Paper §4 says 700 steps; code runs 100  [code: reconstruct.py#L276]
    smplx = fit_smplx(smplx, normal_maps, masks, view_weights, steps=100)
    mesh = smplx_to_mesh(smplx)
    # Deform vertices and topology against the normal and silhouette loss
    # [§3.2] [eq. 8]; 700 steps  [code: remesh.yaml#L12]
    for step in range(700):
        normals, alpha = render(mesh)
        loss = weighted_l1(normals, normal_maps, view_weights)
        loss += weighted_l1(alpha, masks, view_weights)
        # Laplacian smoothness, weight 0.02  [eq. 8] [code: core/opt.py#L73]
        loss += 0.02 * laplacian_term(mesh)
        mesh = remesh_step(mesh, loss)
    # Fill small holes such as the chin with Poisson reconstruction  [§3.2]
    mesh = poisson(mesh, depth=10)  # [code: reconstruct.py#L384]
    return mesh

Hyperparameters

Values in the "Paper" column are stated in the compiled PDF. Values marked "code only" come from the official repository and are not in the paper text.

Results

Author-reported. Baselines are PIFu, PIFuHD, PaMIR, ICON, ECON, GTA, SIFU, HiLo and SiTH. Metrics: Chamfer distance (cm, lower is better), P2S, NC; PSNR, SSIM, LPIPS for appearance §4.

Geometry, with SMPL-X prior (ground-truth SMPL-X used, so pose error is removed):

Geometry, without SMPL-X prior (PIXIE estimates the SMPL-X for baselines): on CAPE-NFP, the paper's Chamfer is 2.1625 against 2.6507 for HiLo tab. 1.

Face region (head crop following ECON):

Appearance on a THuman2.1 subset (four views):

Robustness to SMPL-X noise (variance 0.05 on pose and shape): Chamfer 0.5574 against 0.6427 for SiTH tab. 4.

Ablation, Chamfer on 20 CAPE-NFP samples tab. 5: no CSD and no SMPL-X condition 1.4920; CSD without SMPL-X condition 1.4370; CSD and SMPL-X condition with plain remeshing 1.0938; CSD and SMPL-X condition with ECON-style carving 1.2630; full model with SMPL-X-guided remeshing 0.9597.

Timing for the reconstruction module, in seconds [supp. §3]: pre-processing 7.2, diffusion 17.6, geometry reconstruction 23.3, appearance fusion 6.0. The introduction says the whole process takes about one minute §1.

User study (20 volunteers, 5-point scale) ranks Ours highest on all four questions [supp. §3].

Paper vs code

Checked against raw/code/2409_10141 at commit f2a635126dca3bf0526b4a08723cb84a79b57cab. The repository is linked from the authors' project page. Discrepancies:

• Texture. Paper: per-vertex color optimisation, 100 steps at lr 0.0005 §4 eq. 9. Code: a comment says "Differing from paper, we use the texturing method in Unique3D", then projects the views onto the mesh with confidence threshold 0.2 and fills unseen vertices reconstruct.py reconstruct.py. clr_iters and lr_clr in the config are never read remesh.yaml remesh.yaml.
• Step counts. Paper §4 gives 700 / 100 / 100 steps for alignment / geometry / texture, but §3.2 says remeshing takes 700 steps. Code runs 100 SMPL-X alignment steps and 700 remeshing steps reconstruct.py remesh.yaml. §4 looks like it swaps alignment and geometry.
• Learning rates. Paper §4 gives 0.3 / 0.001 / 0.0005 for alignment / geometry / texture. Code uses lr 0.3 for the remeshing optimiser, Adam 1e-3 for hands and 3e-3 for other SMPL-X parameters opt.py reconstruct.py.
• Loss norm. Paper eqs 7-8 use L2 norms. Code uses mean absolute error (L1) on normals and masks reconstruct.py reconstruct.py.
• Noise blending weight. Paper eq. 5 is additive, with a binary face mask w. Code blends the face tokens into the front view with a fixed convex weight 0.5 / 0.5 and detaches the face tokens, so the blend passes no gradient to the face branch attn_processors.py attn_processors.py attn_processors.py.
• Face region. Paper: resize and pad. Code: bilinear resize to one third of the latent side, at a fixed top-middle position attn_processors.py.
• View weights. Not in the compiled paper. Code uses [1.0, 0.4, 0.8, 1.0, 0.8, 0.4] reconstruct.py.
• SMPL-X source. The paper does not name the estimator for PSHuman itself; it names PIXIE for baselines. The shipped configs read SMPL-X renders from a folder named smpl_image_pymaf inference-768-6view.yaml.
• Hands. Paper eq. 7 optimises translation, shape and pose only. Code also optimises both hand poses reconstruct.py. Hand replacement is off remesh.yaml.
• Classifier-free guidance. Code drops the condition 5 % of the time and uses guidance 3.0 at inference. Not in the compiled paper train-768-6view-onlyscan_face_smplx.yaml inference-768-6view.yaml.
• Remeshing schedule. Code updates the target edge length every 4 steps before step 500, every 2 steps until 800, then every step opt.py. The paper does not describe this.

Pitfalls

• Normals carry no depth. Two or four views give incomplete or unnatural shapes; six views are the stated compromise [supp. §2].
• Texture is not what the paper says. Implement the code's projection if you want the released results, or the paper's optimisation if you want its description. The two will differ.
• Edge-length limits depend on normalisation. The mesh is rescaled to a fixed size before remeshing, so the 0.01 to 0.1 limits only make sense in that frame reconstruct.py reconstruct.py.
• Face tokens are detached. The face branch receives no gradient through the blend, so "joint" training in the paper is weaker in code attn_processors.py.
• Training views are flipped. Left and back views are flipped horizontally during training to align them [supp. §2]. Get the convention wrong and every view is mirrored.
• Background and input. Background is removed with rembg, the foreground is resized to 720, padded to 768, and set to white. Other backgrounds are untested [supp. §2].
• Face detection. Training keeps only views with a clear face, found with InsightFace [supp. §2]. Inputs with unusual face-forward directions may lose the face branch.
• Robustness is limited. The paper reports tolerance to SMPL-X errors and to children and elderly subjects, but shows examples rather than metrics [supp. §3].
• Figure cross-reference. The supplement cites the diffusion block as "Fig. 3(b)" [supp. §2]. In the compiled order the pipeline figure, with panel (b) for the diffusion block, is Fig. 2, so that reference looks stale.
• Hidden cost. The reconstruction timings in [supp. §3] are per-stage on the authors' hardware, which the paper does not specify.
• SMPL-X model licence. The code relies on SMPL-X body models that the README says are borrowed from ECON and SIFU and downloaded separately README.md. Those models carry their own terms, which were not checked here. Confirm them before commercial use.
• Training data licences. THuman2.1, CustomHumans and CAPE were used for training and evaluation. Their licences were not checked here, and the paper's CC BY 4.0 license does not cover them.

Lineage

• Builds on: Era3D and Wonder3D for six-view multiview generation §3; Stable Diffusion 2.1-unclip as the base model §4; ICON and ECON for SMPL-X-based and normal-based reconstruction §3.2; Palfinger et al. for the remeshing optimiser §3.2.
• Compared against: PIFu, PIFuHD, PaMIR, ICON, ECON, GTA, SIFU, HiLo, SiTH §4; Magic123, DreamGaussian, Chupa, TeCH in the supplement [supp. §3].
• Superseded by: not known from this wiki.