DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models

In plain words

DreamAvatar builds a 3D person from a sentence such as "a knight in red armour" plus a rough body shape. It does not look up a 3D model. It grows the figure by repeatedly rendering a fuzzy 3D cloud of colour and density from random camera angles, asking a pretrained image generator whether each render looks like the sentence, and nudging the cloud so the answer improves.

The body shape comes from a standard parametric human model. It gives the cloud a rough starting outline, so the figure has a body in the right place and in the right pose. The paper keeps two versions of that outline in play at once: a rest-pose version, where parts hide each other least, which helps the texture come out complete, and a version bent into the target pose, which keeps the geometry faithful to that pose. Both are trained together.

Faces are a weak spot for this kind of method: the generator sometimes draws extra faces. The paper adds a second, close-up loss on the head so faces come out sharper and the extra faces are less likely.

The authors report that this beats earlier text-to-3D methods on people. The evidence is mostly visual comparisons and a small user study.

TL;DR

• Text plus SMPL shape and pose in, a textured 3D human avatar out (NeRF plus diffusion-based score distillation) §1 §3.2.
• The core moves are a body-model density prior, two linked observation spaces (canonical and posed) and a zoomed-in head loss against the multi-face problem §3.2.
• The released code implements only the canonical part, the SMPL density guide and the full-body VSD loss; the posed-space branch and the head loss are absent (see Paper vs code).
• Results are author-reported and mostly qualitative; the only numbers are a user-study figure (25 raters) §4.2.

Problem

Text-to-3D methods that optimise a NeRF with a 2D diffusion prior (for example DreamFusion and Magic3D) do not enforce a consistent human structure and often produce poor textures on human bodies §1. Common failures include multi-face ("Janus") artifacts §1. NeRF methods trained on human videos do not generate new subjects §2 Related Work, 3D human generative models. Pose control is not achievable by methods without a shape prior §4.1, poses paragraph.

Method

The NeRF is an MLP that maps a 3D point to density and colour §3.1, eq. 1. Images come from volume rendering along each ray §3.1, eq. 2. Training uses Variational Score Distillation (VSD) with a LoRA-adapted diffusion model [§3.1, eq. 3].

# DreamAvatar, one training iteration as the paper describes it  [§3.2]
# Inputs: prompt y, SMPL shape beta, target pose xi
# Updates: NeRF theta, deformation MLP, LoRA phi

def training_iteration(prompt, beta, xi):
    # Sample a camera c; ranges are not stated  [§3.1]
    camera = sample_camera()
    # Canonical render, density from SMPL in A-pose  [§3.2, eq. 7]
    rgb_canonical = render_canonical(nerf, camera, beta)
    # Posed render, points mapped back to canonical space  [§3.2, eq. 8]
    rgb_posed = render_posed(nerf, deform, camera, beta, xi)
    # Full-body VSD loss on both renders  [§3.2, eq. 3]
    loss_body = vsd(rgb_canonical, prompt, camera)
    loss_body = loss_body + vsd(rgb_posed, prompt, camera)
    # Zoomed-in head render  [§3.2, Zoomed-in head]
    rgb_head = render_head(nerf, camera, xi)
    # Head VSD with landmark ControlNet and back-view token  [§3.2, eq. 9]
    loss_head = vsd_head(rgb_head, prompt, camera, head_landmarks(xi))
    # Weights between the terms are not stated  [§3.2]
    loss = loss_body + loss_head
    optimizer_step(loss)

Density guide from the SMPL surface. The paper computes a density value for each point from its signed distance d to the SMPL surface [§3.2, SMPL-derived density fields]. The same construction is used for the canonical point and, with the target-pose surface, for the posed point.

# SMPL-derived density guide  [§3.2, SMPL-derived density fields]
# Input:  d, signed distance to the SMPL surface; a = 0.001
# Output: density offset added before the softplus

def smpl_density_guide(d, a):
    # Target value (1/a) * sigmoid(-d / a)  [§3.2]
    target = sigmoid(-d / a) / a
    # Inverse softplus of the target, floored at zero  [§3.2]
    # Naive log(exp(x) - 1) overflows for large x; see Pitfalls
    guide = log(exp(target) - 1.0)
    return max(0.0, guide)

Deformation field. Points in posed space are mapped to canonical space by the inverse of the SMPL skinning transform plus a learned correction [§3.2, Deformation field; the equation there is unnumbered].

# Posed point to canonical point  [§3.2, Deformation field]
# G is the skinning transform of the posed SMPL vertex nearest to x_p

def posed_to_canonical(x_posed, G, mlp):
    # Inverse articulated (LBS) deformation  [§3.2, Deformation field]
    x_skinned = inverse(G) @ x_posed
    # Non-rigid offset from an MLP on encoded input  [§3.2, Deformation field]
    return x_skinned + mlp(gamma(x_skinned))

The two spaces share one NeRF. A canonical point adds its SMPL guide to the NeRF output before the activation. A posed point adds its own guide to the NeRF output evaluated at its canonical point §3.2, eq. 7, eq. 8. The paper says the canonical space, with the least self-occlusion, helps textures, and the posed space helps geometry match the target pose and adds supervision [§3.2, Dual observation spaces].

The head loss renders a zoomed-in head image each iteration and computes a VSD loss on it. The conditioning uses a landmark ControlNet on the projected SMPL head landmarks and a learned back-view token. The gradient is the same form as eq. 3, with the extra conditioning §3.2, Zoomed-in head, eq. 9.

Key equations, transcribed in the card's notation:

C(\mathbf{r}) = \sum_i W_i \mathbf{c}_i, \quad W_i = \alpha_i \prod_{j<i} (1-\alpha_j) eq. 2

\nabla_\theta \mathcal{L}_{\text{VSD}}(\theta) \triangleq \mathbb{E}_{t,\epsilon,c}\!\left[\omega(t)\, \big(\epsilon_{\text{pre}}(\mathbf{x}_t,t,y) -\epsilon_\phi(\mathbf{x}_t,t,c,y)\big)\, \frac{\partial \mathbf{g}(\theta,c)}{\partial \theta}\right] eq. 3

The text describes \omega(t) only as a weighting that depends on the timestep, and writes it as w(t) there §3.1, eq. 3 text.

Hyperparameters

Paper-stated values come first. Code-only values are marked and are not in the paper. The paper states most training values as "not stated".

Results

Author-reported. Setup: 20 prompts rated by 25 volunteers on a 1 (worst) to 8 (best) scale, three criteria. Baselines: DreamFusion, Magic3D, Latent-NeRF, Fantasia3D, DreamWaltz, AvatarCLIP and ProlificDreamer §4.2. AvatarBooth and AvatarVerse were not compared because their code is not public [§4 Baseline methods]. The rating values below are the numbers printed on the user-study bars §4.2, user-study figure; the text itself gives no numbers.

Qualitative results: a diverse set of avatars (real people, film, anime and game characters) §4.1, and avatars in various poses, including extreme poses with severe self-occlusion §4.1, poses paragraph. Prompt editing and shape changes (thin, short, tall, fat) are shown as examples only §4.1. Ablations are visual only, with no metrics §4.3. No variance, significance test or per-prompt counts are reported.

Paper vs code

The official repository is yukangcao/DreamAvatar at code_commit. Its README describes the release as "canonical DreamAvatar" README.md. The code differs from the paper in these ways:

• No posed-space branch and no deformation field. The paper uses a canonical and a posed space joined by a learned deformation [§3.2, Dual observation spaces, Deformation field]. The geometry's forward pass takes one point and returns density, features and normal, with no inverse skinning or correction MLP implicit_volume.py. inv and deform_lbs are imported but never called implicit_volume.py. lbs runs for the posed mesh, and its transforms are never read implicit_volume.py.
• Density guide uses the fixed star-pose mesh only. The guide is the signed distance to v_star, the mesh in a fixed star pose implicit_volume.py, and that is the only SMPL mesh the density guide reads implicit_volume.py. The target-pose mesh therefore does not affect the density.
• Absolute value in the guide. The paper writes the guide with the signed distance d §3.2, SMPL-derived density fields. The code takes the absolute value before the sigmoid implicit_volume.py, so the guide peaks on the surface and falls off on both sides.
• Undocumented density bias. The code adds a blob bias to the pre-activation density, with scale 10 and std 0.5 implicit_volume.py dreamavatar.yaml. The paper's density equation has no such term §3.2, eq. 7.
• No zoomed-in head loss, no landmark ControlNet, no back-view token. The paper's head branch §3.2, Zoomed-in head, eq. 9 has no counterpart in the released system, which computes one VSD loss plus a LoRA loss per step dreamavatar.py. The code's Janus handling is view-dependent prompting with 30-degree front and back thresholds dreamavatar.yaml base.py.
• Extra loss terms the paper does not mention. A sparsity term with weight 10 dreamavatar.py dreamavatar.yaml, an opacity term whose weight stays at 0 for all 10,000 steps dreamavatar.py dreamavatar.yaml, and orientation and z-variance terms at weight 0 dreamavatar.yaml dreamavatar.yaml. The paper gives no total loss at all §3.2.
• Weighting and LoRA objective. The paper leaves omega(t) undefined §3.1. The code uses 1 - alpha_bar_t stable_diffusion_vsd_guidance.py. The LoRA is trained with a standard noise-prediction MSE on timesteps drawn uniformly over the whole range stable_diffusion_vsd_guidance.py. The paper does not define the LoRA training objective §3.1.
• Resolution schedule. The paper states no schedule. The code trains at 64x64 until step 5,000, then 512x512 dreamavatar.yaml.
• ControlNet absent. The paper uses a landmark ControlNet §3.2. The code does not reference ControlNet.
• Shape and pose hard-wired. The code reads betas and pose from one fixed file, SMPL_poses/0004_smpl.pkl implicit_volume.py. The star mesh that drives the density guide is built from those betas implicit_volume.py. The paper takes shape and pose as inputs §3.2; the config exposes no option to set them.
• Model files. The README's file list names SMPLX_NEUTRAL.pkl and related SMPL-X files README.md. The code builds smplx.body_models.SMPL from ./smpl_data implicit_volume.py. The file names do not match. Not run; check before use.
• Badge id. The README's arXiv badge reads 2306.03038, not this paper's id README.md.

Pitfalls

• Overflow in the density guide. The guide computes log(exp(x) - 1) with x = sigmoid(...)/a, which reaches 500 at the surface. In float32, exp(x) overflows above about 88, so the naive form returns inf near the surface. The code's own version has the same form implicit_volume.py. This is my reading of the expression; I did not run it. Use the stable form x + log(1 - exp(-x)) for x > 0 when reimplementing.
• Sign of d. The paper does not state which side of the SMPL surface has positive d. Reading the formula, density is high where sigmoid(-d/a) is large, so a positive d outside the body gives density inside. That is my reading, not the paper's wording. The released code uses |d| instead, which gives a surface-centred guide. These two choices give different densities.
• Total loss unspecified. The paper combines full-body VSD in two spaces, a head VSD and a LoRA loss without giving weights §3.2. The code uses unit weights for VSD and LoRA dreamavatar.yaml.
• Score distillation noise. The gradient in eq. 3 depends on the timestep weight, which the paper leaves open §3.1. The code uses 1 - alpha_bar_t, with timesteps from 2% to 98% of the range, and the upper bound annealed to 50% after 5,000 steps dreamavatar.yaml. These are the code's settings, not values the paper validates.
• Cost. A 512x512 render needs at least 48 GB VRAM per the README README.md. The paper reports one A40 §4. The 64x64 start is the cheap path dreamavatar.yaml.
• Ablations are qualitative. The claims that the head loss and the dual spaces help rest on figures without metrics §4.3. Treat them as the authors' view, not measured effects.
• Evaluation is small. 20 prompts and 25 raters, with no reported variance or significance test §4.2. Small gaps between baselines are not established by the card's sources.
• Pose dependence. The paper's pose control relies on the posed space, which the released code does not implement (see Paper vs code). Pose-driven results from the paper should not be expected from the released code.
• Not stated. Learning rate, optimiser, guidance scale, ray sampling, camera ranges and batch size are absent from the paper. Use the code values above and label them as code-derived.

Lineage

• Builds on: DreamFusion, arXiv:2209.14988 (score distillation sampling) §2; ProlificDreamer, arXiv:2305.16213 (variational score distillation, eq. 3) §3.1; threestudio, the code base the authors follow [§4 Implementation details]; HumanNeRF, the deformation field idea §3.2; the authors' earlier zoomed-in face work and landmark ControlNet head approach §3.2, Zoomed-in head.
• Compared against: DreamFusion arXiv:2209.14988, Magic3D arXiv:2211.10440, Latent-NeRF arXiv:2211.07600, ProlificDreamer arXiv:2305.16213, plus Fantasia3D, DreamWaltz, AvatarCLIP and TADA! §4.2.
• Superseded by: not known from content on this wiki.