wikillm

Paper · 2022-06-13 · cs.CV

RigNeRF: Fully Controllable Neural 3D Portraits

RigNeRF is a NeRF of one person's head and background, trained on a short smartphone video. Its deformation field starts from a 3D morphable face model (FLAME) and a small MLP adds residuals, so head pose, expression and camera view can be controlled. Author-reported: lowest LPIPS and face MSE of the compared methods on all four subjects.

Neural radiance field (NeRF) and variantsDynamic neural radiance fieldNovel view synthesisneural-renderingface-reanimation3d-morphable-modelhead-pose-control

arXivPDFMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Neural radiance fields can make very convincing views of a scene from new camera positions, but the scene is frozen: you cannot move a person's head or change their expression inside it. Other methods that do let a face move usually learn that motion as an opaque code, so nobody can say which "knob" turns the head.

RigNeRF gives those knobs a physical meaning. It borrows a generic 3D model of a human face, which already knows how a head turns and how a mouth moves, and uses it as a rough first guess for how each point in the scene should move. A small neural network then only learns the corrections the face model gets wrong, such as hair, glasses and the parts of the scene around the head. The result is one model per person, trained on a short video shot on a phone, that lets you change the head pose, the expression and the viewing angle.

TL;DR

RigNeRF is a neural radiance field whose deformation field is the sum of a 3D morphable face model (3DMM, here FLAME) deformation and a learned residual §3.2 eq. 4. It is trained on a short smartphone video of one subject and supports free-view rendering with explicit head-pose and expression control §4. Author-reported, it has the lowest LPIPS and face MSE of the compared methods on all four subjects tab. 1. The main risks are missing details that an implementer must fill in (optimizer, sampling, scale of the camera frame versus the face model) and a per-subject training cost.

Problem

Plain NeRFs are built for static scenes and cannot be controlled §2. Dynamic NeRF variants (for example HyperNeRF, Nerfies) learn a per-frame deformation conditioned on latent codes that have no physical meaning, so the deformation cannot be set by hand §2 §3.1. Parameterising the deformation directly by 3DMM pose and expression is not enough: the authors report artefacts during reanimation from lost rigidity of the head and wrong facial expressions §1 §3.2. A 3DMM is also only accurate on the face surface and is undefined elsewhere, so hair, glasses and the rest of the scene need a learned term §3.2. The paper targets one subject per model, trained on a 40 to 70 second video §4 §5.

Method

Convention: sections of the supplement are unnumbered in the source. They are cited as [§ Supplement] (Subsection). Numbered equations are cited by their order in the source: eq. 2 and eq. 3 have no label in the source, and the numbering of fig. N follows the order of figure floats in the source (see Pitfalls).

Overview fig. 1 §3. Each training frame i gives camera pose, a per-frame head pose and expression, and a per-frame deformation code and appearance code. For a pixel, rays are cast and sampled points are deformed into a canonical space. The canonical space is the head at zero pose and neutral expression, as defined by the 3DMM. A colour network then predicts colour and density at each deformed point, and volume rendering composes the pixel colour §3.1.

Base field (NeRF with a deformation). A NeRF maps a position and a view direction to colour and density; positional encodings are of the form sin and cos of 2^k times the input §3.1. A dynamic NeRF adds a deformation network D that maps each point to canonical space, conditioned on a per-frame deformation code ω_i, and a per-frame appearance code φ_i §3.1 eq. 1. Training minimises the L2 distance between rendered and ground-truth pixels §3.1.

3DMM deformation (prior). For frame i, head pose and expression parameters β_exp and β_pose are estimated with DECA and then refined by landmark fitting §3.2 §4. The 3DMM deformation at a point x is computed from its closest point x̂ on the mesh, damped by the exponential of the distance from x to x̂ §3.2 eq. 2. At a mesh point, the 3DMM deformation is the canonical position (zero pose, neutral expression) minus the position under the current β_exp and β_pose §3.2 eq. 3.

RigNeRF deformation. The full deformation is the 3DMM deformation plus a residual from D §3.2 eq. 4. D receives a positional encoding of x and of the 3DMM deformation, plus ω_i. The 3DMM term is encoded with 2 frequencies; conditioning D on the raw β parameters (100 expression dimensions in the authors' text) overfitted and generalised poorly §3.2. The canonical point is x plus the total deformation §3.2 eq. 4. The authors also rotate all points by the inverse head-pose rotation, so that the head is roughly aligned across frames; the order of this step relative to the others is not stated §3.2.

Colour network. The colour network F takes the positionally encoded canonical point, the encoded view direction, φ_i, features from the penultimate layer of D, and β_exp and β_pose, and outputs colour and density §3.3 eq. 5. Conditioning F on the D features and on the expression and pose is what the authors credit for sharper teeth and expression-dependent texture §3.3.

Training. Each subject has a 40 to 70 second phone video (about 1200 to 2100 frames), down-sampled to 256 by 256 §4. Cameras are estimated with COLMAP §4. The first half of the capture covers a wide range of expressions and speech with a still head; the second half covers head rotations §4. The per-frame codes ω_i and φ_i are optimised jointly with the networks by stochastic gradient descent §3.1.

Pseudocode. This is one sample evaluation, written in our words from §3.1 to §3.3. The paper does not write the volume-rendering quadrature out, so volume_render is a placeholder for the standard NeRF composite.

Pseudocode
# RigNeRF colour and density at one sample point of frame i  [§3]
# Inputs:  point x (3,), direction d (3,)
#          beta_exp, beta_pose: FLAME expression and pose of frame i  [§3.2]
#          omega: deformation code (8,)  [§ Supplement] (Experimental Details)
#          phi: appearance code (8,)     [§ Supplement] (Experimental Details)

def colour_and_density(point, direction, beta_exp, beta_pose, omega, phi):
    # Posed FLAME mesh for frame i  [§3.2]
    posed_mesh = flame_mesh(beta_exp, beta_pose)
    # Closest point on the mesh and its distance to x  [§3.2] [eq. 2]
    mesh_point, distance = closest_point(point, posed_mesh)
    # Canonical minus posed position of that mesh point  [§3.2] [eq. 3]
    mesh_shift = canonical_position(mesh_point) - mesh_point
    # 3DMM deformation at x: mesh shift divided by exp(distance)  [§3.2] [eq. 2]
    flame_deform = mesh_shift / exp(distance)            # (3,)
    # Residual from the deformation MLP D: encoded x (10 frequencies),
    # encoded 3DMM deformation (2 frequencies), deformation code  [§3.2] [eq. 4]
    residual, deform_features = deformation_mlp(
        encode(point, 10),
        encode(flame_deform, 2),
        omega)
    # Canonical point: x plus the total deformation  [§3.2] [eq. 4]
    canonical_point = point + flame_deform + residual
    # Colour and density from the colour MLP F  [§3.3] [eq. 5]
    colour, density = colour_mlp(
        encode(canonical_point, 10),
        encode(direction, 4),
        phi,
        deform_features,
        beta_exp,
        beta_pose)
    return colour, density


def render_pixel(origin, direction, depths, beta_exp, beta_pose, omega, phi):
    # Sample points on the camera ray, 128 per ray  [§3.1]
    # [§ Supplement] (Experimental Details)
    points = origin + depths[:, None] * direction        # (N, 3)
    colours, densities = [], []
    for point in points:
        # Evaluate the field at each sample  [§3.3] [eq. 5]
        colour, density = colour_and_density(
            point, direction, beta_exp, beta_pose, omega, phi)
        colours.append(colour)
        densities.append(density)
    # Volume rendering gives the pixel colour  [§3.1]
    return volume_render(colours, densities, depths)

Reanimation and evaluation. For reanimation, the appearance and deformation codes of the first frame are used, and the driving β_exp and β_pose come from a driving video §4. For evaluation on held-out frames, the deformation code ω_v of each validation image is optimised for 200 epochs with all other weights frozen, to minimise the rendering error for that frame §4.1 eq. 6. This step is applied to RigNeRF and HyperNeRF only §4.1.

Hyperparameters

Values not listed in the paper are marked "not stated". All values come from the paper's text and supplement, since no official code was found.

NameValueWhereNotes
Training video length40 to 70 s (about 1200 to 2100 frames)§4per subject
Training resolution256 by 256§4video down-sampled "due to compute restrictions"
Samples per ray128§ Supplement (Experimental Details)coarse and fine split not stated
Rays per batch1550§ Supplement (Experimental Details)
Position frequencies into F10§ Supplement (Experimental Details)encodes the canonical point
Direction frequencies into F4§ Supplement (Experimental Details)encodes the view direction
Position frequencies into D10§ Supplement (Experimental Details)encodes x in eq. 4
3DMM deformation frequencies into D2§3.2 eq. 4encodes the 3DMM deformation
Architecture of F8 layers, 256 units§ Supplement (Experimental Details)same as the original NeRF MLP
Architecture of D8 layers, 128 units§ Supplement (Experimental Details)
Learning rate, RigNeRF5e-4 initial, 5e-5 at end§ Supplement (Experimental Details)decay shape not stated
Training length, RigNeRF150000 (subjects 1, 2); 100000 (subjects 3, 4)§ Supplement (Experimental Details)the source table says "epochs"; unit as written
Deformation code size, RigNeRF8§ Supplement (Experimental Details)
Appearance code size, RigNeRF8§ Supplement (Experimental Details)
Coarse-to-fine regularisation on D40k "epochs"§ Supplement (Experimental Details)schedule and weight not stated
Vertex deformation regularisationused, weight not stated§4borrowed from Nerfies
Deformation code optimisation (evaluation)200 epochs§4.1 eq. 6only the code is optimised
Optimizernot stated
Hardware4 V100 GPUs§ Supplement (Experimental Details)
Capture deviceiPhone XR or iPhone 12§4
Expression and pose estimationDECA, then landmark fitting with 3DDFA-V2§4
Camera estimationCOLMAP§4
Baseline HyperNeRF learning rate1e-3 initial, 1e-4 at end§ Supplement (Experimental Details)for comparison only
Baseline NerFACE learning rate5e-4 initial, 5e-5 at end§ Supplement (Experimental Details)for comparison only

Results

Author-reported. Setup: four subjects, each a smartphone video of one person with a model trained per subject, evaluated on held-out frames with the camera, pose and expression of those frames §4 §4.1. Baselines are HyperNeRF, NerFACE and FOMM §4. PSNR is higher-is-better, LPIPS and face MSE are lower-is-better tab. 1.

SubjectModelPSNR ↑LPIPS ↓Face MSE ↓Source
1RigNeRF29.550.1369.6e-5tab. 1
1HyperNeRF24.580.228.14e-4tab. 1
1NerFACE24.20.2177.84e-4tab. 1
1FOMM11.450.4327.65e-3tab. 1
2RigNeRF29.360.1021e-4tab. 1
2HyperNeRF22.550.15469.48e-4tab. 1
2NerFACE24.570.1746.7e-4tab. 1
2FOMM12.70.5826.31e-3tab. 1
3RigNeRF28.390.1098e-5tab. 1
3HyperNeRF19.290.262.74e-3tab. 1
3NerFACE28.000.12921.2e-4tab. 1
3FOMM10.170.6011.7e-2tab. 1
4RigNeRF27.00.0922.3e-4tab. 1
4HyperNeRF21.190.1821.58e-3tab. 1
4NerFACE28.470.1342.7e-4tab. 1
4FOMM11.170.5296.8e-3tab. 1

Against HyperNeRF with added pose and expression control (HyperNeRF+E/P), which the authors built themselves, only two subjects were measured tab. 2.

SubjectModelPSNR ↑LPIPS ↓Face MSE ↓Source
1RigNeRF29.550.1369.6e-5tab. 2
1HyperNeRF+E/P31.30.1611.3e-4tab. 2
2RigNeRF29.360.1021e-4tab. 2
2HyperNeRF+E/P300.1161.9e-4tab. 2

Ablation of the architecture, reported in the supplement; the subject is not named § Supplement (Ablations). Config A feeds β to D directly, Config B feeds the 3DMM deformation to D, and Config C adds 3DMM-conditioned appearance (the final model).

ConfigPSNR ↑LPIPS ↓Face MSE ↓Source
A28.50.1831.5e-4§ Supplement (Ablations)
B29.00.1611.3e-4§ Supplement (Ablations)
C (final)29.450.1199.8e-5§ Supplement (Ablations)

Generalisation to unseen poses and expressions, LPIPS against the nearest training frame, reported in the supplement § Supplement (Generalization to Novel pose and expression). The subject is not named.

ProbeOffsetLPIPS ↓
Head yaw-15 / -10 / -5 / +5 / +10 / +15 degrees0.174 / 0.156 / 0.131 / 0.112 / 0.140 / 0.183
Head pitch0 / 5 / 10 / 15 degrees0.100 / 0.111 / 0.125 / 0.163
Jaw pose0 / 10 / 15 / 20 degrees0.110 / 0.115 / 0.120 / 0.125
ExpressionL2 0.001 / 0.010 / 0.100 / 0.3400.09 / 0.113 / 0.121 / 0.152

Qualitatively, fig. 4 and fig. 5 show reanimation with novel head pose and expression and novel view synthesis, and fig. 6 compares against HyperNeRF+E/P §4.2 §4.3. The authors also describe a general-purpose reanimation baseline (FOMM), which is image-based and cannot change the view §4.1.

Paper vs code

No official code.

Search, in order. (1) The paper gives no code link. The arXiv metadata has no comment field. (2) The project page [http://shahrukhathar.github.io/2022/06/06/RigNeRF.html] has one GitHub link, to the profile srxdev0619, not to a RigNeRF repository. That profile lists 37 repositories (checked on both pages). They are forks of NeRF, HyperNeRF and DECA and unrelated projects, with no RigNeRF repository. (3) GitHub repository search returned zero results for rignerf, for neural 3d portraits fully controllable, and for rignerf org:adobe. The authors list Adobe Research as affiliation.

Internal inconsistencies in the paper, to be checked against any reimplementation (see Pitfalls): the text says the method gives better reconstructions than HyperNeRF+E/P, but tab. 2 reports higher PSNR for HyperNeRF+E/P on both subjects. The ablation's Config C row (§ Supplement (Ablations)) does not equal the Subject 1 row of tab. 1 or tab. 2.

Pitfalls

  • No code to check against. Everything about the FLAME model, DECA and 3DDFA-V2 fitting, and COLMAP settings has to be taken from the referenced works. The paper does not give fitting settings or the FLAME version.
  • Scale between camera and face model is not stated. The damping exp(distance) in eq. 2 depends on the units of the distance. If the COLMAP world units and the FLAME mesh units differ, the 3DMM term reaches a different distance than the authors intended. The paper does not state the scale.
  • Closest point and which mesh. eq. 2 uses the closest point on "the mesh" without saying whether it is the posed or the canonical mesh. The sensible reading is the posed mesh, since the distance is to the current head. Points far from the mesh get almost no 3DMM term; no cut-off is stated.
  • Sign convention. The 3DMM deformation is canonical minus posed eq. 3, and the canonical point is x plus the deformation eq. 4. Flipping either sign still runs but maps points to the wrong place.
  • Rotation step is unplaced. The inverse head-pose rotation of all points is mentioned in the text §3.2, but not in eq. 4. Its order relative to the 3DMM lookup and to D is not stated.
  • Which point feeds the D features. eq. 5 writes the D features as a function of the canonical point, but §3.3 describes them as the penultimate layer of D applied to x. The two readings give different features. The paper does not say which is meant, so an implementer should pick one and test it.
  • Expression conditioning. The authors say conditioning D on the raw expression and pose parameters overfits, and they describe that code as 100-dimensional §3.2. The paper does not say whether the 100 refers to the expression only or to expression plus pose, and the pose dimension is not stated.
  • Epochs versus iterations. The supplement gives training lengths and the coarse-to-fine schedule in "epochs" (for example 40k epochs). With about 1200 to 2100 frames these numbers are hard to interpret as epochs. The unit should be read as iterations unless the authors say otherwise; the paper does not say.
  • Missing optimizer and schedule details. The optimizer, the decay shape between 5e-4 and 5e-5, the near and far bounds, whether coarse and fine networks are used, and the weight of the vertex regularisation are not stated.
  • Volume-rendering quadrature. The paper does not write out the composite formula. The NeRF composite is assumed, not stated.
  • Loss norm. eq. 6 uses a generic norm and the paper says the training loss is an L2 distance. The metric definitions (PSNR, LPIPS, face MSE) and how the face region is masked are not stated.
  • Table 2 contradicts the text. HyperNeRF+E/P has higher PSNR on both listed subjects (31.3 and 30 against 29.55 and 29.36). The text's claim of better reconstruction holds only for LPIPS and face MSE in that comparison.
  • Table 1 is not uniformly best. NerFACE has higher PSNR on subject 4 (28.47 against 27.0). RigNeRF is best on LPIPS and face MSE for all four subjects. The caption says "most metrics across all subjects", which is accurate.
  • Unequal evaluation. RigNeRF and HyperNeRF get a 200-epoch optimisation of the deformation code on each test frame; NerFACE and FOMM do not §4.1. The comparison therefore favours the two methods with a deformation code.
  • Reanimation uses first-frame codes. The reanimation results use the deformation and appearance codes of frame 1 §4. Controls that depend on these codes will differ from the reconstruction numbers.
  • Ablation row mismatch. Config C reports PSNR 29.45, LPIPS 0.119 and face MSE 9.8e-5; the Subject 1 row of tab. 1 reports 29.55, 0.136 and 9.6e-5. The subject and run behind the ablation are not named.
  • Subject-specific and capture-bound. One model per person, trained on a 40 to 70 second video; the authors say this is a limitation and note that camera registration quality directly affects results §5. The generalisation tables are in the supplement only and cover one unnamed subject.
  • Figure numbering. The source has six figure floats in the main text, in the order method, mesh deformation, depth, view-and-expression, all controls and HyperNeRF+E/P comparison. The supplement refers to reanimation results as "Figs. 1, 4, 5, 6 and 7", which does not match this order (no teaser is included in the source). The card uses source order; check numbers against the published PDF before citing.
  • Misuse. The authors say the method can be misused for deepfakes and point to classifiers and watermarking as mitigations § Supplement (Societal Impact).
  • Costs the abstract hides. Training is per subject, on four V100 GPUs § Supplement (Experimental Details), with 150000 "epochs" for two subjects and 100000 for the other two. No runtime per frame is reported. The rendering cost per pixel scales with 128 samples per ray and with two MLP evaluations per sample (D and F), and the paper gives no render time.

Lineage

  • Builds on: NeRF (arXiv:2003.08934) for the radiance field and volume rendering §3.1; dynamic deformation NeRFs, notably HyperNeRF (arXiv:2106.13228) and Nerfies, for the deformation and codes §2 §3.1; the 3DMM for the canonical space §3.2; DECA and 3DDFA-V2 for the fitting §4. The 3DMM and fitting tools have no card in this wiki.
  • Compared against: HyperNeRF (arXiv:2106.13228), NerFACE (arXiv:2012.03065), and FOMM (FOMM card) §4. HyperNeRF+E/P is the authors' own baseline built from HyperNeRF §4.3.
  • Superseded by: not known from content already in this wiki.

Source and license

Adapted from “RigNeRF: Fully Controllable Neural 3D Portraits” by ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, Zhixin Shu (arXiv:2206.06481), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.