Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network

In plain words

Many earlier methods first fit a template face model to the photo and then adjust its settings. This paper skips that step. It draws a fixed square grid, like an unfolded face-shaped map, and asks a small neural network to fill each cell with the 3D position of the face point that belongs there. One picture in, one grid out, and the grid gives both the 3D shape and the location of every point on the face.

The network is a plain encoder-decoder trained with a loss that counts the central face more than the edges and ignores the neck. Because it needs one pass and no fitting, it takes 9.8 ms per image on the authors' GPU. The authors report that it beats earlier methods on the face-alignment and face-reconstruction benchmarks they tested.

TL;DR

A single RGB face image goes through a light residual encoder-decoder that regresses a 256x256 UV position map §3.1 §3.2. Each map cell holds the x, y, z position of one point of the full face, so the same output gives 68 landmarks, dense correspondence and 3D shape §3.1. The loss weights the 68 landmarks, eyes, nose and mouth most and sets the neck to zero §3.2 eq. 1 tab. 3. Results are author-reported.

Problem

Earlier 3D face methods either fit a 3D Morphable Model (3DMM) or a template, which limits the shape to the model's space and needs slow fitting or warping, or regress a volume or point set that is coarse, large or slow §1 §2. Dense alignment that also covers invisible points usually needs a separate step or network §2. The paper aims for one model-free network that returns the full 3D geometry and dense alignment together §1.

Method

1. Position map representation §3.1 fig. 2

• A UV position map is a 256x256 image whose three channels store the x, y and z coordinates of a 3D face point. Each map cell has a fixed UV location, so the cell also fixes which semantic point it represents §3.1.
• The 3D points are in a left-handed coordinate system. The origin is the upper-left corner of the input image and x grows to the right §3.1.
• The UV layout comes from Bas et al. (Tutte embedding with conformal Laplacian weights, mesh boundary mapped to a square) §3.1. The paper computes the training maps from the Basel Face Model (BFM) fits in 300W-LP §3.1 §3.3.
• Invisible points are included, unlike depth images or visible-only UV coordinates §3.1.

2. Network §3.2 fig. 3

• Input: a 256x256x3 face crop §3.2.
• Encoder: one convolution, then 10 residual blocks. The encoder maps the input to an 8x8x512 feature map §3.2.
• Decoder: 17 transposed convolutions that go back up to a 256x256x3 position map §3.2.
• All convolutions use kernel size 4 and ReLU activation, as stated §3.2.
• A single network handles alignment and reconstruction. No extra head is needed for either task §3.2.

3. Weighted loss §3.2 eq. 1 fig. 4

• The loss is a sum over map cells of the distance between predicted and ground-truth positions, multiplied by a per-cell weight W eq. 1.
• W has four regions: the 68 landmark cells, the eye, nose and mouth cells, the other face cells, and the neck cells. The ratio used is 16:4:3:0 §3.2 tab. 3.
• Neck cells get weight 0, so the network does not learn hair or clothing shapes §3.2.

4. Training data and augmentation §3.1 §3.3

• Training pairs come from 300W-LP, more than 60K images with 3DMM fits §3.1.
• Each image is cropped with its ground-truth bounding box and rescaled to 256x256. The ground-truth map is rendered from the fitted 3DMM parameters into UV space at 256x256 §3.3.
• Augmentation: random in-plane rotation of -45 to 45 degrees, translation of up to 10 percent of the input size, scale 0.9 to 1.2, colour-channel scaling 0.6 to 1.4, and synthetic occlusion by adding noise texture §3.3.

5. Optimisation §3.3

• Adam, learning rate 0.0001, halved after every 5 epochs, batch size 16, implemented in TensorFlow §3.3.

6. Inference §4.4 api.py

• One forward pass per cropped face gives the position map §4.4.
• 68 landmarks are read at fixed UV cells from the map api.py.
• Dense points are the face-region cells, about 45K points in the paper §4.2 §4.3.
• The paper times only the pass from cropped image to output geometry, not face detection §4.4.

Training and inference in pseudocode. The crop and the z handling for inference come from the code, not the paper.

# Training step: regress the position map  [§3.2] [eq. 1] [§3.3]
# Input:  images (B, 256, 256, 3) in 0..1; targets (B, 256, 256, 3) xyz
# Input:  weight mask (256, 256), from 16:4:3:0 region weights [§3.2]
# Output: scalar loss

def training_step(images, targets, weight_mask):
    # Encoder: one convolution, then 10 residual blocks  [§3.2]
    features = conv(images, kernel=4, stride=1)     # (B, 256, 256, 16)
    for block in residual_blocks:                   # 10 blocks, 5 with stride 2
        features = block(features)                  # ends at (B, 8, 8, 512)

    # Decoder: 17 transposed convolutions back to 256 x 256  [§3.2]
    raw = decoder(features)                         # (B, 256, 256, 3)

    # Output range is not stated in the paper; code uses a sigmoid times
    # 1.1 x 256 [code: predictor.py#L70] [code: predictor.py#L83]
    predicted = sigmoid(raw) * 281.6                # (B, 256, 256, 3)

    # Weighted per-cell error; neck cells have weight 0  [§3.2] [eq. 1]
    # Norm is written without a square and without a mean  [eq. 1]
    distance = norm(predicted - targets, axis=-1)   # (B, 256, 256)
    return sum(distance * weight_mask)

# Inference: crop, regress, map back to the image
# [§3.1] [code: api.py#L85-L128]
# Input:  image (H, W, 3) in 0..255; face box (left, right, top, bottom)
# Output: position map (256, 256, 3) in image coordinates

def predict_position_map(image, box):
    # Square crop: side = 1.6 x mean box side, centre from the box
    # Paper does not state the test-time crop [code: api.py#L95-L97]
    centre = box_centre(box)
    side = int(1.6 * mean_box_side(box))

    # Warp the crop to 256 x 256 and scale to 0..1
    # [code: api.py#L111-L116]
    crop = warp_similarity(image / 255.0, centre, side, output=256)

    # One forward pass gives the crop's position map  [§3.1] [§4.4]
    crop_map = network(crop)                        # (256, 256, 3)

    # Divide z by the crop scale, then undo the warp for x and y
    # [code: api.py#L125-L129]
    return undo_warp(crop_map)                      # (256, 256, 3)

Hyperparameters

Results

Author-reported. Setup: AFLW2000-3D (2000 images, 68 landmarks and about 45K dense points) and AFLW-LFPA (34 visible landmarks) for alignment; AFLW2000-3D and Florence for reconstruction; a GTX 1080 GPU and a Xeon E5-2640 v4 CPU for timing §4.1 §4.4.

3D face alignment, NME % (lower is better) tab. 1

Values for the other methods are taken from their papers, except 3D-FAN, which the authors re-ran §4.2 tab. 1. In the 68-landmark CED figure, PRN is slightly ahead of 3D-FAN in 2D and further ahead in 3D §4.2 fig. 5.

Dense alignment, AFLW2000-3D §4.2 fig. 7

• The paper reports a margin of more than 27 percent over the best of 3DDFA and DeFA on both 2D and 3D coordinates. The exact values are only in the CED plot.

3D reconstruction §4.3 fig. 8 fig. 9

• Florence (about 19K common points, VRN-Guided and 3DDFA compared): 28.7 percent relative improvement over VRN-Guided, read from the figure fig. 8.
• Against 3DDFA, DeFA and VRN-Guided on AFLW2000-3D, the CED curves are reported only as plots fig. 8. The paper also reports stable error across yaw angles where 3DDFA and VRN-Guided degrade fig. 9.

Runtime, milliseconds per image §4.4 tab. 2

Model size: 160 MB, against 1.5 GB for VRN §4.4. The contributions list claims more than 100 FPS §1.

Ablation, weight mask §4.5 tab. 3 fig. 10

The authors state that no mask is worst, and ratio 3 beats ratio 2 on the 68-point metric. The ablation values are only in fig. 10.

Paper vs code

The official repository is YadiraF/PRNet, linked in the abstract. It contains inference code only: the network, the cropping, the post-processing and a demo. Its README describes a training section, but no training script is in the repository README.md.

• Architecture matches the text: input and output are 256x256, 10 residual blocks, 17 transposed convolutions, and an 8x8x512 bottleneck predictor.py.
• Kernel size: the paper says 4 for all layers §3.2. The residual blocks also use 1x1 convolutions as bottlenecks predictor.py [code: predictor.py#L15].
• Output: the code applies a sigmoid and multiplies by 1.1 x 256 = 281.6 predictor.py predictor.py [code: predictor.py#L100]. The paper says nothing about this.
• Normalisation: the code uses batch norm and an L2 weight penalty of 0.0002 predictor.py predictor.py. Neither is in the paper.
• Input range: the image is divided by 255 to lie in 0 to 1 [code: api.py#L115].
• Test-time crop: the paper does not describe one. The code centres a square on the box and uses 1.6 times the mean box side (bounding-box path) [code: api.py#L95-L97], or 1.58 times with a 0.14 downward shift (dlib path) [code: api.py#L107-L108].
• Point count: the paper says "more than 45K" points §3.1 §4.2. Code comments say "about 40K" api.py and "45128 here" [code: api.py#L162]. The count depends on the face-index file.
• Evaluation on AFLW2000 uses the 68 ground-truth points from each .mat file to crop the image run_basics.py. The repository has no NME, CED or ICP evaluation script, so the paper's numbers cannot be re-run from the repository.
• Weight mask: the README says the mask is in Data/uv-data [code: README.md#L142]. The folder holds only canonical_vertices.npy, face_ind.txt, triangles.txt and uv_kpt_ind.txt. No mask file is present, and the exact masks are not recoverable from the repository.
• Trained weights: the clone has only the .index file of the checkpoint. The .data shard is not in the repository net-data. The README points to an external download and says the licence of the model file is open (issue 28) README.md. This card does not use or redistribute the weights.

Pitfalls

• Loss is ambiguous. Eq. 1 sums a norm of the position error, not a squared error, and the text calls MSE the common alternative §3.2 eq. 1. The paper does not say whether the per-point norm is over xyz, whether the sum is normalised, or whether the weight multiplies each channel. The code that would settle this is not released.
• Weight mask is not reproducible from the repository. The paper gives region ratios §3.2 and colours in fig. 4 fig. 4, but not the landmark neighbourhood size, the region boundaries, or the segmentation source. Only the ratio is in the text.
• Output scale and activation are not in the paper. Without the sigmoid and the 281.6 scaling used in the code, a regression head trained on raw coordinates will not match these weights predictor.py predictor.py. Confirm the target scale before training.
• Test-time crop changes the result. Training crops use the ground-truth box §3.3. The paper never specifies the test box, and the code's 1.6 and 1.58 factors differ between paths api.py api.py. Reproduce the alignment numbers only with the same box protocol.
• z-coordinate handling. The code divides z by the crop scale on the way out api.py. Without this, reconstruction is off by the crop scale. The paper does not state the z convention beyond "left-handed" §3.1.
• Architecture details are missing. Channel widths, residual-block strides and normalisation are absent from the paper. The code uses 16, 32, ..., 512 channels, with five stride-2 blocks predictor.py. Batch norm is in the code only.
• Training details are partly missing. Epochs, total steps, initialisation, Adam betas and the rotation, translation and occlusion distributions are not stated §3.3. The schedule halves the rate every 5 epochs §3.3, but the epoch count that sets the schedule is not given.
• Training code and weights are not released. Training must be rebuilt from the paper, including the UV parameterisation from Bas et al., which the repository only supports through its precomputed uv-data files §3.1 api.py.
• Ground truth depends on 3DMM fits. Training maps and the AFLW2000-3D ground truth both come from 3DMM fitting. The authors note that the AFLW2000 ground truth is itself disputed §3.1 §4.2. Reconstruction numbers on AFLW2000-3D therefore measure agreement with fits, not with true geometry. Florence is the independent check §4.1.
• Timing excludes detection and cropping. The 9.8 ms figure covers only the network from cropped image to geometry §4.4. The README reports about 0.01 s on GPU and 0.2 s on CPU README.md. Both timings are for different setups, so compare them with care.
• Baselines are mixed. 3DDFA and 3DSTN numbers are copied from their papers, while 3D-FAN is re-run tab. 1 §4.2. The pose table uses a random 696-face subset §4.2, while the CED plots use all 2000 images fig. 5. Subsets and protocols differ, so compare the means with care.
• Several headline claims exist only in plots. The 27 percent dense margin, the 28.7 percent Florence gain and the ablation curves are read from figures §4.2 §4.3 fig. 7 fig. 8 fig. 10. Use the table values where you can.

Lineage

• Builds on: residual blocks §3.2, Basel Face Model and 3DMM fitting for training data §3.1 §3.3, and the UV-space parameterisation of Bas et al. §3.1. The paper cites these as references without arXiv links here.
• Compared against: 3DDFA, DeFA, 3D-FAN and 3DSTN for alignment and runtime tab. 1 tab. 2; VRN-Guided and 3DDFA for reconstruction §4.3.
• Superseded by: not stated in this wiki.