General Facial Representation Learning in a Visual-Linguistic Manner

In plain words

Face tasks such as parsing and landmark placement are usually trained separately, each with its own hand-labelled data, which is expensive to make. This paper asks whether one general "face understanding" model, trained once, can serve all these jobs.

The authors collect about 20 million face photos that have web captions. They train an image model with two goals. The first matches each photo to its own caption and pushes it away from the captions of other photos, which teaches the model what the face shows in words. The second hides some small patches of the photo and asks the model to guess what was hidden, which keeps finer detail in the representation. After this training, the image model is reused: a small extra head is added for each task, and the model is either left frozen or fine-tuned.

The reported result is that this face-specific model transfers better than general-purpose pre-trained models on most of the tasks tested, and that it reaches new best numbers on several face parsing and landmark benchmarks.

TL;DR

• FaRL is a visual-linguistic pre-training recipe for face images: an image-text contrastive loss plus masked image modeling, on LAION-FACE (20M face image-text pairs) §3.1 §3.2 §3.3.
• The backbone is a ViT-B/16 image encoder; downstream heads are UPerNet-style for parsing and alignment and a small linear-combination head for attributes §3.4 §4.1.
• Author-reported: the frozen backbone beats ImageNet, web-image and CLIP-style baselines on LaPa, AFLW-19 and CelebA tab. 1. Fine-tuned FaRL at 448 px sets new results on LaPa, CelebAMask-HQ, AFLW-19 and WFLW tab. 4 tab. 5 tab. 6 tab. 7.
• Code is public (MIT) but contains only the downstream and model code, not the pre-training loop, so the pre-training recipe cannot be checked against code.

Problem

Face analysis tasks such as parsing, alignment and attribute recognition are usually solved by supervised training on manually labelled data, which is costly, especially for parsing and alignment §1. Pre-training on large general image sets (ImageNet, and later image-text pairs) had been shown to help transfer, but its effect on face tasks had been little studied §1 §2.2. The paper also assumes that web image-text pairs are a usable, if noisy, source of supervision §1 fig. 1. A face-specific subset is needed for this, since general image-text data contains many non-face images §4.3; the authors build one by face detection and filtering §3.1.

Method

Data (LAION-FACE). Start from LAION-400M image-text pairs. Run the RetinaFace face detector and keep images whose detection score is above 0.9. Randomly sample 20 million pairs from those; the result is LAION-FACE §3.1. The distribution of face counts per image is shown in fig. 2. Captions are noisy: the authors note that web texts are not always accurate fig. 1.

Image-text contrastive learning (ITC). For a batch of B pairs, the image encoder E_I gives a class-token feature and the text encoder E_T gives an end-of-sequence feature. Each goes through a small projection head into a metric embedding. The loss is an InfoNCE softmax in both directions, image to text (L_I) and text to image (L_T), averaged over the batch, with temperature sigma that is learned §3.2 eq. 1.

Masked image modeling (MIM). A second copy of each image has some patches replaced by one learnable mask vector §3.3 eq. 2. The masked image goes through E_I and then a one-layer Transformer E_MIM. For each masked position, a classifier predicts the index of the discrete visual token that a discrete VAE assigns to the original patch. The loss sums the negative log-likelihood over masked positions §3.3 eq. 3. The discrete VAE is the publicly available one of [ramesh2021zero]; the vocabulary size |V| is not stated §3.3.

Architecture. The image encoder is a 12-layer, 768-width ViT-B/16 with 224x224 input, 196 patches plus a class token, and 87M parameters. The text encoder follows CLIP: 12 layers, width 512, 8 heads, 77 tokens, 63M parameters. Both features are projected to 512 dimensions for the contrastive loss §3.4.

Input alignment. Each face is aligned to a mean face and cropped to 224 px §3.4 §4.3. If an image has several faces, one is chosen at random §3.4. The paper does not give the landmark method for this pre-training alignment; its appendix describes five-landmark mean-face alignment for LaPa parsing app. G.

One step, two passes. Every image is encoded twice: once unmasked for ITC, and once with up to 75 randomly masked patches for MIM §3.4 fig. 3.

Downstream use. For every task the heads read features from blocks 4, 6, 8 and 12 of E_I §4.1. Face parsing and landmark alignment use UPerNet on multi-scale feature maps §4.1. Landmarks are predicted as Gaussian heatmaps of 128x128 §4.1. Attribute recognition pools each selected layer (class token, mean and max over patch tokens), mixes them with learned weights, and applies a fully connected layer for multi-label binary classification §4.1. Two regimes are reported: a frozen backbone (the "FaRL" rows) and a fully fine-tuned backbone (the "FaRL_ft" rows). A 448 px variant is obtained by bicubic up-sampling of the position embeddings §4.4.

# FaRL pre-training step  [§3.2] [§3.3] [eq. 1] [eq. 2] [eq. 3]
# Input:  faces (B, 3, 224, 224), captions (B, 77) token ids
# Output: scalar loss; the weighting of the two terms is not stated

def pretrain_step(faces, captions):
    # Contrastive branch on the unmasked pair  [§3.2] [§3.4]
    image_emb = project_image(image_encoder(faces))  # (B, 512)
    text_emb = project_text(text_encoder(captions))  # (B, 512)
    # Raw dot products scaled by the learned temperature  [eq. 1]
    # Whether the embeddings are L2-normalised is not stated
    logits = image_emb @ text_emb.T / temperature  # (B, B)
    targets = arange(B)
    # L_I and L_T, each averaged over the batch  [eq. 1]
    loss_itc = cross_entropy(logits, targets)
    loss_itc = loss_itc + cross_entropy(logits.T, targets)

    # Masked branch: a second pass over the same images  [§3.3] [§3.4]
    # Up to 75 of the 196 patches are replaced by the mask vector
    masked_faces = mask_patches(faces, max_masked=75)
    hidden = mim_head(image_encoder(masked_faces))  # (B, 196, width)
    # Target: index of each patch's discrete token from the dVAE  [§3.3]
    tokens = dvae_encode(faces)  # (B, 196) integer ids
    # Negative log-likelihood summed over masked positions only  [eq. 3]
    logp = log_softmax(classifier(hidden[masked]))
    loss_mim = -logp[tokens[masked]].sum()

    return loss_itc + loss_mim  # how these two are weighted is not stated

# Face parsing forward pass, FaRL^448  [§3.5] [§4.1] [§4.4]
# Input:  image (B, 3, 448, 448), values in [0, 1]
# Output: class logits (B, 11, 512, 512) for LaPa

def parse_faces(image):
    # Normalise with the CLIP mean and std
    # [code: farl/network/transformers.py#L106-L109]
    image = (image - clip_mean) / clip_std
    # Patch-token maps after blocks 4, 6, 8 and 12  [§4.1]
    # [code: farl/network/transformers.py#L128]
    maps = [patch_tokens(block_k(image)) for k in (4, 6, 8, 12)]
    # Learned deconvolutions (x4, x2), identity, max-pool (x0.5)
    # [code: farl/network/transformers.py#L19-L41]
    maps = resample_fpn(maps)
    # UPerNet-style fusion, 768 channels, 1x1 conv to logits  [§4.1]
    # [code: farl/network/mmseg.py#L11-L26]
    logits = uper_head(maps)
    # Bilinear up-sampling to 512 x 512
    # [code: farl/experiments/face_parsing/network.py#L31]
    return upsample(logits, size=512)

Hyperparameters

Pre-training (from the paper; not verifiable in code, see Paper vs code):

Downstream values from the released code (farl/experiments/). They are the official configs, not the paper's text; where they differ, see Paper vs code.

Results

Author-reported. Setup: pre-training on LAION-FACE (20M). Downstream heads on a ViT-B/16 backbone at 224 px unless marked. Frozen rows train only heads; FaRL_ft rows fine-tune the whole backbone; 448 uses 448 px input.

Table 1: frozen backbones, LaPa, AFLW-19, CelebA tab. 1. Face Transformer uses patch size 8 and 20 layers §4.2 footnote.

Table 2: few-shot, frozen backbones tab. 2. Percent of downstream training data used. LaPa is F1-mean, AFLW-19 is NME_diag, CelebA is mAcc.

FaRL is best on LaPa at 1% and tied with CLIP at 10%, and best on CelebA at 1% and 10%. CLIP is better on AFLW-19 at 1% and 10% (1.30 and 1.11, against FaRL's 1.35 and 1.15) §4.2.

Table 3: ablation of pre-training components tab. 3. Frozen backbone. ITC: image-text contrastive. MIM_k: a k-layer MIM head. ALIGN: face alignment during pre-training. LAION-RANDOM: a random LAION subset of the same size.

Table 4: LaPa, fine-tuned, mean F1 tab. 4. Per-class F1 is in the table.

Table 5: CelebAMask-HQ, mean F1 tab. 5. Per-class F1 is in the table.

Table 6: AFLW-19 tab. 6. NME_diag on the Full and Frontal sets; NME_box and AUC_box^7 on Full.

Table 7: WFLW tab. 7. NME_io on Full and on six subsets; FR^10 and AUC^10 on Full.

Table 8: 300W tab. 8. NME_io.

Table 9: attribute recognition, few-shot and full-shot tab. 9. Mean accuracy. CelebA columns are training proportions; LFWA columns likewise.

At full shot on CelebA, FaRL_ft ranks second. The paper attributes the top result to a method that uses extra identity and attribute-relation information from the downstream data §4.4.

Table 10: FairFace race-group gaps tab. 10. Accuracy in percent, frozen backbone. The discrepancy is the difference between the non-white and white groups. The paper reads the gaps as "relatively moderate" compared with others §5.

Table 11: share of face images in pre-training tab. 11 app. D. Only ITC.

Table 12: comparison with other self-supervised face pre-training tab. 12 app. E. Same architecture and fine-tuning as FaRL.

Table 13: longer pre-training, fine-tuned at 448 tab. 13 app. F.

Table 14: tanh-warping factor alpha on LaPa tab. 14 app. G. Default is 0.8.

Other results, in words. A per-layer study finds that the best layer differs by task: the deepest layers serve attribute recognition, while layer 5 serves face parsing best fig. 6 app. C. The fused multi-layer features beat each single layer on LaPa and CelebA app. C. Text-driven face editing with FaRL in place of CLIP is shown qualitatively fig. 4 app. A. Grad-CAM maps localize regions for text queries fig. 5 app. B.

Paper vs code

The official repository is https://github.com/faceperceiver/farl at commit 3fb46941cb4444183442eb09748c310114bcb846, linked in the paper's author block. The code is MIT-licensed, Microsoft LICENSE. The README says the released backbone is "used in paper" README.md.

• Pre-training code is absent. The repository contains the model class with an optional MIM branch and a forward_mim method model.py model.py, but no contrastive loss, mask sampler, dVAE, data loader or pre-training config. The ITC and MIM hyperparameters in §3.4 therefore cannot be checked, and the code does not show how the two losses are weighted.
• Released weights are the paper's model. README gives two checkpoints, ep16 "used in paper" and ep64, with no license stated for either README.md README.md. The longer-training results in tab. 13 match the README's reproduced ep64 scores.
• Reproduced scores differ slightly. The README reports paper and reproduced values: LaPa 93.88 / 93.86; CelebAMask-HQ 89.56 / 89.65; AFLW-19 0.943 / 0.943; 300W 2.93 / 2.92; WFLW 3.96 / 3.98 README.md. It attributes the gaps to code refactoring README.md.
• Head loss is not as described. The paper uses a "soft-label cross-entropy" for landmarks §4.1. The code uses a per-pixel binary cross-entropy on the Gaussian heatmaps task.py plus a coordinate term with weight 1.0 aflw19_farl.yaml. The paper mentions no coordinate loss.
• Coordinate term is a distance, not L1. The term named coord_l1_loss computes the Euclidean norm per point task.py.
• Heatmap width. The paper says "one-pixel sigma" §4.1. The code renders heatmaps with a Gaussian-like kernel exp(-8 r^2 / R^2) and radius R = 5.0 p2i_max.h aflw19_farl.yaml. By our reading of that formula, sigma is R/4 = 1.25 px. The code's own default radius is 16.0 task.py.
• Alignment shift and augmentation. The paper gives translation 0.01 x s for alignment app. G. The configs use shift sigma 0.05 aflw19_farl.yaml. The configs also use mean scale 0.8 for 300W and WFLW ibug300w_farl.yaml wflw_farl.yaml; the paper does not state this. The code adds gray and gamma jitter and noise with p 0.5 train.yaml.
• Alignment schedule. The paper states no schedule. AFLW-19 and WFLW train 150 epochs and 300W trains 250, but the LR drop is at epoch 200 aflw19_farl.yaml ibug300w_farl.yaml wflw_farl.yaml refine_backbone.yaml. So AFLW-19 and WFLW never decay.
• Backbone learning rate. The paper uses one rate per task: 1e-3 for parsing and 0.01 for alignment heads §4.1. The configs fine-tune the backbone at 1e-4 with heads at 1e-3 (parsing) or 1e-2 (alignment) refine_backbone.yaml refine_backbone.yaml. The frozen configs match the paper's head rates freeze_backbone.yaml freeze_backbone.yaml.
• Parsing training length and EMA. The paper gives no epoch count or EMA. The config trains 300 epochs with an LR drop at 200 lapa_farl.yaml refine_backbone.yaml, and evaluates an EMA copy with decay 0.999 lapa_farl.yaml lapa_farl.yaml.
• Feature head. The paper describes UPerNet on the four layers §4.1. The code adds a learned multi-scale resampling before UPerNet (x4, x2, identity, max-pool x0.5) transformers.py transformers.py. The head uses 768 channels, pool scales 1, 2, 3, 6, dropout 0.1 and SyncBN mmseg.py. None of these are stated in the paper.
• Input normalization. The code uses CLIP's mean and std transformers.py. The paper does not state it.
• Resolution change. Position embeddings are bicubic-resized and images bilinearly resized to 448 transformers.py transformers.py. This matches §4.4.
• Parsing augmentation. The paper's ranges (+-18 deg, scale 0.9-1.1, shift 0.01 x s) match the configs' sigmas (rotation 0.314 rad, scale 0.1, shift 0.01) train.yaml. The sampler is in the external blueprint package, so whether the sigmas are uniform half-widths cannot be checked here.
• Tanh-warping. Default alpha 0.8 on LaPa matches app. G lapa_farl.yaml. CelebAMask-HQ uses no warping train.yaml. The warp itself is implemented in the external blueprint package.
• Input resolution. The README example and every shipped config use 448 README.md. The 224 px frozen rows need the trainer run with input_resolution=224 and the frozen optimizer; no such config is shipped.
• Attribute recognition. No attribute code is in the repository. The head, its LR of 0.3 and the 100-epoch schedule §4.1 cannot be checked.
• Contrastive gather. The paper says ITC gathers logits across GPUs app. G. No training code is present.
• Dataset licenses. The paper's Data Usage section lists non-commercial or research-only terms for LaPa, CelebAMask-HQ, the original AFLW, CelebA and 300W app. H. The repository's DS_DATA.md gives download sources for LaPa, CelebAMask-HQ, AFLW-19, 300W and WFLW, without license terms DS_DATA.md. No data is redistributed by this card.
• Latent bug, no effect in this code path. The backward pass of the point-to-image (p2i) op writes dy+dy where dy*dy is meant p2i_max.h. It only affects the gradient with respect to point coordinates. Ground-truth heatmaps are rendered under torch.no_grad() task.py, so training here never uses that gradient.

Pitfalls

• The pre-training recipe is partly unstated. The paper gives no weight between the ITC and MIM losses, no vocabulary size |V| for the dVAE, and no rule for how many patches are masked within "at most 75". Its learning-rate description ("warmed up to 1e-3 in one epoch and then cosine decay to 9e-4 in 15 epochs") does not say the rate after epoch 15. An implementer needs these choices, and the code does not supply them.
• Reproduced numbers. The repository's reproduced scores differ from the paper's by up to about 0.1 points on some metrics README.md. Use these as the expected spread when re-running the released configs.
• Frozen and fine-tuned rows answer different questions. Tables 1 to 3 are frozen-backbone comparisons. Tables 4 to 9 mix frozen (FaRL) and fine-tuned (FaRL_ft) rows, and the 448 rows also change input resolution. Do not read them as one comparison.
• The class default of the alignment heatmap radius is 16.0, but every shipped config uses 5.0 task.py aflw19_farl.yaml.
• Coordinate convention. Points are normalised with a +0.5 pixel offset geometry.py and denormalised with -0.5 geometry.py. Any conversion to or from another toolkit's pixel coordinates must match this offset.
• Attribute LR of 0.3. The paper states an AdamW learning rate of 0.3 with cosine decay over 100 epochs §4.1. This is unusually large for AdamW and no code is available to confirm it. Treat it as unverified.
• Cost of pre-training. Pre-training uses 32 V100s for 16 epochs §3.4, and the authors report that the ITC-plus-MIM setup needs roughly double the compute of CLIP-style training app. G. The downstream models have the same cost as other ViT-B/16 models at inference app. G.
• Data for commercial use. LaPa, CelebAMask-HQ, CelebA, the original AFLW and 300W carry non-commercial or research-only terms app. H. The LAION-FACE subset is derived from LAION-400M, which the paper says is CC BY 4.0 app. H. Neither the paper nor the repository states a license for the LAION-FACE subset itself. Before commercial fine-tuning, check each dataset's own terms.
• Face attributes are only partly verified. The attribute head, its training recipe and its 100-epoch schedule have no code here.

Lineage

• Builds on: image-text contrastive pre-training as in CLIP and ALIGN §2.1 §3.2; masked image modeling inspired by BEiT §1 §3.3; the discrete VAE of Ramesh et al. for MIM targets §3.3; UPerNet heads and the BEiT-style layer selection {4, 6, 8, 12} §4.1. The face-specific features are compared against Face Transformer (Zhong et al.) and against self-supervised face pre-training (SwAV, SimCLR) app. E.
• Compared against: MoCo v3, BEiT, ViT, DeiT, CLIP, Face Transformer (frozen) §4.2; AGRNet, EAGR, EHANet, BASS, Wei et al. (face parsing) §4.4; LAB, LUVLi, AWing, ADNet (alignment) §4.4; PS-MCNN, SSPL (attributes) §4.4.
• Superseded by: not stated in this wiki.