Face recognition

Overview

Face recognition here means open-set matching: the people seen at test time are not the people seen in training. The two recipe cards, VGGFace2 and ArcFace, solve it the same way. A CNN is trained as a classifier over thousands of training identities, then the classifier layer is dropped and the layer before it serves as the face descriptor; two faces (or two sets of faces) are compared by the cosine similarity of their L2-normalized descriptors 1710.08092 §5.1 1801.07698 §4.1. Four further cards cover a detector that runs before recognition YOLO5Face, a larger web-scale dataset cleaned by self-training WebFace260M, a use of the recogniser as a loss for 3D face fitting GANFIT, and a use that goes the other way: making your own photos harder to train on Unlearnable Examples.

The common pipeline has five parts, and the cards vary different parts:

1. Training data: how many identities, how many images each, how varied, how clean. VGGFace2 contributes a dataset and keeps the rest plain 1710.08092 §5.1. WebFace260M scales the data to millions of identities and cleans it automatically.
2. Detection and cropping: face box and landmarks, then a crop. Detectors can be swapped without changing the recogniser 2105.12931 §4.4 (see the detection variant below).
3. Backbone: ResNet-50 / SE-ResNet-50 1710.08092 §5.1 or ResNet50 / ResNet100 with a BN-Dropout-FC-BN embedding head 1801.07698 §4.1.
4. Loss: plain softmax 1710.08092 §5.1 or an angular margin softmax 1801.07698 §3.1. ArcFace contributes the loss and a noise-robust variant of it.
5. Matching: per-image embeddings, averaged into template embeddings, compared by cosine 1710.08092 §5.1 1801.07698 §4.1.

Evaluation uses two tasks. Verification (1:1) asks whether two templates show the same person and reports TAR (or TPR) at a fixed false accept rate, e.g. TAR@FAR=1e-4. Identification (1:N) searches a gallery and reports Rank-N accuracy and, for open-set search, TPIR at a fixed FPIR 1710.08092 tab. 6, tab. 7 1801.07698 tab. 8. The IJB-A, IJB-B and IJB-C benchmarks score templates (sets of images and video frames) rather than single images 1710.08092 §5.1.

Variants

Training data: breadth, depth and variation (VGGFace2)

VGGFace2 argues that earlier public datasets gave either many identities or many images per identity, and none targeted pose and age variation 1710.08092 §1. It collects 3.31M images of 9,131 identities, 80 to 843 per identity, split into 8,631 training and 500 test identities 1710.08092 §3.1 / tab. 1. Pose and age variety come from extra search queries per name ("sideview", "very young"), and label noise is reduced by a six-stage pipeline: name screening, download, MTCNN detection, filtering by classifier score, near-duplicate removal, and score-banded manual cleaning aimed at more than 96% purity 1710.08092 §4 / tab. 2.

With the same ResNet-50 and softmax loss, training on VGGFace2 beats training on VGGFace and on MS-Celeb-1M (MS1M): 3.9% vs 10.6% vs 5.6% top-1 error on the VGGFace2 test set 1710.08092 tab. 3, and 0.895 vs 0.620 vs 0.851 TAR@FAR=0.001 on IJB-A 1710.08092 tab. 6. The authors stress that MS1M has about 10× as many identities 1710.08092 §5.3. Pre-training on MS1M and then fine-tuning on VGGFace2 is better still; with SE-ResNet-50 this is the paper's best setting, 0.921 TAR@FAR=1e-3 on IJB-A and 0.862 TAR@FAR=1e-4 on IJB-C 1710.08092 tab. 6, tab. 8. Card: VGGFace2: A dataset for recognising faces across pose and age.

Loss: angular margin softmax (ArcFace)

ArcFace normalizes features and class weights, scales the cosine logits by s=64, and adds an angle m=0.5 to the target class only: the target logit is s\cos(\theta_{y_i}+m) 1801.07698 eq. 3 / §4.1. In a ResNet50 / CASIA ablation it beats plain softmax (95.56 vs 94.39 CFP-FP accuracy), triplet loss, and the authors' reimplementations of SphereFace and CosFace 1801.07698 tab. 2. With the training data fixed to VGGFace2, a ResNet50 ArcFace reports 89.8 / 92.79 TPR@FPR=1e-4 on IJB-B / IJB-C, against 80.0 / 84.0 for the softmax-trained SENet50 baseline 1801.07698 tab. 7. The baseline numbers match the VGGFace2 card's SENet trained on VGGFace2 at FAR=1e-4 (0.800 IJB-B, 0.840 IJB-C) 1710.08092 tab. 7, tab. 8, but backbone and preprocessing differ between the two rows, so the gap is not only due to the loss. Variants and tuning are on Angular margin losses. Card: ArcFace: Additive Angular Margin Loss for Deep Face Recognition.

Cleaning noisy web data with the model (sub-center ArcFace)

VGGFace2 cleans with classifiers plus human annotators (about 3 months for the name list and 21 days for final filtering) 1710.08092 tab. 2. ArcFace instead cleans with the face model itself: train with K=3 sub-centers per class, keep each class's dominant sub-center, drop samples more than 75° from it, and retrain a K=1 model from scratch 1801.07698 §3.2 / §4.1. On MS1MV0, which the authors estimate is 47.1% to 54.4% noisy, IJB-C TPR@FPR=1e-4 for ResNet50 rises from 90.27 to 95.92; on Celeb500K it rises from 92.15 to 96.91 1801.07698 §1 / tab. 3. Merging cleaned MS1MV0 and Celeb500K gives IBUG-500K (11.96M images, 493K identities), on which ResNet100 ArcFace reports 96.02 / 97.27 on IJB-B / IJB-C 1801.07698 tab. 7.

Cleaning web-scale data by self-training (CAST, WebFace260M)

WebFace260M collects about 4M celebrity names and up to 200 search-engine images per name, giving 260.9M faces after detection and cropping 2103.04098 §2 2103.04098 tab. 2. Its cleaner, CAST, runs three self-training rounds. Each round uses the previous teacher, an ArcFace ResNet-100 that starts from one trained on MS1MV2, to embed every face; it clusters each name's folder with DBSCAN, keeps the largest cluster if it has more than two faces, then merges or drops folders by comparing their centres (merge above cosine 0.7, drop the smaller folder between 0.5 and 0.7). A new student is trained on the cleaned set and becomes the next teacher 2103.04098 §3 2103.04098 §5.4. The output, WebFace42M, has 2.06M identities and 42.47M faces after duplicate and benchmark-overlap removal 2103.04098 tab. 2.

On the MS1M cleaning comparison, ArcFace ResNet-100 rises from 93.45 IJB-C TAR@FAR=1e-4 on raw MS1M to 96.55 after three rounds; a fourth round gives 96.52, so the authors stop at three 2103.04098 tab. 7 2103.04098 §5.4. The same method on WebFace gives ArcFace ResNet-100 97.70 IJB-C, against 96.03 for MS1MV2 2103.04098 tab. 5. These are different backbones and data sizes from the sub-center numbers above, and the two cleaning methods are not compared on the same data, so their gains should not be ranked against each other 1801.07698 tab. 3 2103.04098 tab. 7.

Two cautions apply. The paper's code search found no release, and several steps are not specified: the DBSCAN eps and min_samples, how a folder centre is formed, and what happens below cosine 0.5 2103.04098 §3 2103.04098 Paper vs code. The dataset is also not free to use: Section 6 grants access only to licence signatories for research, although Table 1 labels it public 2103.04098 §6 2103.04098 tab. 1. The card treats this as unresolved, so do not assume commercial use of WebFace42M. Card: WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition.

Fitting a 3D face with the recogniser as a loss (GANFIT)

GANFIT recovers a 3D face, shape and skin colour, from one photo by gradient descent. A 3D morphable model supplies shape and expression, and a GAN trained on about 10,000 UV skin maps supplies a 512-value texture latent 1902.05978 §3.1 1902.05978 §4.1. The face-recognition network is used only to compare the input photo with the rendered result: the identity loss is the cosine distance between embeddings of the two, and a content loss sums normalised distances on intermediate layers 1902.05978 eq. 8 1902.05978 eq. 9. Pixel and 68-point landmark terms complete the loss, with weights 2.0, 50.0, 1.0 and 0.001 for identity, content, pixel and landmark respectively 1902.05978 §4.1. The identity network is ArcFace, but the checkpoint is not named and the paper says other networks work similarly 1902.05978 §3.3.1.

On the MICC Florence benchmark, the mean point-to-plane error in the cooperative setting is 0.95 mm for GANFIT against 1.50 mm for the best baseline, Genova et al. 1902.05978 tab. 1. Identity similarity on LFW, measured with VGG-Face embeddings, is above 0.5 for GANFIT against 0.35 for the same baseline 1902.05978 app. A. The cost is about 30 s per image on one GTX 1080 Ti, since every photo needs its own optimisation 1902.05978 §4.1. The paper does not give iteration counts or initial values, and the released repository contains only the MICC evaluation scripts, not the fitting code 1902.05978 Paper vs code. Card: GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction.

Preprocessing: enlarged box vs aligned crop

The two recipes crop differently. VGGFace2 enlarges the MTCNN box by a factor of 0.3 to cover the whole head, does not align, resizes the shorter side to 256 and crops 224×224 (random crop for training, centre crop for testing) 1710.08092 §4.3 / §5.1. ArcFace aligns 112×112 crops with five RetinaFace landmarks 1801.07698 §4.1. Neither card compares the two choices. What both stress is that test crops must be made exactly like training crops; VGGFace2 re-detects IJB-A faces with MTCNN for that reason 1710.08092 §5.3.

Detection before recognition (YOLO5Face)

YOLO5Face treats face detection as ordinary object detection: YOLOv5 with a five-point landmark head trained with wing loss, a stem block, smaller SPP kernels, a stride-64 output for large faces, and ShuffleNetV2 backbones for the smallest models 2105.12931 §3.2. The landmarks are there partly for recognition, to align faces before the recogniser runs 2105.12931 §3.3 2105.12931 §4.4.

On the WiderFace validation set (Easy / Medium / Hard), the author-reported numbers are: YOLOv5x6 96.67 / 95.08 / 86.55 at 141.2M parameters, SCRFD-34GF 96.06 / 94.92 / 85.29 at 9.8M, and YOLOv5n 93.61 / 91.54 / 80.53 at 1.7M parameters with 2.1 GFLOPs 2105.12931 tab. 3. The P6 output helps Easy and Medium on YOLOv5s (95.29 / 93.61 vs 94.31 / 92.52) and leaves Hard unchanged (83.13 vs 83.15) 2105.12931 tab. 2.

For recognition, the paper swaps the detector in an ArcFace-style pipeline and reports FNMR on a WebFace test set. RetinaFace gives 0.1065, YOLOv5s trained on WiderFace 0.1060, and YOLOv5m 0.1056 2105.12931 tab. 4. The gaps are at the fourth decimal place and the card reports no spread across runs, so treat them as parity rather than a ranking 2105.12931 §4.4. The paper gives no latency measurements on any hardware, so its real-time claim for mobile use is not backed by a timing table 2105.12931 §1 2105.12931 §5. Card: YOLO5Face: Why Reinventing a Face Detector.

Protecting your photos from training (unlearnable examples)

This is a defence on the data side, not a better recogniser. Each image gets a small, bounded noise pattern, found by an error-minimizing loop, so that a model trained on the marked images learns little that transfers to clean images 2101.04898 §3.1 2101.04898 §3.2. The paper's noise bound is L_\infty 8/255 for CIFAR-style data 2101.04898 §4 2101.04898 app. B.

For faces, the paper reports a WebFace case study with an Inception-ResNet model. Recognition accuracy on the 50 unlearnable identities is 16%, against 86% for clean identities 2101.04898 §4.5 2101.04898 fig. 4. When the whole training set is unlearnable, verification AUC on LFW falls to 0.5321 from 0.9975 on clean data 2101.04898 §4.5 2101.04898 fig. 7.

The protection depends on coverage. On CIFAR-10 with ResNet-18, accuracy stays high until most of the data is protected: at 80% unlearnable the clean-only 20% reaches 84.30% and the mixed set 87.23% (class-wise), against 16.42% at 100% 2101.04898 §4.3 2101.04898 tab. 2. Adversarial training with the same budget restores about 85% clean accuracy on unlearnable CIFAR-10 2101.04898 app. D. The card does not check the face configs, so the face numbers are paper-only. Card: Unlearnable Examples: Making Personal Data Unexploitable.

Template aggregation

Both average embeddings. VGGFace2 averages descriptors within each image or video (media), averages media into a template, then L2-normalizes; each video counts once however many frames it has 1710.08092 §5.1. ArcFace takes the mean of per-image features and does not say whether they are normalized before averaging 1801.07698 §4.1.

When to use which

Implementation sketch

Training
1. detect faces, get 5 landmarks (MTCNN, RetinaFace or YOLO5Face)
2. crop: either aligned 112x112 from landmarks (ArcFace)
         or enlarged box -> shorter side 256 -> 224x224 crop (VGGFace2)
3. backbone -> embedding (512-d via BN-Dropout-FC-BN, or 2048-d pool layer)
4. classifier over N training identities:
     softmax                         (VGGFace2 baselines)
     or normalized + angular margin  (ArcFace, see angular-margin-loss)
     optional: K=3 sub-centers -> clean -> retrain with K=1 (noisy data)
     optional: CAST self-training -> DBSCAN per name, merge/drop by centre,
               retrain, repeat for 3 rounds (web-scale data)
5. SGD; optional pre-train on a large set, fine-tune on a cleaner one
6. optional, on the data you publish: unlearnable noise before upload
   (defence only; see the unlearnable-examples variant)

Inference
1. same detection and crop as training
2. e = L2_normalize(backbone(crop))          # classifier discarded
3. template: mean of media embeddings (media = mean over an image's or video's crops),
             then L2_normalize
4. score(a, b) = cosine(t_a, t_b)
5. verification: accept if score > threshold set at the target FAR
   identification: rank gallery templates by score

Notes:

• Where VGGFace2 normalizes per image, per media and per template, ArcFace states only the mean 1710.08092 §5.1 1801.07698 §4.1; normalizing per image before averaging is a choice, not a documented requirement.
• Numerical issues of the margin loss (clamping before arccos, behaviour for \theta+m>\pi) are covered on Angular margin losses.
• Optimizer details: VGGFace2 gives SGD, batch 256, LR 0.1 with two ÷10 drops on plateau, and leaves momentum and weight decay unstated 1710.08092 §5.1; ArcFace gives SGD, momentum 0.9, weight decay 5e-4, batch 512, and an initial LR only for CASIA 1801.07698 §4.1. The cards list the remaining gaps.
• CAST's cleaning teacher for MS1M is trained on IMDB-Face, not MS1MV2; the WebFace cleaning starts from MS1MV2 2103.04098 §5.4 2103.04098 §3.
• Detector training values differ between the YOLO5Face paper and its code (learning rate, weight decay, batch size, landmark weight); use the card's Paper vs code list before copying any of them 2105.12931 §4.2 2105.12931 Paper vs code.

Contradictions

• Is MS-Celeb-1M a good training set? In VGGFace2, ResNet-50 trained on MS1M is behind ResNet-50 trained on VGGFace2 on IJB-A at every operating point except FAR=0.1, where both score 0.980 1710.08092 tab. 6 / §5.3. In ArcFace, the cleaned MS1MV3 is the strongest single training set for ResNet50 (96.50 IJB-C at FPR=1e-4, vs 92.79 for VGG2) 1801.07698 tab. 3, tab. 7. The papers use different versions of MS1M (VGGFace2's with benchmark overlap removed; ArcFace's raw MS1MV0 and cleaned MS1MV3) and different losses and crops, so the results do not directly conflict. Both point to label noise as the reason raw MS1M underperforms 1710.08092 §2 1801.07698 §1.

• Which benchmark ranks the 10% WebFace subset first? In WebFace260M, the 10% subset WebFace4M beats MS1MV2 on IJB-C with ArcFace ResNet-100 (96.77 vs 96.03), but trails it on MegaFace rank-1 (97.60 vs 98.40) 2103.04098 tab. 5. The same paper flags this mixed result 2103.04098 Pitfalls; the two benchmarks measure different things, so neither result overrides the other.

Papers

• VGGFace2: A dataset for recognising faces across pose and age: 3.31M-image, 9,131-identity dataset collected for pose and age variety with a six-stage cleaning pipeline, pose/age template protocols, and softmax ResNet-50 / SE-ResNet-50 baselines on IJB-A/B/C.
• ArcFace: Additive Angular Margin Loss for Deep Face Recognition: additive angular margin loss, sub-center ArcFace for automatic cleaning of noisy web data, IBUG-500K, and inversion of a trained face model into images.
• WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition: a 4M-name, 260M-face web collection and its 42M-face cleaned subset WebFace42M, cleaned by three rounds of CAST self-training, plus the FRUITS time-limited test set. Data access is licence-gated (see above).
• GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction: single-photo 3D face fitting with a GAN texture prior and face-recognition identity and content losses, evaluated on MICC. Reconstruction code is not released.
• Unlearnable Examples: Making Personal Data Unexploitable: error-minimizing noise that makes images hard to learn from, with a WebFace case study and partial-coverage and adversarial-training results.
• YOLO5Face: Why Reinventing a Face Detector: YOLOv5-based face detector with a five-point landmark head, from YOLOv5n to YOLOv5x6, with WiderFace results and a recognition-pipeline FNMR check.