--- kind: concept slug: face-recognition title: Face recognition summary: >- Deep face recognition maps each face crop to a fixed-length embedding and compares embeddings by cosine similarity, so unseen people can be verified or identified. Training is a classifier over many identities that is discarded at test time. Results depend on both the training data (breadth, pose and age variety, label noise) and the loss (plain softmax vs angular margin). tags: [face-recognition, computer-vision, metric-learning, embeddings] papers: ["1710.08092", "1801.07698", "2101.04898", "2103.04098", "2105.12931", "1902.05978"] related: [angular-margin-loss, noisy-labels, self-training, dataset-curation, model-inversion, data-poisoning, adversarial-robustness, object-detection, efficient-vision-backbone, gan-inversion, differentiable-rendering] status: draft updated_at: 2026-11-01 --- ## Overview Face recognition here means open-set matching: the people seen at test time are not the people seen in training. The two recipe cards, [VGGFace2](/papers/1710.08092) and [ArcFace](/papers/1801.07698), solve it the same way. A CNN is trained as a classifier over thousands of training identities, then the classifier layer is dropped and the layer before it serves as the face descriptor; two faces (or two sets of faces) are compared by the cosine similarity of their L2-normalized descriptors [arxiv:1710.08092 §5.1] [arxiv:1801.07698 §4.1]. Four further cards cover a detector that runs before recognition [YOLO5Face](/papers/2105.12931), a larger web-scale dataset cleaned by self-training [WebFace260M](/papers/2103.04098), a use of the recogniser as a loss for 3D face fitting [GANFIT](/papers/1902.05978), and a use that goes the other way: making your own photos harder to train on [Unlearnable Examples](/papers/2101.04898). The common pipeline has five parts, and the cards vary different parts: 1. **Training data**: how many identities, how many images each, how varied, how clean. [VGGFace2](/papers/1710.08092) contributes a dataset and keeps the rest plain [arxiv:1710.08092 §5.1]. [WebFace260M](/papers/2103.04098) scales the data to millions of identities and cleans it automatically. 2. **Detection and cropping**: face box and landmarks, then a crop. Detectors can be swapped without changing the recogniser [arxiv:2105.12931 §4.4] (see the detection variant below). 3. **Backbone**: ResNet-50 / SE-ResNet-50 [arxiv:1710.08092 §5.1] or ResNet50 / ResNet100 with a BN-Dropout-FC-BN embedding head [arxiv:1801.07698 §4.1]. 4. **Loss**: plain softmax [arxiv:1710.08092 §5.1] or an angular margin softmax [arxiv:1801.07698 §3.1]. [ArcFace](/papers/1801.07698) contributes the loss and a noise-robust variant of it. 5. **Matching**: per-image embeddings, averaged into template embeddings, compared by cosine [arxiv:1710.08092 §5.1] [arxiv:1801.07698 §4.1]. Evaluation uses two tasks. **Verification (1:1)** asks whether two templates show the same person and reports TAR (or TPR) at a fixed false accept rate, e.g. TAR@FAR=1e-4. **Identification (1:N)** searches a gallery and reports Rank-N accuracy and, for open-set search, TPIR at a fixed FPIR [arxiv:1710.08092 tab. 6, tab. 7] [arxiv:1801.07698 tab. 8]. The IJB-A, IJB-B and IJB-C benchmarks score templates (sets of images and video frames) rather than single images [arxiv:1710.08092 §5.1]. ## Variants ### Training data: breadth, depth and variation (VGGFace2) VGGFace2 argues that earlier public datasets gave either many identities or many images per identity, and none targeted pose and age variation [arxiv:1710.08092 §1]. It collects 3.31M images of 9,131 identities, 80 to 843 per identity, split into 8,631 training and 500 test identities [arxiv:1710.08092 §3.1 / tab. 1]. Pose and age variety come from extra search queries per name ("sideview", "very young"), and label noise is reduced by a six-stage pipeline: name screening, download, MTCNN detection, filtering by classifier score, near-duplicate removal, and score-banded manual cleaning aimed at more than 96% purity [arxiv:1710.08092 §4 / tab. 2]. With the same ResNet-50 and softmax loss, training on VGGFace2 beats training on VGGFace and on MS-Celeb-1M (MS1M): 3.9% vs 10.6% vs 5.6% top-1 error on the VGGFace2 test set [arxiv:1710.08092 tab. 3], and 0.895 vs 0.620 vs 0.851 TAR@FAR=0.001 on IJB-A [arxiv:1710.08092 tab. 6]. The authors stress that MS1M has about 10× as many identities [arxiv:1710.08092 §5.3]. Pre-training on MS1M and then fine-tuning on VGGFace2 is better still; with SE-ResNet-50 this is the paper's best setting, 0.921 TAR@FAR=1e-3 on IJB-A and 0.862 TAR@FAR=1e-4 on IJB-C [arxiv:1710.08092 tab. 6, tab. 8]. Card: [VGGFace2: A dataset for recognising faces across pose and age](/papers/1710.08092). ### Loss: angular margin softmax (ArcFace) ArcFace normalizes features and class weights, scales the cosine logits by $s=64$, and adds an angle $m=0.5$ to the target class only: the target logit is $s\cos(\theta_{y_i}+m)$ [arxiv:1801.07698 eq. 3 / §4.1]. In a ResNet50 / CASIA ablation it beats plain softmax (95.56 vs 94.39 CFP-FP accuracy), triplet loss, and the authors' reimplementations of SphereFace and CosFace [arxiv:1801.07698 tab. 2]. With the training data fixed to VGGFace2, a ResNet50 ArcFace reports 89.8 / 92.79 TPR@FPR=1e-4 on IJB-B / IJB-C, against 80.0 / 84.0 for the softmax-trained SENet50 baseline [arxiv:1801.07698 tab. 7]. The baseline numbers match the VGGFace2 card's SENet trained on VGGFace2 at FAR=1e-4 (0.800 IJB-B, 0.840 IJB-C) [arxiv:1710.08092 tab. 7, tab. 8], but backbone and preprocessing differ between the two rows, so the gap is not only due to the loss. Variants and tuning are on [Angular margin losses](/concepts/angular-margin-loss). Card: [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](/papers/1801.07698). ### Cleaning noisy web data with the model (sub-center ArcFace) VGGFace2 cleans with classifiers plus human annotators (about 3 months for the name list and 21 days for final filtering) [arxiv:1710.08092 tab. 2]. ArcFace instead cleans with the face model itself: train with $K=3$ sub-centers per class, keep each class's dominant sub-center, drop samples more than 75° from it, and retrain a $K=1$ model from scratch [arxiv:1801.07698 §3.2 / §4.1]. On MS1MV0, which the authors estimate is 47.1% to 54.4% noisy, IJB-C TPR@FPR=1e-4 for ResNet50 rises from 90.27 to 95.92; on Celeb500K it rises from 92.15 to 96.91 [arxiv:1801.07698 §1 / tab. 3]. Merging cleaned MS1MV0 and Celeb500K gives IBUG-500K (11.96M images, 493K identities), on which ResNet100 ArcFace reports 96.02 / 97.27 on IJB-B / IJB-C [arxiv:1801.07698 tab. 7]. ### Cleaning web-scale data by self-training (CAST, WebFace260M) WebFace260M collects about 4M celebrity names and up to 200 search-engine images per name, giving 260.9M faces after detection and cropping [arxiv:2103.04098 §2] [arxiv:2103.04098 tab. 2]. Its cleaner, CAST, runs three self-training rounds. Each round uses the previous teacher, an ArcFace ResNet-100 that starts from one trained on MS1MV2, to embed every face; it clusters each name's folder with DBSCAN, keeps the largest cluster if it has more than two faces, then merges or drops folders by comparing their centres (merge above cosine 0.7, drop the smaller folder between 0.5 and 0.7). A new student is trained on the cleaned set and becomes the next teacher [arxiv:2103.04098 §3] [arxiv:2103.04098 §5.4]. The output, WebFace42M, has 2.06M identities and 42.47M faces after duplicate and benchmark-overlap removal [arxiv:2103.04098 tab. 2]. On the MS1M cleaning comparison, ArcFace ResNet-100 rises from 93.45 IJB-C TAR@FAR=1e-4 on raw MS1M to 96.55 after three rounds; a fourth round gives 96.52, so the authors stop at three [arxiv:2103.04098 tab. 7] [arxiv:2103.04098 §5.4]. The same method on WebFace gives ArcFace ResNet-100 97.70 IJB-C, against 96.03 for MS1MV2 [arxiv:2103.04098 tab. 5]. These are different backbones and data sizes from the sub-center numbers above, and the two cleaning methods are not compared on the same data, so their gains should not be ranked against each other [arxiv:1801.07698 tab. 3] [arxiv:2103.04098 tab. 7]. Two cautions apply. The paper's code search found no release, and several steps are not specified: the DBSCAN eps and min_samples, how a folder centre is formed, and what happens below cosine 0.5 [arxiv:2103.04098 §3] [arxiv:2103.04098 Paper vs code]. The dataset is also not free to use: Section 6 grants access only to licence signatories for research, although Table 1 labels it public [arxiv:2103.04098 §6] [arxiv:2103.04098 tab. 1]. The card treats this as unresolved, so do not assume commercial use of WebFace42M. Card: [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](/papers/2103.04098). ### Fitting a 3D face with the recogniser as a loss (GANFIT) GANFIT recovers a 3D face, shape and skin colour, from one photo by gradient descent. A 3D morphable model supplies shape and expression, and a GAN trained on about 10,000 UV skin maps supplies a 512-value texture latent [arxiv:1902.05978 §3.1] [arxiv:1902.05978 §4.1]. The face-recognition network is used only to compare the input photo with the rendered result: the identity loss is the cosine distance between embeddings of the two, and a content loss sums normalised distances on intermediate layers [arxiv:1902.05978 eq. 8] [arxiv:1902.05978 eq. 9]. Pixel and 68-point landmark terms complete the loss, with weights 2.0, 50.0, 1.0 and 0.001 for identity, content, pixel and landmark respectively [arxiv:1902.05978 §4.1]. The identity network is ArcFace, but the checkpoint is not named and the paper says other networks work similarly [arxiv:1902.05978 §3.3.1]. On the MICC Florence benchmark, the mean point-to-plane error in the cooperative setting is 0.95 mm for GANFIT against 1.50 mm for the best baseline, Genova et al. [arxiv:1902.05978 tab. 1]. Identity similarity on LFW, measured with VGG-Face embeddings, is above 0.5 for GANFIT against 0.35 for the same baseline [arxiv:1902.05978 app. A]. The cost is about 30 s per image on one GTX 1080 Ti, since every photo needs its own optimisation [arxiv:1902.05978 §4.1]. The paper does not give iteration counts or initial values, and the released repository contains only the MICC evaluation scripts, not the fitting code [arxiv:1902.05978 Paper vs code]. Card: [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](/papers/1902.05978). ### Preprocessing: enlarged box vs aligned crop The two recipes crop differently. VGGFace2 enlarges the MTCNN box by a factor of 0.3 to cover the whole head, does not align, resizes the shorter side to 256 and crops 224×224 (random crop for training, centre crop for testing) [arxiv:1710.08092 §4.3 / §5.1]. ArcFace aligns 112×112 crops with five RetinaFace landmarks [arxiv:1801.07698 §4.1]. Neither card compares the two choices. What both stress is that test crops must be made exactly like training crops; VGGFace2 re-detects IJB-A faces with MTCNN for that reason [arxiv:1710.08092 §5.3]. ### Detection before recognition (YOLO5Face) YOLO5Face treats face detection as ordinary object detection: YOLOv5 with a five-point landmark head trained with wing loss, a stem block, smaller SPP kernels, a stride-64 output for large faces, and ShuffleNetV2 backbones for the smallest models [arxiv:2105.12931 §3.2]. The landmarks are there partly for recognition, to align faces before the recogniser runs [arxiv:2105.12931 §3.3] [arxiv:2105.12931 §4.4]. On the WiderFace validation set (Easy / Medium / Hard), the author-reported numbers are: YOLOv5x6 96.67 / 95.08 / 86.55 at 141.2M parameters, SCRFD-34GF 96.06 / 94.92 / 85.29 at 9.8M, and YOLOv5n 93.61 / 91.54 / 80.53 at 1.7M parameters with 2.1 GFLOPs [arxiv:2105.12931 tab. 3]. The P6 output helps Easy and Medium on YOLOv5s (95.29 / 93.61 vs 94.31 / 92.52) and leaves Hard unchanged (83.13 vs 83.15) [arxiv:2105.12931 tab. 2]. For recognition, the paper swaps the detector in an ArcFace-style pipeline and reports FNMR on a WebFace test set. RetinaFace gives 0.1065, YOLOv5s trained on WiderFace 0.1060, and YOLOv5m 0.1056 [arxiv:2105.12931 tab. 4]. The gaps are at the fourth decimal place and the card reports no spread across runs, so treat them as parity rather than a ranking [arxiv:2105.12931 §4.4]. The paper gives no latency measurements on any hardware, so its real-time claim for mobile use is not backed by a timing table [arxiv:2105.12931 §1] [arxiv:2105.12931 §5]. Card: [YOLO5Face: Why Reinventing a Face Detector](/papers/2105.12931). ### Protecting your photos from training (unlearnable examples) This is a defence on the data side, not a better recogniser. Each image gets a small, bounded noise pattern, found by an error-minimizing loop, so that a model trained on the marked images learns little that transfers to clean images [arxiv:2101.04898 §3.1] [arxiv:2101.04898 §3.2]. The paper's noise bound is $L_\infty$ 8/255 for CIFAR-style data [arxiv:2101.04898 §4] [arxiv:2101.04898 app. B]. For faces, the paper reports a WebFace case study with an Inception-ResNet model. Recognition accuracy on the 50 unlearnable identities is 16%, against 86% for clean identities [arxiv:2101.04898 §4.5] [arxiv:2101.04898 fig. 4]. When the whole training set is unlearnable, verification AUC on LFW falls to 0.5321 from 0.9975 on clean data [arxiv:2101.04898 §4.5] [arxiv:2101.04898 fig. 7]. The protection depends on coverage. On CIFAR-10 with ResNet-18, accuracy stays high until most of the data is protected: at 80% unlearnable the clean-only 20% reaches 84.30% and the mixed set 87.23% (class-wise), against 16.42% at 100% [arxiv:2101.04898 §4.3] [arxiv:2101.04898 tab. 2]. Adversarial training with the same budget restores about 85% clean accuracy on unlearnable CIFAR-10 [arxiv:2101.04898 app. D]. The card does not check the face configs, so the face numbers are paper-only. Card: [Unlearnable Examples: Making Personal Data Unexploitable](/papers/2101.04898). ### Template aggregation Both average embeddings. VGGFace2 averages descriptors within each image or video (media), averages media into a template, then L2-normalizes; each video counts once however many frames it has [arxiv:1710.08092 §5.1]. ArcFace takes the mean of per-image features and does not say whether they are normalized before averaging [arxiv:1801.07698 §4.1]. ## When to use which | Situation | Use | Why | |---|---|---| | Building a model from public data, default recipe | Angular margin loss (ArcFace, $s=64$, $m=0.5$) on a large clean dataset, 5-point aligned 112×112 crops, 512-d embedding | Highest IJB-B / IJB-C numbers on either recipe card [arxiv:1801.07698 tab. 7] | | Training set is fixed and small (e.g. only VGGFace2) | Switch the loss before anything else | On VGGFace2, ArcFace R50 reports 92.79 IJB-C at FPR=1e-4 vs 84.0 for the softmax SENet50 baseline [arxiv:1801.07698 tab. 7]; backbones differ | | Need robustness to pose (profile) or age gaps | Train on data collected for that variation, e.g. VGGFace2 | VGGFace2-trained ResNet-50 has higher front-to-profile (0.6222 vs 0.4821 / 0.4715) and young-to-mature (0.5524 vs 0.4338 / 0.4005) similarity than VGGFace- or MS1M-trained ones [arxiv:1710.08092 tab. 4, tab. 5] | | Have a large noisy web scrape, no annotators | Sub-center ArcFace cleaning, then retrain | +5.65 IJB-C on MS1MV0 and +4.76 on Celeb500K for ResNet50 [arxiv:1801.07698 tab. 3] | | Millions of web images, no annotators, GPU budget for several retraining rounds | Self-training cleaning (CAST) before final training, three rounds | MS1M IJB-C 93.45 to 96.55 after three rounds [arxiv:2103.04098 tab. 7]; the gain comes from the authors' own pipeline, and no code is released [arxiv:2103.04098 Paper vs code] | | Planning to use a web-scale dataset such as WebFace42M commercially | Do not use it until its licence is checked | Access is granted to signatories for research use only [arxiv:2103.04098 §6], although Table 1 labels it public [arxiv:2103.04098 tab. 1] | | Need a textured 3D face from one photo, offline, per-image cost acceptable | Fit a 3DMM with a GAN texture prior and face-network losses (GANFIT) | 0.95 vs 1.50 mm mean error on MICC cooperative [arxiv:1902.05978 tab. 1]; about 30 s per image on one GTX 1080 Ti [arxiv:1902.05978 §4.1] | | Building a new dataset and can afford human review | VGGFace2-style staged filtering | Targets more than 96% purity, but takes months of human time and leaves several steps under-specified [arxiv:1710.08092 tab. 2 / §4] | | Large noisy set plus a smaller clean, varied set | Pre-train on the large one, fine-tune on the clean one | MS1M pre-training plus VGGFace2 fine-tuning beats VGGFace2-only on most IJB-A/B/C metrics for both backbones; the exceptions are ties or small drops at Rank-5/10 with SENet [arxiv:1710.08092 tab. 6, tab. 7, tab. 8] | | Compute budget under 1 GFlop | Small backbone (EfficientNet-B0) with ArcFace, plus distillation from a large teacher | 88.65 vs 86.44 on the LFR2019 image track with vs without distillation [arxiv:1801.07698 tab. 9 / §4.4] | | Need a detector that also gives five landmarks for alignment, on a phone or embedded device | YOLO5Face YOLOv5n (ShuffleNetV2) | 80.53 on WiderFace Hard at 1.7M parameters and 2.1 GFLOPs [arxiv:2105.12931 tab. 3]; no latency measured [arxiv:2105.12931 §5] | | Need the strongest detector on the WiderFace Hard subset, server-side | YOLO5Face YOLOv5x6, or SCRFD-34GF from its comparison table | 86.55 vs 85.29 on Hard, with 141M vs 9.8M parameters [arxiv:2105.12931 tab. 3] | | Need to stop others from training on your own photos | Unlearnable-example noise on everything you publish | Protection holds only near full coverage and is weak under adversarial training; it is not a privacy guarantee [arxiv:2101.04898 §4.3] [arxiv:2101.04898 app. D] | | Comparing to published numbers | Match benchmark version and protocol exactly | MegaFace refined vs original gives 98.98 vs 81.43 for the same model [arxiv:1801.07698 tab. 6]; prior IJB-B/C baselines in VGGFace2 are read off curves with a different gallery protocol [arxiv:1710.08092 tab. 7, tab. 8] | ## Implementation sketch ```text Training 1. detect faces, get 5 landmarks (MTCNN, RetinaFace or YOLO5Face) 2. crop: either aligned 112x112 from landmarks (ArcFace) or enlarged box -> shorter side 256 -> 224x224 crop (VGGFace2) 3. backbone -> embedding (512-d via BN-Dropout-FC-BN, or 2048-d pool layer) 4. classifier over N training identities: softmax (VGGFace2 baselines) or normalized + angular margin (ArcFace, see angular-margin-loss) optional: K=3 sub-centers -> clean -> retrain with K=1 (noisy data) optional: CAST self-training -> DBSCAN per name, merge/drop by centre, retrain, repeat for 3 rounds (web-scale data) 5. SGD; optional pre-train on a large set, fine-tune on a cleaner one 6. optional, on the data you publish: unlearnable noise before upload (defence only; see the unlearnable-examples variant) Inference 1. same detection and crop as training 2. e = L2_normalize(backbone(crop)) # classifier discarded 3. template: mean of media embeddings (media = mean over an image's or video's crops), then L2_normalize 4. score(a, b) = cosine(t_a, t_b) 5. verification: accept if score > threshold set at the target FAR identification: rank gallery templates by score ``` Notes: - Where VGGFace2 normalizes per image, per media and per template, ArcFace states only the mean [arxiv:1710.08092 §5.1] [arxiv:1801.07698 §4.1]; normalizing per image before averaging is a choice, not a documented requirement. - Numerical issues of the margin loss (clamping before arccos, behaviour for $\theta+m>\pi$) are covered on [Angular margin losses](/concepts/angular-margin-loss). - Optimizer details: VGGFace2 gives SGD, batch 256, LR 0.1 with two ÷10 drops on plateau, and leaves momentum and weight decay unstated [arxiv:1710.08092 §5.1]; ArcFace gives SGD, momentum 0.9, weight decay 5e-4, batch 512, and an initial LR only for CASIA [arxiv:1801.07698 §4.1]. The cards list the remaining gaps. - CAST's cleaning teacher for MS1M is trained on IMDB-Face, not MS1MV2; the WebFace cleaning starts from MS1MV2 [arxiv:2103.04098 §5.4] [arxiv:2103.04098 §3]. - Detector training values differ between the YOLO5Face paper and its code (learning rate, weight decay, batch size, landmark weight); use the card's Paper vs code list before copying any of them [arxiv:2105.12931 §4.2] [arxiv:2105.12931 Paper vs code]. ## Contradictions - **Is MS-Celeb-1M a good training set?** In VGGFace2, ResNet-50 trained on MS1M is behind ResNet-50 trained on VGGFace2 on IJB-A at every operating point except FAR=0.1, where both score 0.980 [arxiv:1710.08092 tab. 6 / §5.3]. In ArcFace, the cleaned MS1MV3 is the strongest single training set for ResNet50 (96.50 IJB-C at FPR=1e-4, vs 92.79 for VGG2) [arxiv:1801.07698 tab. 3, tab. 7]. The papers use different versions of MS1M (VGGFace2's with benchmark overlap removed; ArcFace's raw MS1MV0 and cleaned MS1MV3) and different losses and crops, so the results do not directly conflict. Both point to label noise as the reason raw MS1M underperforms [arxiv:1710.08092 §2] [arxiv:1801.07698 §1]. - **Which benchmark ranks the 10% WebFace subset first?** In WebFace260M, the 10% subset WebFace4M beats MS1MV2 on IJB-C with ArcFace ResNet-100 (96.77 vs 96.03), but trails it on MegaFace rank-1 (97.60 vs 98.40) [arxiv:2103.04098 tab. 5]. The same paper flags this mixed result [arxiv:2103.04098 Pitfalls]; the two benchmarks measure different things, so neither result overrides the other. ## Papers - [VGGFace2: A dataset for recognising faces across pose and age](/papers/1710.08092): 3.31M-image, 9,131-identity dataset collected for pose and age variety with a six-stage cleaning pipeline, pose/age template protocols, and softmax ResNet-50 / SE-ResNet-50 baselines on IJB-A/B/C. - [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](/papers/1801.07698): additive angular margin loss, sub-center ArcFace for automatic cleaning of noisy web data, IBUG-500K, and inversion of a trained face model into images. - [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](/papers/2103.04098): a 4M-name, 260M-face web collection and its 42M-face cleaned subset WebFace42M, cleaned by three rounds of CAST self-training, plus the FRUITS time-limited test set. Data access is licence-gated (see above). - [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](/papers/1902.05978): single-photo 3D face fitting with a GAN texture prior and face-recognition identity and content losses, evaluated on MICC. Reconstruction code is not released. - [Unlearnable Examples: Making Personal Data Unexploitable](/papers/2101.04898): error-minimizing noise that makes images hard to learn from, with a WebFace case study and partial-coverage and adversarial-training results. - [YOLO5Face: Why Reinventing a Face Detector](/papers/2105.12931): YOLOv5-based face detector with a five-point landmark head, from YOLOv5n to YOLOv5x6, with WiderFace results and a recognition-pipeline FNMR check.