WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition

In plain words

Face recognition models learn from large collections of labelled face photos, and the biggest public collections are small compared with what large companies hold. This paper builds a much larger one. It starts from about four million celebrity names, searches the web for photos of each name, and cuts out the faces. Web photos are messy, so the authors also designed an automatic cleaning method. A model cleans the data, a new model is trained on the cleaner data, and that new model then does the cleaning again. After three rounds, about 42 million faces of 2 million people remain.

The paper also proposes a way to test face recognition under a time limit, since a real device has only a fixed time per check. Models are scored on a new, hand-built test set in three speed tracks: 100, 500 and 1000 milliseconds. The authors report that training on their cleaned set beats the public sets they compared with, and that the cleaned set gives an entry that ranks third among 430 in a NIST face-recognition benchmark.

For a practitioner, the lesson is that the training data and its cleaning matter as much as the model, and that a time limit can change which model is the right choice.

TL;DR

• Dataset: WebFace260M has 4M identities and 260M faces; the cleaned WebFace42M has 2M identities and 42M faces §2 tab. 1 tab. 2.
• CAST cleans the web data with three self-training rounds. Each round uses an ArcFace ResNet-100 teacher, starting from one trained on MS1MV2 §3 §5.4.
• ArcFace ResNet-100 trained on WebFace42M reaches 97.70% TAR@FAR=1e-4 on IJB-C, against 96.03% for MS1MV2 (author-reported) tab. 5.
• FRUITS measures accuracy under 100, 500 and 1000 ms budgets on a new test set of 2,225 identities §4 tab. 3.
• Code: none found. Data access is by signed licence, for research use §6.

Problem

Public face training sets lag behind those available inside companies. Among public sets, MegaFace2 has the most identities and MS1M the most faces, and the paper puts MS1M's noise rate at around 50% §2 tab. 1. Hand or semi-automatic cleaning does not scale to millions of identities, and the purely automatic MegaFace2 cleaning is reported to leave more than 30% noise §3. Clustering the whole web set at once is impractical §3.

Public benchmarks are also close to saturated, and the time-limited NIST-FRVT test is restricted by a submission policy of one submission every four calendar months §1 §4. The paper argues that real deployments are limited by inference time, which those benchmarks do not measure §1 §4.

Method

The paper has four parts: (1) how the web data was collected and pre-processed §2; (2) CAST, the cleaning pipeline §3; (3) the FRUITS protocol and test set §4; and (4) training and evaluation experiments with a distributed training setup §5. No numbered equations appear in the paper. The only formulas are prose definitions of cosine thresholds §3 and of the FNMR and FMR metrics §4.3.

Data collection and pre-processing §2

1. Names: 1M names come from the MS1M list, which was built from Freebase, and 3M are chosen from the IMDB database out of about 4M names there §2.
2. Images: a commercial image search engine is queried per name. The top 10% of subjects get 200 images each, and the next 20%, 30% and 40% get 100, 50 and 25 §2. The ranking used to form these tiers is not stated.
3. Faces: RetinaFace predicts five landmarks for each image. If an image has several faces, only the largest one above a score threshold is kept. The threshold value is not stated §2.
4. Result: 4,008,130 identities and 260,890,076 faces after pre-processing tab. 2. That is 65 images per identity on average tab. 1.
5. Attributes: WebFace42M gets seven attribute labels (pose, age, race, gender, hat, glasses, mask). The paper shows their distributions but not how they were labelled §2 fig. 3.

CAST: Cleaning Automatically by Self-Training §3 fig. 4

CAST treats cleaning as self-training on open-set face data. Each round uses a teacher network to embed every face. The faces in each folder are then clustered to find the dominant person, and folders are merged or dropped by comparing their centres. The next round trains a new student on the cleaned data, and that student becomes the next teacher §3. Three rounds are used §5.4.

# Cleaning Automatically by Self-Training (CAST)  [§3] [fig. 4]
# Input:  faces: per name, a folder of pre-processed face crops
# Output: cleaned folders, after duplicate and overlap removal

def cast(faces, ms1mv2_teacher, num_rounds):
    # The first teacher is ArcFace ResNet-100 trained on MS1MV2  [§3]
    teacher = ms1mv2_teacher
    # Three rounds are used; the fourth gave saturated results  [§5.4]
    for round_index in range(num_rounds):
        # Each round cleans the same initial set with the current teacher
        # [§3] [fig. 4 caption]
        cleaned = clean_folders(faces, teacher)
        # Train a student on the cleaned faces; it becomes the next teacher
        # [§3]
        teacher = train_arcface_resnet100(cleaned)
    # Remove duplicates inside each subject, and identities that overlap
    # benchmarks or the FRUITS test set  [§3]
    return remove_duplicates_and_overlaps(cleaned)

The intra-class and inter-class steps that each round uses are as follows.

# Intra-class cleaning, then inter-class cleaning  [§3]
# Input:  faces per name, and a teacher that outputs 512-d features
# Output: one kept set of faces per remaining folder

def clean_folders(faces, teacher):
    # Embed every face with the teacher  [§3]
    embeddings = teacher.embed(faces)                  # (N, 512)
    kept = {}
    for name, folder in faces.items():
        # Cluster the faces of one folder with DBSCAN  [§3]
        # DBSCAN's eps and min_samples are not stated
        clusters = dbscan(embeddings[folder])
        # Keep the largest cluster only, if it has more than 2 faces  [§3]
        kept[name] = largest_cluster(clusters, min_size=3)
    # Compare the feature centre of each folder  [§3]
    # How the centre is formed is not stated
    centres = {name: mean(embeddings[kept[name]]) for name in kept}
    # Merge two folders with cosine above 0.7  [§3]
    # Delete the folder with fewer faces when cosine is in 0.5 to 0.7  [§3]
    # Behaviour below 0.5 is not stated
    return merge_and_drop(kept, centres, merge_at=0.7, drop_from=0.5)

Duplicate and overlap removal, per §3:

• Within a subject, faces with cosine above 0.95 to another face are duplicates and removed. Which copy is kept is not stated.
• Each subject's feature centre is compared with LFW, CALFW, CPLFW, FaceScrub, IJB-C and the FRUITS test set. Overlapping identities with cosine above 0.7 are removed.

The paper reports identity and face counts after each stage tab. 2. The inference that every round cleans the same pre-processed set, rather than the output of the round before, comes from the text, which says the intra- and inter-class passes both start from the initial set and use a different teacher each time, and from the table's counts §3 tab. 2.

FRUITS protocol and test set §4

FRUITS (Face Recognition Under Inference Time conStraint) times the whole system: pre-processing (detection and alignment), feature extraction, and matching. It has three tracks §4.1:

• FRUITS-100: 100 ms per image pair, for lightweight systems on mobile devices.
• FRUITS-500: 500 ms, for models deployed on local surveillance systems.
• FRUITS-1000: 1000 ms, following NIST-FRVT, for larger cloud models.

Timing uses a single core of an Intel Xeon E5-2630-v4 at 2.20 GHz, following NIST-FRVT §4.1.

The test set has 2,225 identities and 38,578 faces. Annotators picked only celebrities they knew, to avoid mistakes between look-alikes, and were asked to balance attributes. Recognition models were used to guide hard-sample selection (the models are not named) §4.2. Attributes are age, race, gender, and controlled or wild capture §4.2 tab. 3.

Metric: 1:1 verification. "All" pairs the impostors without regard to attributes; the attribute subsets pair within them, and cross-age and cross-scene comparisons are also defined §4.3. The metric is FNMR at a fixed FMR, with FMR set to 1e-5 in the reported results §4.3 tab. 5 tab. 9. The FNMR is the share of genuine pairs below the threshold. FMR is the share of impostor pairs at or above the threshold §4.3.

# FNMR at a fixed FMR on verification pairs  [§4.3]
# Input:  genuine_scores, impostor_scores (match scores), target_fmr = 1e-5
# Output: FNMR at the threshold that gives the target FMR

def fnmr_at_fmr(genuine_scores, impostor_scores, target_fmr):
    # Choose the threshold so that the share of impostor pairs at or above
    # it equals the target FMR  [§4.3]
    threshold = impostor_quantile(impostor_scores, 1 - target_fmr)
    # FNMR is the share of genuine pairs below the threshold  [§4.3]
    return mean(genuine_scores < threshold)

Distributed training §5.2 fig. 6

Two ideas are used. The feature matrix X and the class-centre matrix W are both parallelised across GPUs, and the models are trained in FP16 mixed precision with large batches §5.2. The paper does not describe the communication pattern or how W is split. Learning rate is 0.05 for one node of 8 GPUs and scales with the number of machines §5.1. The paper names the linear scaling rule for this and gives no warm-up schedule §5.1.

# Learning rate for multi-machine training  [§5.1]
# Input:  num_machines, with 8 GPUs per machine
# Output: base learning rate for the run

def base_learning_rate(num_machines):
    # 0.05 for one node of 8 GPUs, scaled linearly by machine count
    # [§5.1]; warm-up is not stated
    return 0.05 * num_machines

Training schedule for the compared losses §5.1

# Step schedule used for all compared models  [§5.1]
# Input:  epoch index (0-based), base learning rate
# Output: learning rate for this epoch

def learning_rate_at(epoch, base_lr):
    # Multiply by 0.1 at epochs 8, 12 and 16; training stops at epoch 20
    # [§5.1]. Whether the decay applies from or after each epoch is not stated.
    decays = sum(epoch >= milestone for milestone in (8, 12, 16))
    return base_lr * 0.1 ** decays

Hyperparameters

Results

Author-reported. Setup: ResNet-100 with ArcFace, CosFace or CurricularFace on each training set, scored on several verification sets tab. 5; distributed training speed on WebFace42M tab. 4; FRUITS baselines tab. 9 fig. 8; NIST-FRVT tab. 10.

Headline numbers:

The paper says the IJB-C error falls by about 40% relative §1 §5.3. Taking the table values, (100 - 96.03) to (100 - 97.70) is a drop of about 42% tab. 5.

Dataset statistics tab. 1 tab. 2

The paper's Table 1 compares training sets. Rows marked private are left out here.

Note: the paper's Table 1 marks WebFace260M and WebFace42M as "Public" in availability. Section 6 says access is granted on application under a licence for research use §6. See Pitfalls.

CAST stages tab. 2:

FRUITS test set tab. 3

Distributed training tab. 4 fig. 6

B x G x M is batch size per GPU x GPUs per machine x machines. Memory is in MB, in brackets. Accuracy is IJB-C TAR@FAR=1e-4. Speed is in samples per second, as the text gives it §5.2.

The paper says scaling efficiency is above 80% on 256 GPUs (32 nodes) §5.2. From the 100% rows, 233 h on one machine and 9 h on 32 machines is about 81% of linear scaling.

Training-data comparison tab. 5 fig. 7

ResNet-100 without flip test. Columns are: Pairs (average over LFW, CFP-FP, CPLFW, AgeDB, CALFW), RFW (average), Mega (MegaFace rank-1 identification), IJB-C (TAR@FAR=1e-4), and Our test (FNMR@FMR=1e-5 on All pairs, lower is better).

Small-model check, ResNet-14 with ArcFace, IJB-C TAR@FAR=1e-4 tab. 6:

Cleaning comparison tab. 7 tab. 8

ResNet-100 with ArcFace for all rows. The MS1M cleaning rows use a teacher trained on IMDB-Face §5.4.

Intra-class cleaning on MS1M, ResNet-100 with ArcFace tab. 8:

FRUITS baselines tab. 9 fig. 8

Table 9 gives model settings and time. Accuracy for these baselines appears only in Figure 8, as plots. The paper's text states one value (RegNet-800MF, 5.88%) §5.5.

The table caption says M-0.25 is RetinaFace MobileNet-0.25 (23 ms) and R-50 is RetinaFace ResNet-50 (272 ms). Time is for the whole system tab. 9.

Text-reported FRUITS results §5.5:

• FRUITS-100: the best baseline is RegNet-800MF, with 5.88% FNMR@FMR=1e-5. Lower is better, and the paper says there is much room for improvement.
• FRUITS-500: ResNet-100 performs best overall. ResNet-50 with flip testing is best on the Wild and Male subsets. ResNeXt is first on Cross-scene.
• FRUITS-1000: ResNet-200 is best overall and wins five attribute comparisons. SENet-152 and AttentionNet-152 win three and two. The paper says the large models are close together in this track.

NIST-FRVT tab. 10 §5.6

For the NIST submission, the detector is a ResNet-50 RetinaFace model. The embedding network is a 200-layer ResNet with the ArcFace loss, trained on WebFace42M. OpenVINO is used for acceleration, with a flip test. The paper reports about 1300 ms against a 1500 ms limit §5.6.

The values are FNMR at the corresponding FMR, as reported in the paper tab. 10.

Paper vs code

No official code.

What was searched:

1. Links in the paper: the abstract gives the project page https://www.face-benchmark.org and no code repository. Section 6 says access to the data is granted to applicants who sign a licence §6.
2. The project page: WebFetch failed on a TLS certificate mismatch (the certificate names dynadot.com), so its content could not be read.
3. GitHub search: "WebFace42M" and "CAST face cleaning self-training" return no repositories. "WebFace260M" returns three repositories, none verifiably owned by the authors. WebFace260M/webface260m-iccv21-mfr is a submission package for an ICCV 2021 masked-face challenge. It ships inference and submission code only, and no training code. It has no licence. The owner account is not verifiable: the organisation endpoint returned 404. biometric-community/WebFace260M is a catalogue of data links, and ducanhho2296/MaskedFaceRecognition is a third-party challenge repository. None of these is used.

Pitfalls

• Data access and licence. The dataset is not included here. Section 6 says access is granted to licence signatories for research purposes only §6. Table 1 labels both WebFace datasets "Public" tab. 1, which conflicts with that text. Do not assume the data is usable commercially, and do not redistribute it.
• No code, so cleaning is not reproducible from the paper alone. Missing: DBSCAN eps and min_samples §3; how a folder's centre is formed §3; what happens below cosine 0.5 §3; which duplicate is kept §3; the face score threshold §2; the ranking that sets the 200/100/50/25 image tiers §2; how the seven attributes were labelled §2; and the MS1MV2 teacher's training schedule (only the compared-loss schedule is given in §5.1).
• The 10% claim is not uniform. The paper says WebFace4M beats MS1M families and MegaFace2. On IJB-C, WebFace4M ArcFace (96.77) is above MS1MV2 ArcFace (96.03). On MegaFace rank-1, it is below: 97.60 against 98.40 for MS1MV2 and 98.48 for MS1M-Glint tab. 5. Curricular on WebFace4M is 97.94, below MS1MV2 Curricular at 98.46 tab. 5.
• The headline IJB-C number is ArcFace, not the best loss. WebFace42M ArcFace gives 97.70. CurricularFace gives 97.76 on the same data, and CosFace gives 97.68 tab. 5.
• Protocols differ between tables. Table 5 is ResNet-100 without flip test. The NIST entry uses ResNet-200 with flip test tab. 5 §5.6. The FRUITS baselines add detection and alignment and are timed as whole systems tab. 9. Numbers across these tables are not directly comparable.
• The CAST teacher for MS1M is different. For MS1M cleaning, the initial teacher is trained on IMDB-Face, not MS1MV2 §5.4. The WebFace cleaning uses the MS1MV2 teacher §3.
• Dataset construction is not random. Annotators chose celebrities they already knew, and recognition models guided hard-sample selection. The models are not named §4.2. The test set therefore does not measure typical error rates. The paper also says it will be maintained and updated §4.2, but gives no version or date.
• FNMR at FMR = 1e-5 needs very many impostor pairs. The All row has 743,683,994 impostor pairs tab. 3. The paper does not say whether the threshold is found exactly or estimated. Confidence intervals are not given.
• Timing depends on hardware. The timing setup is a single Xeon core §4.1. Times in tab. 9 are whole-system times, and the caption does not restate the hardware. Re-timing elsewhere will not match.
• Fairness is shown only in plots. The attribute results are in Figure 8 and the discussion says bias may exist §6 fig. 8. No numeric bias metric is given in the text.
• Batch size is inconsistent. The default is 64 per GPU §5.1, but the 100%-data rows of the distributed table use 32 tab. 4. The paper does not say which batch size produced the dataset comparison.
• Learning rate and warm-up. The rate scales with machines §5.1. No warm-up is stated for the distributed runs §5.2.
• The overall NIST rank method is not stated. Rank 3 of 430 is given, but the paper does not say how five track values combine into one rank tab. 10. Among these five entries, it is best only on Mugshot (0.0028).
• Loss margins and scales are not stated. The paper says the compared losses follow their original papers §5.1. Those values belong to the cited papers, so an implementer must take them from there.

Lineage

• Builds on: ArcFace, the loss used for the teacher and student §3 §5.1 (/papers/1801.07698); CosFace and CurricularFace, also compared §5.1 tab. 5.
• Compares its datasets against: VGGFace2 tab. 1 (/papers/1710.08092), MS1M, MS1M-IBUG, MS1MV2, MS1M-Glint, MegaFace2 and IMDB-Face tab. 1 tab. 5.
• Cleaning methods compared: DBSCAN, K-Means, GCN-D and GCN-V tab. 8; a GCN cleaner from MillionCelebs is compared in tab. 7.
• Self-training is the base for CAST; the paper cites several self-training works for image classification §3.
• Superseded by: not stated.