VGGFace2: A dataset for recognising faces across pose and age

In plain words

A face recognition model learns by looking at many photos of many people. Two things matter: how many different people it sees (breadth) and how differently each person is shown (depth): side views, childhood photos, bad lighting. Earlier public collections tended to favour one or the other, and the large ones scraped from search engines contained many photos of the wrong person.

VGGFace2 is a collection of about 3.3 million photos of about 9,000 public figures, gathered on purpose to include profiles and younger pictures of each person. The authors downloaded images from Google Image Search, also searching for each name plus words like "sideview" and "very young", then cleaned the result in stages: a face detector, automatic classifiers that throw out photos unlikely to show the right person, removal of near-identical copies, and finally human annotators who check the doubtful cases.

They also added a small test protocol: groups of five photos of the same person seen from the front, at three-quarter view or in profile, and groups of young versus mature photos, so one can measure how well a model matches the same person across pose and age. Standard image classifiers (ResNet-50 and a variant with squeeze-and-excitation blocks) trained on VGGFace2 matched faces across pose and age better than the same networks trained on older datasets, and did best on the IJB-A, IJB-B and IJB-C benchmarks when first trained on the much larger MS-Celeb-1M and then refined on VGGFace2.

TL;DR

VGGFace2 is a dataset paper: 3.31M images, 9,131 identities, mean 362.6 images per identity, split into 8,631 training and 500 test identities, with face boxes, five landmarks, and predicted pose and apparent age per image §3.1, tab. 1. The contribution is the collection pipeline (Freebase name list, 1,400 Google images per name, MTCNN detection, classifier-score filtering, VLAD near-duplicate removal, score-banded manual cleaning) and pose/age template protocols on the test set §3.2, §4. Baselines are plain softmax-trained ResNet-50 / SE-ResNet-50 with 2048-d descriptors, template averaging and cosine similarity §5.1. MS1M pre-training followed by VGGFace2 fine-tuning with SENet gives the best IJB-A/B/C numbers in the paper tab. 6, tab. 7, tab. 8. Read it if you need to rebuild a similar web-scraped identity dataset or reproduce the VGGFace2 baselines; there is no new loss or architecture.

Problem

Public face datasets before this paper emphasise either many images per identity (depth) or many identities with few images each (breadth), and none is designed around pose and age variation §1. MS-Celeb-1M has the most images but limited intra-identity variation (the text gives an average of 81 images per person) and label noise because its images were taken from a search engine without manual filtering §2. VGGFace has 2.6M images but only 2,622 identities §2, tab. 1. The goals for VGGFace2 are many identities and many images per identity, wide pose, age and ethnicity coverage, and minimal label noise [abstract].

Method

The method is a six-stage collection pipeline §4, tab. 2, plus annotation and baseline training.

Algorithm: VGGFace2 collection pipeline  [§4, tab. 2]
Input: Freebase knowledge graph, Google Image Search, annotators
1. Name list [§4.1] (manual, ~3 months):
   - take 500K public-figure names from Freebase
   - per name, download 100 Google images; annotators keep a name only if
     roughly 90% or more of the 100 images show a single person
   - result: 9,244 names; ethnicity/kinship attributes taken from DBPedia
2. Image download [§4.2]:
   - per name: 1,000 images for the plain name query,
     200 for "<name> sideview", 200 for "<name> very young"  -> 1,400 per name
   - total 12.94M images
3. Face detection [§4.3]:
   - MTCNN (Zhang et al.) with the hyperparameters that work recommends
   - enlarge each face box by a factor of 0.3 to cover the whole head
   - keep the 5 landmarks predicted by the same model
   - total 7.31M faces
4. Classifier filtering [§4.4]:
   - features: VGGFace model descriptors
   - train 1-vs-rest classifiers over the 9,244 identities:
       positives = faces from that identity's top-100 retrieved images
       negatives = faces from the top-100 images of all other identities
   - score every face in [0, 1]; drop faces with score < 0.5
     (threshold picked by manually inspecting 500 random identities)
   - total 6.99M
5. Near-duplicate removal [§4.5]:
   - compute VLAD descriptors for all remaining images, cluster them,
     keep one image per cluster
   - total 5.45M
6. Final filtering [§4.6] (automatic + manual, 21 days):
   a. Overlapping identities: per class, split images 50/50; train ResNet-50
      on one half, build a top-1 confusion matrix on the other half.
      20 confused subjects found; 19 noisy classes removed.
   b. Remove 94 identities with fewer than 80 images -> 9,131 identities.
   c. Outlier images (target purity > 96%):
      - retrain the classifier on the current data; per identity, bin images by score:
          H = [1, 0.95], I = (0.95, 0.8], L = (0.8, 0.5]
      - if H is noisy (several people): annotators clean H, I and L
        else: annotators clean only L
      - train a new model on cleaned H and L; clean I using that model's predictions
   - total 3.31M images
Output: 9,131 identities, 3.31M images

Per-stage counts as reported tab. 2:

(M = manual, A = automatic.)

Pose and age labels §4.7. A 5-way head-pose classifier (ResNet-50) is trained on CASIA-WebFace and an 8-way apparent-age classifier (ResNet-50) on IMDB-WIKI-500k+; both are run over the whole dataset. The bin edges of the 5 pose and 8 age classes are not stated in the paper.

Dataset contents §3.1. 59.3% male; 80 to 843 images per identity (mean 362.6); human-verified face boxes; five landmarks from MTCNN; predicted yaw/pitch/roll and apparent age. Split: 8,631 training classes, 500 test classes §3.1. fig. 1 shows histograms of the pose and age labels plus sample images of eight subjects of different ethnicities.

Test-set templates §3.2. A template is 5 images of one subject.

• Pose: 300 test subjects, two templates each for frontal, three-quarter and profile view, so 1.8K templates / 9K images.
• Age: 100 test subjects, two templates each for "young" (apparent age below 34) and "mature" (34 or above), so 400 templates / 2K images.

Baseline training §5.1. Backbones are ResNet-50 and SE-ResNet-50 ("SENet"), trained with softmax loss in four settings: from scratch on VGGFace (VF), on MS-Celeb-1M (MS1M), on VGGFace2 (VF2), and MS1M pre-training then VGGFace2 fine-tuning (VF2_ft). Training subjects do not overlap the benchmarks, and the MS1M overlap with "the two benchmarks" was removed §5.

Training input pipeline  [§5.1]
1. take the enlarged face box; resize so the shorter side is 256 px
2. random 224x224 crop
3. subtract the per-channel mean
4. with probability 0.2 convert to monochrome
5. SGD, batch 256, class-balanced sampling per mini-batch

Descriptor and template matching §5.1.

Descriptor: resize enlarged box to shorter side 256, take the centre 224x224 crop,
            read the 2048-d layer just before the classifier, L2-normalise.
VGGFace2 test identification: template vector = mean of image descriptors; 1-vs-rest SVMs.
IJB templates: media vector = mean of descriptors within one image or one video;
               template vector = mean of media vectors, then L2-normalise.
Similarity: cosine between template vectors.

Hyperparameters

Results

Author-reported. ResNet-50 trained from scratch on each dataset, evaluated on the VGGFace2 test set; lower is better tab. 3:

Pose probing: mean cosine similarity between pose templates over 300 subjects, ResNet-50 from scratch; higher is better tab. 4:

Age probing: mean cosine similarity between age templates over 100 subjects tab. 5:

Across both protocols VGGFace2-trained models give the highest similarities and MS1M-trained the lowest, and cross-pose / cross-age pairs score lower than same-pose / same-age pairs for every model §5.2. fig. 3 and fig. 5 show histograms of front-to-profile and young-to-mature scores for the three models.

IJB-A, 10 splits, mean ± std; TAR at FAR for 1:1 verification and TPIR at FPIR / Rank-N for 1:N identification tab. 6:

† read from Bansal et al. tab. 6.

IJB-B, 1:1 Mixed Media verification and 1:N Mixed Media identification over galleries S1 and S2 tab. 7:

IJB-C, galleries G1 and G2 tab. 8:

On IJB-A, ResNet-50 on VGGFace2 beats ResNet-50 on MS1M by about 4.4 points TAR at FAR=0.001, 3.7 points TPIR at FPIR=0.01 and 1.5 points Rank-1, although MS1M has roughly 10× the identities §5.3. SENet from scratch on VGGFace2 roughly matches the MS1M-pre-trained ResNet-50 §5.3. ROC, DET and CMC curves are in fig. 9, fig. 10 and fig. 11.

Paper vs code

No official code. The paper links only a dataset and model download page; no training or collection code is referenced [abstract].

Pitfalls

• This is a data pipeline, not a model. The baselines are ordinary softmax-trained ResNet-50 / SE-ResNet-50; any gain comes from the training data §5.1.
• "Union" in the abstract means sequential training. The abstract talks about training on the union of VGGFace2 and MS-Celeb-1M, but the method section describes MS1M pre-training followed by VGGFace2 fine-tuning (VF2_ft) [abstract, §5.1].
• Under-specified collection steps. The type of the stage-4 1-vs-rest classifier, the VLAD vocabulary and clustering threshold, how the 0.3 box enlargement is applied, the classifier retrained in stage 6, and the bin edges of the 5 pose and 8 age classes are all not stated §4.3–§4.7. Treat any reproduction of the filtering as approximate.
• Score bins have one-sided brackets. H is the closed range [1, 0.95], I is (0.95, 0.8], L is (0.8, 0.5]; scores of exactly 0.5 sit at the edge of L while stage 4 drops scores below 0.5 §4.4, §4.6.
• Training details missing. Momentum, weight decay, epoch count, the plateau criterion, the per-channel means and the exact class-balancing scheme are not stated; the fine-tuning LR is given only as starting at 0.005 and ending at 0.001 §5.1.
• Split image counts are not given. The paper states 8,631 training and 500 test classes but not how many images fall in each split §3.1.
• Ambiguous overlap removal. Training subjects are said to be disjoint from the benchmarks, but MS1M overlap is removed with "the two benchmarks" while three IJB benchmarks are evaluated §5. Which benchmarks were deduplicated against for IJB-C is not stated.
• Preprocessing must match. For IJB-A the authors re-detect faces with MTCNN so crops match training §5.3; the paper does not say whether the same was done for IJB-B and IJB-C. Your descriptor is computed on the enlarged box, so tight crops will shift results.
• Template aggregation is unweighted. Each image or video counts once at the media level, then media are averaged, regardless of how many frames a video has §5.1.
• Baseline comparisons are not like-for-like. Prior-work numbers for IJB-B and IJB-C are read off published curves, and those works averaged verification over galleries evaluated separately, a different protocol from the one used here tab. 7, tab. 8. Bansal et al. values on IJB-A are also partly read from curves tab. 6.
• Small inconsistencies in the text. Images per identity are given as 80–800 §1 and 80–843 §3.1, tab. 1; MegaFace has 690,572 identities in tab. 1 but 672,057 in §2; MS-Celeb-1M has 100 images per subject in tab. 1 but an average of 81 in §2. On IJB-C the bold Rank-10 entry (0.961, VF2_ft SENet) is below VF2 SENet's 0.962 tab. 8.
• IJB-C has no VGGFace / MS1M baseline rows, so cross-dataset comparisons there are only between VGGFace2 variants and Maze et al. tab. 8.

Lineage

• Builds on: VGGFace (Parkhi et al., 2015), whose name-list strategy stage 1 follows and whose model provides stage-4 features §4.1, §4.4; pre-training on MS-Celeb-1M §5.1.
• Architectures: ResNet-50 (arXiv:1512.03385) and Squeeze-and-Excitation networks (arXiv:1709.01507) §5.1.
• Compared against: Crosswhite et al., Sohn et al., Bansal et al., Yang et al. on IJB-A; Whitelam et al. on IJB-B; Maze et al. on IJB-C tab. 6, tab. 7, tab. 8.
• Related card: ArcFace also reports IJB-B and IJB-C results.