YOLO5Face: Why Reinventing a Face Detector

In plain words

Most face detectors are built around faces: special shapes, special tricks. This paper asks whether a general object detector, one built to find cars, dogs and chairs, can find faces just as well after some changes. The authors start from YOLOv5, a fast and widely used detector, and adapt it to faces.

The changes are small in number. The detector also predicts a handful of key points on each face, which helps later steps such as lining faces up for recognition. A few internal layers are swapped or resized so that very large faces are not missed and small faces are still seen. One set of models is built on a lightweight backbone meant for phones and other small devices. The authors release several model sizes, so a user can trade accuracy for speed.

TL;DR

• YOLO5Face adapts the YOLOv5 object detector to faces and treats face detection as a general object detection task §1.
• Main changes: a five-point landmark head with wing loss, a stem block, smaller SPP kernels, a P6 output for large faces, and ShuffleNetV2 backbones for the smallest models §3.2.
• Author-reported WiderFace validation results put the largest model (YOLOv5x6) ahead of the strongest compared detector in the table (SCRFD-34GF) on all three subsets. The smallest model (YOLOv5n) is ahead of SCRFD-2.5GF on Hard only §4.5 tab. 3.
• Use it if you need a single detector family from tiny mobile to large models; check the Pitfalls section first, since the paper and its code disagree on several training values.

Problem

Face detectors have usually been designed for faces, with anchor, scale and context choices made for small faces §2. The authors argue that large faces are easy to overlook and that a single detector family should cover a range of accuracy and speed targets, from high accuracy on servers to real-time use on embedded or mobile hardware §1 §2 §3.6. The paper assumes VGA-sized inputs: the longer image edge is scaled to 640 pixels §3.1.

Method

The detector is a YOLOv5-style network §3.1 fig. 1. It has a backbone that extracts features, a neck that combines them across scales, and a head that predicts boxes, confidence, class and landmarks. The backbone is CSPNet-style for the CSP models and ShuffleNetV2 for the two smallest models §3.1 tab. 1. The neck uses SPP followed by a PAN path §3.1 §3.5 §3.6.

Each anchor in the head outputs 16 values: 4 box values, 1 confidence, 1 class score and 10 landmark values (x and y for five points) §3.1 fig. 1. In the code the landmark count is added to the usual YOLOv5 output size [code: models/yolo.py#L35]. Landmark outputs are decoded like the box offsets, with the anchor size and grid position applied yolo.py. During training, landmark labels are normalised to the 0-1 range by the image width and height face_datasets.py.

Changes relative to YOLOv5, as listed in the paper §3.2:

1. Landmark head. A five-point landmark regression output trained with the wing loss, added to the detection loss §3.3 eq. 1 eq. 2 eq. 3.
2. Stem block. The first convolution, which YOLOv5 implements as a Focus layer, is replaced by a stem block with a stride-2 first down-sampling and more channels §3.2 §3.4. The paper says the block is similar to a published design it cites §3.4. The code's block has a stride-2 3x3 convolution, a branch of a 1x1 convolution and a second stride-2 3x3 convolution, and a parallel max-pool branch; the two are concatenated and fused with a 1x1 convolution common.py.
3. SPP with smaller kernels. The SPP kernels become 7x7, 5x5 and 3x3 instead of 13x13, 9x9 and 5x5 §3.5. The code uses the same set, listed as [3, 5, 7] yolov5s.yaml.
4. P6 output. An extra detection output at stride 64, with a 10x10x16 map for a 640-pixel input, meant for large faces §3.6.
5. Augmentation. Up-down flipping is removed, and random cropping is said to help §3.2 §4.3. See Pitfalls for what the code does here.
6. ShuffleNetV2 backbones. Two very small models use ShuffleNetV2 and a ShuffleNetV2 with half width §3.7 tab. 1.

The paper's model family is listed in tab. 1. The code has configurations for the same sizes and uses the depth and width multiples below [code: models/yolov5s.yaml#L3-L4] yolov5s6.yaml [code: models/yolov5n.yaml#L3-L4] yolov5n-0.5.yaml.

Landmark loss. Each landmark coordinate is scored with the wing loss, which is logarithmic for small errors and linear for large ones. Its curvature is set by a range parameter w and a curvature parameter e. The total landmark loss is a sum over the ten coordinates §3.3 eq. 1 eq. 2. The total training loss is the YOLOv5 loss plus lambda_L times the landmark loss eq. 3.

# Wing loss for one coordinate error  [§3.3] [eq. 1]
# Input:  error (scalar), w (range), e (curvature)
# Output: scalar loss
def wing(error, w, e):
    # C joins the logarithmic and linear parts [eq. 1]
    C = w - w * log(1 + w / e)
    # Logarithmic branch for small errors, linear branch otherwise
    # The paper writes x < w; the code tests |x| < w
    # [eq. 1] [code: utils/loss.py#L95-L102]
    if abs(error) < w:
        return w * log(1 + abs(error) / e)
    return abs(error) - C

# Landmark loss for one prediction  [§3.3] [eq. 2]
# Input:  predicted, target: (10,) x/y values
#         mask: (10,) 1 where the landmark is labelled, else 0
# Output: scalar landmark loss
def landmark_loss(predicted, target, mask):
    # Sum the wing loss over the ten coordinates [eq. 2]
    total = 0
    for i in range(10):
        total += mask[i] * wing(predicted[i] - target[i], w, e)
    # The code divides by the number of labelled coordinates; eq. 2 does not
    # [code: utils/loss.py#L113]
    return total / (sum(mask) + eps)

# Total training loss  [eq. 3]
# Input:  detection_loss (YOLOv5 box, objectness and class terms)
#         landmark (output of landmark_loss)
# Output: scalar training loss
def total_loss(detection_loss, landmark, lambda_l):
    # lambda_L = 0.5 in the paper, found by exhaustive search [§4.2]
    # The code uses a gain of 0.005 scaled by 3 / (number of detection
    # layers) instead [code: utils/loss.py#L138] [code: utils/loss.py#L184-L188]
    return detection_loss + lambda_l * landmark

Test-time input size, VGA setting. The longer edge is scaled to 640 pixels and the shorter edge follows the aspect ratio. Its size is then rounded to a multiple of the largest stride: 32 without P6, 64 with P6 §3.1. The test script uses the same letterbox routine as training, which pads to multiples of 64 in auto mode test_widerface.py datasets.py.

# Resize for the VGA setting  [§3.1]
# Input:  image (H, W, 3), longest_edge (640)
# Output: padded image (H', W', 3), scale factor
def prepare_test_image(image, longest_edge=640):
    # Longer edge becomes 640; the shorter edge keeps the aspect ratio [§3.1]
    scale = longest_edge / max(image.height, image.width)
    resized = resize(image, scale)
    # Pad both sides to a multiple of the stride, 64 in auto mode
    # [§3.1] [code: utils/datasets.py#L795-L796]
    padded = pad_to_multiple(resized, 64)
    return padded, scale

Multi-scale test. With the test flag, the network runs at scales 1, 0.83 and 0.67, and a left-right flipped pass is included §4.5 yolo.py. The test script leaves this off by default test_widerface.py.

Hyperparameters

Values from the paper §4.2 and the code's defaults. Where the paper is silent, the entry says so.

Results

Author-reported. Setup: WiderFace validation set, Easy, Medium and Hard subsets §4.1 tab. 3. The paper says the focus is VGA inputs §4.5. Whether the table numbers use the multi-scale and flip test is not stated §4.5.

Main comparison, WiderFace validation tab. 3. Params and FLOPs as reported by the authors.

Ablation on YOLOv5s, WiderFace validation tab. 2 §4.3. The paper says each panel changes one thing against its own baseline.

Face recognition, WebFace test set tab. 4 §4.4. Lower FNMR is better. The metric combines masked and standard face recognition §4.4. The recogniser is an ArcFace setup, with the face detector swapped §4.4.

FDDB, cross-domain tab. 5 §4.6. True positive rate at 1000 false positives. The paper uses the original FDDB annotation without modification, and notes that the RefineFace result depends on a modified annotation §4.6.

Paper vs code

Official code: https://github.com/deepcam-cn/yolov5-face at commit 152c688d551aefb973b7b589fb0691c93dab3564, under GPL-3.0. The paper's abstract links this repository, and its owner name matches the authors' affiliation on the title page (Shenzhen Deepcam) §abstract. No code was quoted here, only paths and line numbers.

Discrepancies found:

• Final learning rate. Paper: 1e-5 §4.2. Code: cosine decay to lr0 x lrf = 2e-3 hyp.scratch.yaml train.py.
• Weight decay. Paper: 5e-3 §4.2. Code: 5e-4 hyp.scratch.yaml.
• Batch size. Paper: 64 §4.2. Code default: 16 train.py. The README does not pass a batch size README.md.
• Landmark weight. Paper: lambda_L = 0.5 §4.2. Code: gain 0.005, scaled by 3 over the number of detection layers hyp.scratch.yaml loss.py loss.py.
• Landmark loss normalisation. Paper: plain sum eq. 2. Code: divided by the number of labelled coordinates loss.py.
• Missing landmarks. Not described in the paper. The code gives coordinates with target -1 zero weight loss.py.
• Wing loss branch test. Paper text places the log branch on x < w, yet describes the nonlinear part as (-w, w). Code tests the absolute error [code: utils/loss.py#L95-L102].
• Random crop. Paper: random cropping helps §4.3. Code: no crop operation found in the training pipeline. A replicate-small-boxes routine exists but is commented out face_datasets.py face_datasets.py. The nearest active augmentation is random translate and scale, at 0.1 and 0.5 hyp.scratch.yaml.
• Ignoring small faces. Paper: needed for Mosaic to work §4.3. Code: the only ignore parameter found is in the validation conversion script, default 0 val2yolo.py. No training-time filter was found.
• Model list. Paper text: eight large models, including YOLOv5x and YOLOv5x6 §4.2. Table 1 lists six CSP models and two ShuffleNet models tab. 1. The repository has no yolov5x or yolov5x6 configuration at this commit.
• Pretrained initialisation. Not stated in the paper. The code defaults to a yolov5s pretrained file train.py.
• Test-time augmentation. Consistent: scales 1, 0.83 and 0.67, plus a left-right flip, behind the test flag §4.5 yolo.py.

No discrepancies were checked for hardware or training time, which the paper does not state.

Pitfalls

• Missing wing-loss plot. The text says the wing-loss curves are plotted in "Fig. 2". Figure 2 in the source is the precision-recall figure. The stem-block figure is commented out in the source §3.3 fig. 2.
• Results text versus tables. The text gives validation numbers of 96.9, 96.0 and 91.6 (Easy, Medium, Hard) for YOLOv5x6 and test numbers of 95.8, 94.9 and 90.5 §4.5. Neither set appears in any table. The validation row in tab. 3 gives 96.67, 95.08 and 86.55 for the same model. Implementers should use the table values and treat the text values as unverified.
• SPP Hard gain. The text says the smaller SPP kernels gain 1.41 points on Hard §4.3. The table values (84.15 and 82.64) give 1.51 tab. 2.
• Ablation versus main table. The ablation's YOLOv5s baseline (94.31, 92.52, 83.15) differs slightly from the main table's YOLOv5s row (94.33, 92.61, 83.15) tab. 2 tab. 3. The paper does not explain this.
• Table numbering. The FDDB text says "Table 4", while the FDDB table is Table 5 in the compiled order §4.6 tab. 5.
• Data augmentation table. Rows are labelled with minus and plus signs, which reads as cumulative, yet the text says the configurations are not incremental §4.3 tab. 2.
• Training input size. Not stated in the paper. The default in the code is 800. Choosing 640 for training is a guess and changes the scale distribution.
• Input shape for P6. The paper requires the shorter edge to be a multiple of 64 when P6 is used §3.1. The test letterbox pads to multiples of 64 [code: utils/datasets.py#L795-L796]. A custom pipeline that skips this padding may produce feature maps that do not line up in the neck. I did not run the code.
• Stem block reference. The paper cites a published design for the stem block §3.4. The bibliography entry is not checked here. The code's block is described above from its source common.py.
• SPP reference. The bibliography entry keyed as SPP carries the title of a different paper, on sigmoid-weighted linear units §3.5. Cite SPP from the paper's own text, not from the bibliography.
• Face recognition backbone. The paper names a 124-layer ResNet for the recognition backbone §4.4. That depth is unusual, so the name may be a typo. Unverified.
• Landmark usage. The paper shows landmarks only in face recognition, to align faces before recognition §3.3 §4.4. Its landmark quality claims are supported by a recognition metric and a qualitative figure fig. 3.
• Hidden costs. The P6 model adds parameters and FLOPs (12.386M and 6.28G for YOLOv5s6, against 7.075M and 5.751G for YOLOv5s) tab. 2 tab. 3. Larger models use 141M parameters at the top end tab. 3.
• Speed. The paper gives FLOPs and parameters, not measured latency on any hardware. Its real-time claim for embedded and mobile use has no latency table behind it §1 §5.
• Licence. The repository is GPL-3.0. Any verbatim code copied from it would carry copyleft obligations. This card uses paths, line numbers and pseudocode only.

Lineage

• Builds on: YOLOv5, released as code without a paper, and used as the baseline §3.1. YOLOv4 for the Mosaic augmentation §3.2 (arXiv:2004.10934). PAN, for the bottom-up path in the neck §3.6 (arXiv:1803.01534). ShuffleNetV2 for the two small backbones §3.7 (arXiv:1807.11164). The wing loss §3.3 (arXiv:1711.06753, the id given in the code's comment loss.py).
• Compared against: SCRFD (arXiv:2105.04714), TinaFace (arXiv:2011.13183), RetinaFace, DSFD, HAMBox, FaceBoxes and others, in the WiderFace comparison tab. 3.
• Superseded by: not stated in this wiki.