Sample and Computation Redistribution for Efficient Face Detection

In plain words

Face detectors look for faces in a photo and draw a box around each one. The strongest ones are large and slow. This paper argues that two things decide how well a small, fast detector can work: which face sizes the training data shows it, and where the network spends its computation.

The first idea, sample redistribution, cuts more training crops at more sizes, including zoomed-out ones, so that the small faces that dominate the benchmark show up more often during training. The second, computation redistribution, samples many small networks at one fixed compute budget, looks at which layouts score best, and narrows the search from there. The result, called SCRFD, is more accurate than a heavier detector at a fraction of the compute.

The accuracy numbers are the authors' own and come from one benchmark.

TL;DR

SCRFD is an anchor-based single-stage face detector trained from scratch, with a ResNet-style backbone, a PAFPN neck and a shared head. It has two changes: a wider set of training crop sizes (sample redistribution) and a two-step random search over how the backbone, neck and head share the FLOP budget (computation redistribution). It is tested at a fixed VGA-sized input. The authors report 85.29 hard AP for SCRFD-34GF against 81.43 for TinaFace, with latency more than 3x lower on a 2080 Ti tab. 4.

Problem

TinaFace, the strongest baseline the paper names, tests at a large input scale. It scores 93.4 AP with multi-scale testing but 81.4 at VGA size, and its cost at 640 is about 173 GFLOPs §1 §3 tab. 1. The paper argues that most faces in the benchmark are below 32 x 32 pixels at VGA scale, so most of them are predicted by the shallow stages §1 fig. 3.

TinaFace was also designed around a generic object detector (RetinaNet), with its backbone taken unchanged from an ImageNet classifier and dense anchors on every pyramid level, including stride 4. At VGA scale, about 68% of its compute falls on the stride-4 level, and about 79% sits in the head §3 fig. 2. High-precision operation (above 98%) is also needed to avoid false alarms, and TinaFace's AP drops sharply at that threshold §3 tab. 1.

The paper's assumptions: testing at a fixed VGA budget, no anchors on the stride-4 map, and training from scratch with no pretraining §4 §5.1.

Method

The paper's design has two parts. Both are developed for a fixed VGA test size.

Base detector. The detector is RetinaNet-style and uses ResNet-like blocks with basic or bottleneck residuals and a fixed bottleneck ratio of 4 §4.2. The neck is PAFPN, the head is a stack of 3x3 convolutions shared across all three levels, and group normalisation is used in the head §4.2 §5.1. Anchors are tiled at strides 8, 16 and 32, with two sizes per stride and aspect ratio 1.0 §4 §5.1. Positives are chosen with ATSS, the classification branch uses the generalised focal loss family, and box regression uses DIoU §5.1.

Sample redistribution (SR) §4.1. Training crops are square patches whose side is a multiple of the image's short edge, drawn from a fixed set of scales. The baseline set is 0.3 to 1.0. SR adds 1.2 to 2.0. The larger crops zoom out and place more small faces at the scales that the stride-8 anchors cover, and they can extend past the image border §4.1 §5.1. The paper reports positive anchors per epoch rising from 72.3K to 118.3K at anchor size 16, and from 95.9K to 115.1K at size 32 §4.1. Boxes whose centres fall outside the patch are dropped in the code transforms.py. Extremely small faces that match no anchor are effectively ignored during training §4.1.

Computation redistribution (CR) §4.2. The search space uses four backbone stages with a stem, each with a block count and a width; a neck width; a head width; and a head depth §4.2. Models are sampled uniformly and trained on the WIDER FACE training set for 80 epochs. Their validation hard AP is then used with an empirical bootstrap to estimate the computation share where the best models fall §4.2. The search runs in two steps:

• Step 1 (SCRFD-1) searches only the backbone split, with neck and head fixed at the baseline values §4.2.
• Step 2 (SCRFD-2) searches the whole detector, keeping only candidates whose backbone split follows step 1 §4.2.

The paper reports two findings from the bootstrap. The backbone's computation moves toward the shallow stages, and about 80% of the budget moves to the shallow stages §4.2 fig. 5 fig. 6. For the high-compute regime, the searched models get depth in the early stages rather than width, because width scaling is prone to overfitting §4.2 fig. 7.

The paper has no numbered equations and no algorithm boxes. The two blocks below are our reading of the method, not the paper's pseudocode.

# Sample redistribution: one training view  [§4.1] [§5.1]
# Input:  image (H, W, 3); boxes (N, 4) in pixels
# Output: image (640, 640, 3); boxes (M, 4) with M <= N
def make_training_view(image, boxes):
    # 1. Choose a square side as a fraction of the short edge.
    #    SR set: 0.3 to 1.0 plus 1.2 to 2.0  [§5.1]
    scale = random_choice(SR_SCALES)
    side = scale * min(height, width)
    # 2. Place the square at a random offset. It may extend past the
    #    image; missing pixels get a constant fill  [§4.1]
    #    [code: mmdet/datasets/pipelines/transforms.py#L965]
    crop = random_square(image, side)
    # 3. Keep only boxes whose centre lies inside the crop, and redraw
    #    the crop if none do
    #    [code: mmdet/datasets/pipelines/transforms.py#L915-L926]
    crop, boxes = keep_boxes_with_centre_inside(crop, boxes)
    # 4. Resize to 640 x 640, flip horizontally with p = 0.5, and jitter
    #    colour  [§5.1] [code: configs/scrfd/scrfd_2.5g.py#L33-L42]
    view = resize(crop, 640, 640)
    view = maybe_flip(view, boxes, p=0.5)
    view = colour_jitter(view)
    return view, boxes

# Training loss for one image over all pyramid levels  [§5.1]
# Input:  per level: cls logits (H, W, 1) and box deltas (H, W, 4)
# Output: scalar loss
def detection_loss(levels, gt_boxes, gt_labels):
    # 1. Tile anchors: two sizes per stride, aspect ratio 1.0  [§5.1]
    #    [code: configs/scrfd/scrfd_2.5g.py#L195-L200]
    anchors = tile_anchors(levels)
    # 2. Assign positive anchors with ATSS, top-k = 9 in the code  [§5.1]
    #    [code: configs/scrfd/scrfd_2.5g.py#L224]
    pos, labels = atss_assign(anchors, gt_boxes, gt_labels, k=9)
    # 3. Decode boxes and score each positive by its detached IoU
    #    with its target
    #    [code: mmdet/models/dense_heads/scrfd_head.py#L519-L523]
    pred_boxes = decode(anchors[pos], box_deltas[pos])
    quality = iou(pred_boxes.detach(), gt_boxes[pos])
    # 4. Classification: quality focal loss over all anchors, divided
    #    by the number of positives (GPU average)  [§5.1]
    #    [code: configs/scrfd/scrfd_2.5g.py#L201-L205]
    cls_loss = quality_focal_loss(cls_logits, quality, beta=2.0)
    cls_loss = cls_loss / num_positives
    # 5. Weight each positive box by its detached max sigmoid score
    #    [code: mmdet/models/dense_heads/scrfd_head.py#L484-L485]
    weight = max_sigmoid(cls_logits[pos]).detach()
    # 6. Regression: DIoU weighted by that score, divided by the sum
    #    of weights  [§5.1]
    #    [code: mmdet/models/dense_heads/scrfd_head.py#L528-L532]
    box_loss = sum(weight * diou(pred_boxes, gt_boxes[pos]))
    box_loss = box_loss / sum(weight)
    # 7. Total: classification weight 1.0, DIoU weight 2.0
    #    [code: configs/scrfd/scrfd_2.5g.py#L205-L208]
    return 1.0 * cls_loss + 2.0 * box_loss

# Two-step computation search at one FLOP budget  [§4.2]
# Input:  FLOP target T (for example 2.5 G at VGA), 320 sampled models
# Output: one detector configuration at budget T
def search_two_step(target_flops):
    # 1. Step SCRFD-1: sample backbone splits only; neck and head fixed
    #    at baseline values  [§4.2]
    backbone_pool = sample_models(target_flops, free="backbone")
    # 2. Train each model for 80 epochs; score by validation hard AP  [§4.2]
    scores = [train_and_eval(model) for model in backbone_pool]
    template = backbone_pool[argmax(scores)]
    # 3. Step SCRFD-2: sample neck and head too, keeping only models
    #    whose backbone split matches the template  [§4.2]
    #    [code: search_tools/generate_configs_2.5g.py#L210-L237]
    pool = sample_models(target_flops, free="all", match=template)
    # 4. Train and score the new pool; return the best model  [§4.2]
    return pool[argmax([train_and_eval(m) for m in pool])]

Hyperparameters

Code paths are relative to detection/scrfd/ in the repository, with mmdet/ as the library directory there. "Paper" cites the paper's text. The values in the code column are the released configs, not the paper's words.

Results

Author-reported. Setup: WIDER FACE validation split (Easy, Medium and Hard subsets), VGA-bounded single-scale testing, inference latency at 640 x 480 on an NVIDIA 2080 Ti §5.3 tab. 4. Parameters and FLOPs are multiply-adds, per the table caption tab. 4. Setup for the baselines: same training schedule and testing scale §5.1.

Table 4: accuracy and efficiency on WIDER FACE validation.

Table 3: ablation of SCRFD-2.5GF, WIDER FACE validation.

Table 2: baseline network configurations (the ResNet-2.5GF, 10GF, 34GF and MobileNet-0.5GF baselines).

Table 1: TinaFace at different test scales (the subset is not named in this table).

Headline numbers, from the text: SCRFD-34GF beats TinaFace by 3.86 hard AP and runs about 3.3x faster on the GPU, at about 20% of its compute §5.3 tab. 4. SCRFD-0.5GF beats RetinaFace-MobileNet0.25 by 21.19 hard AP at about 63% of its compute and 46% of its latency §5.3. At precision above 98%, SCRFD-2.5GF reaches 53.7 AP against 53.9 for TinaFace, with about 1.5% of the compute and 11% of the latency §5.3 fig. 10.

The repository README lists the same SCRFD numbers README.md. We have not re-run training or evaluation.

Paper vs code

Official code: the arXiv abstract links github.com/deepinsight/insightface at detection/scrfd, and the authors are affiliated with InsightFace. We cloned the repository at commit c6c3af7bd2d0a0bf4b78c78437357f3521aa5d52. The abstract link was enough, so no further search was needed. The SCRFD code sits in detection/scrfd/, and paths below are relative to it. The LICENSE file sits in detection/scrfd/ (Apache-2.0, derived from Open-MMLab's MMDetection). The repository root has no LICENSE file LICENSE.

Discrepancies:

• Baseline crops. The paper's ResNet-2.5GF baseline uses the 0.3 to 1.0 set and no SR §5.1 tab. 3. The released base config for that baseline (configs/scrfd/base_2.5g.py: depth 34, base width 16, neck 48, head 96, which matches tab. 2) samples up to 2.0 base_2.5g.py.
• Learning rate. The paper gives LR 0.01 §5.1. The baseline configs for ResNet-2.5GF and ResNet-34GF use 0.02 base_2.5g.py base_34g.py. The SCRFD-34GF config uses 0.02 scrfd_34g.py. The 2.5GF and 10GF SCRFD configs use 0.01 scrfd_2.5g.py.
• Baseline schedule. The paper says baselines use the same schedule as the searched models: 640 epochs §5.1. The ResNet-34GF baseline config runs 400 epochs, with decays at 275 and 340 base_34g.py. Its backbone is ResNetV1e with depth 56 and base width 64 base_34g.py, while tab. 2 names ResNet50 for this baseline.
• Warmup. The paper gives 3 epochs §5.1. The code uses 1500 iterations scrfd_2.5g.py. Epoch length in iterations is not stated.
• Crop fill. The paper says missing pixels take the average RGB value §4.1. The code fills them with the constant 128 transforms.py.
• Classification loss. The paper names GFL §5.1. The code uses quality focal loss with the distribution (DFL) term turned off scrfd_2.5g.py.
• Box loss weighting. The code weights each positive's DIoU by its detached max sigmoid score and divides by the sum of those weights scrfd_head.py scrfd_head.py scrfd_head.py. The paper does not describe this.
• Learned scale. The head multiplies box regression by a learned scale per level (scale_mode = 2) scrfd_head.py scrfd_head.py. The paper does not mention it.
• Test scale. The paper bounds testing at VGA 640 x 480 §5.1. The code fits the image in a 640 x 640 box, keeps the aspect ratio and pads to 640 x 640 scrfd_2.5g.py. The two agree for 4:3 frames.
• Candidate tolerance. The search generator checks backbone FLOP ratios against the template with a tolerance of eps x 5 generate_configs_2.5g.py. The paper does not state a tolerance. The generator writes 64 configs by default generate_configs_2.5g.py.
• Keypoints. The keypoint branch exists in the code, but the released 2.5GF, 10GF and 34GF configs turn it off scrfd_2.5g.py. The paper does not discuss keypoints.

The anchor sizes, the batch size, the SGD momentum and weight decay, and the 440/544/640 schedule of the SCRFD-2.5GF config agree with the paper. We did not re-run training or evaluation, so the code's numbers are not checked against the paper's tables.

Pitfalls

• Test-set reuse. The paper selects models by validation hard AP during the search and reports results on the same validation split §4.2 tab. 3 tab. 4. No separate test split, number of runs or seed variance is stated. Treat the margins (a few tenths of a point) as tentative.
• Bootstrap details missing. The paper uses an empirical bootstrap to pick the computation ranges shown in fig. 5, fig. 6 and fig. 9. The number of resamples, the confidence level and the fitting method are not stated.
• Search cost. Each sampled model trains for 80 epochs, and 320 models run per step per budget §4.2. The GPU-hours for the search are not stated.
• FLOP definition. The table caption calls FLOPs multiply-adds tab. 4. The search tool measures FLOPs at 480 x 640 search_stat.py. Do not compare these FLOPs with FLOPs from papers that use other conventions.
• Crop geometry. SR crops larger than the image are padded, and boxes whose centres fall outside the crop are dropped. The crop is redrawn until at least one centre is inside transforms.py. The paper does not state the redraw rule.
• Small faces. The paper's example is faces under 4 x 4 pixels, which get no anchor match and are effectively ignored §4.1. No formal threshold is given.
• Table 1 subset. The paper does not name the subset for Table 1 tab. 1. Figure 2 uses the hard validation subset fig. 2.
• Anchor scales. The positive-anchor counts in §4.1 are given "at the scale of" 16 and 32. These match the anchor sizes in §5.1, so read them as anchor sizes in pixels.
• Keypoints and checkpoints. The README lists keypoint checkpoints, which the paper does not cover. Keep those separate from the reported numbers.

Lineage

• Builds on: RetinaNet, feature pyramid and PAFPN necks, ATSS label assignment, generalised focal loss, DIoU, and the RegNet-style design-space method §2 §4.2 §5.1. The paper's TinaFace analysis is its starting point §3.
• Compared against: TinaFace, RetinaFace, DSFD, HAMBox and FaceBoxes tab. 4. Related searches in face detection are CR-NAS, BFbox and ASFD §2.
• Superseded by: not stated in this card's sources.