YOLO-Z: Improving small object detection in YOLOv5 for autonomous vehicles

In plain words

Object detectors find things like cars, people or traffic cones in a picture and draw a box around each one. YOLOv5 is a fast detector built for general use, and it struggles when the objects are small, as they are when they are far away. Small objects shrink to a few pixels, and the layers of the network tend to lose those pixels before the final decision is made.

The authors took YOLOv5 apart and changed its internal structure instead of only tuning its settings. They tried swapping the main feature extractor, simplifying how the layers are joined together, and changing which feature maps the detection step sees. The strongest change was to feed the detector a higher-resolution feature map and drop the coarsest one, so that small objects keep more of their detail.

They trained and tested on a dataset of traffic cones seen from an autonomous racing car, where cones are the small objects that matter. The modified models were more accurate on small objects, and they stayed fast enough for real-time use, with a small extra cost in time per image. The results come from one dataset, so they may not carry over to other scenes.

TL;DR

YOLO-Z is a family of four YOLOv5-based detectors (S, M, L, X) tuned for small objects. Each one uses a DenseNet backbone, an FPN neck (bi-FPN at X) and an extra higher-resolution feature map that replaces the lowest-resolution one. On a cone dataset, the authors report 5.5 to 7.7 percent relative mAP@0.5 gains for small objects over YOLOv5 at the same scale, for 0.9 to 3.7 ms more inference time on a GTX 1080 Ti at batch size 1. The results come from a single dataset with no variance reported, so treat differences of about one point as unconfirmed.

Problem

Small objects lose detail as they pass through a convolutional backbone, and YOLOv5 is a general-purpose detector that is not optimised for them §1. Two-stage detectors do better on accuracy but are slower §1. Raising the input resolution helps but adds processing time §1. Autonomous vehicles need detection that is both accurate for distant objects and fast enough to run on board §1 §2.5. The study assumes this setting: one cone-type dataset with many small objects, and a real-time budget §1 §3.2.

Method

The authors start from the four YOLOv5 scales, S, M, L and X, which differ in depth and width multipliers while keeping the same overall structure §3. They train and test each unmodified scale as a baseline. They then test each change alone against that baseline, drop the changes that help neither accuracy nor inference time, and combine the rest in further rounds §3.

The changes fall into three groups.

• Backbone §3.3.1: ResNet50 or DenseNet replaces the YOLOv5 backbone. ResNet50 is used as is. DenseNet is scaled down in proportion so that the layer count stays close to the original. The SPP layer between backbone and neck is kept unchanged.
• Neck §3.3.2: the default PAN-style neck is simplified to an FPN, and a bi-FPN neck is also tested. The text describes both replacements in one sentence, and the results tables include both variants.
• Feature maps §3.3.3 fig. 5: two ways of feeding a higher-resolution map to the neck and head. XS_inc adds it and keeps all original maps. XS_ex drops the lowest-resolution map and uses the higher-resolution one in its place. Anchor sizes in the head have to be adjusted to the new map resolutions §3.3.3.

The paper also tests depth and width modifiers, set to the values of the next larger YOLOv5 scale fig. 6, and learning rates of 0.02 and 0.005 fig. 6 §4.5. Anchors are generated from the dataset with 3 or 5 per scale fig. 6 §4.4.

The final YOLO-Z models combine these changes as shown in tab. 1. Each scale uses its YOLOv5 counterpart as the base.

The pseudocode below summarises how a YOLO-Z config is built from a YOLOv5 config. It is a reading of the text and tab. 1, not the authors' code.

# YOLO-Z config construction  [§3.3] [tab. 1]
# Input:  scale in {S, M, L, X}
# Output: detector config for that scale

def build_yolo_z(scale):
    # Start from the YOLOv5 config of the same scale  [tab. 1] [§3]
    config = yolov5_config(scale)
    # Replace the backbone with a DenseNet scaled to the scale  [§3.3.1]
    config.backbone = densenet_scaled(scale)
    # Keep the SPP layer between backbone and neck unchanged  [§3.3.1]
    config.spp = keep_original(config.spp)
    # Use an FPN neck, or bi-FPN at scale X  [§3.3.2] [tab. 1]
    if scale == "X":
        config.neck = bifpn_neck()
    else:
        config.neck = fpn_neck()
    # Replace the lowest-resolution map with a higher-resolution one  [§3.3.3]
    config.feature_maps = replace_lowest_with_high_res(config)
    # Use 3 anchors per scale at S, 5 at M, L and X  [tab. 1] [§4.4]
    if scale == "S":
        config.anchors_per_scale = 3
    else:
        config.anchors_per_scale = 5
    # Generate anchors from the dataset, then resize to the new maps  [§4.4]
    config.anchors = dataset_anchors(config.anchors_per_scale)
    config.anchors = resize_to_feature_maps(config.anchors, config.feature_maps)
    return config

No equations appear in the paper. The metric is defined in words only §3.1.

Hyperparameters

Values the paper states are listed first. Anything not stated is marked as such.

Architecture of the final YOLO-Z models tab. 1:

Results

Author-reported. Setup: a cone dataset with four classes, about 4,000 images and over 30,000 labelled objects §3.2 fig. 1. Results are reported for the blue and yellow classes only §4. The authors chose those two classes because they are the best represented §3.2 §4. Hardware and timing are described in §3.2. The baseline is the unmodified YOLOv5 at the same scale §4 tab. 2.

Headline results per scale tab. 2. Each cell shows YOLOv5 → YOLO-Z, with the relative change in brackets.

The paper's averages, checked against tab. 2: 2.7 points of mAP@0.5 on all objects, 5.9 points on small objects and 2.6 ms of inference time §4.6. The table's "difference" column is relative, so the averages are in absolute points.

Per-change results. [supp. tab. 1] is the paper's Table 1 in the supplementary material, which the arXiv source ships as a separate file. Changes are shown against the YOLOv5 baseline of the same scale. The "% mAP change" columns are relative. Results are for blue and yellow classes.

What the per-change table shows, as the paper describes it:

• Feature maps had the largest effect. XS_ex gave the largest small-object gain at every scale, 3.84% to 6.96% [supp. tab. 1] §4.3. At X, XS_inc is worse than the baseline on all objects [supp. tab. 1] §4.3.
• Backbone: DenseNet gains 1.67% to 3.74% on small objects across the scales, and it adds 2.2 to 3 ms [supp. tab. 1] §4.1. ResNet50 lowers mAP@0.5 at S, M and X, and it is much slower at L and X [supp. tab. 1] §4.1.
• Neck: FPN beats bi-FPN on small objects only at S (0.894 against 0.889) [supp. tab. 1] §4.2. At S, mAP@0.5 is tied at 0.935 for both [supp. tab. 1].
• Anchors: 3 anchors work better at S and M, and 5 at L and X [supp. tab. 1] §4.4. At M the two are close (0.905 against 0.9035 on small objects).
• Depth and width: wide helps at S and M, deep helps at L, and at X both slightly hurt on small objects [supp. tab. 1] §4.5 fig. 6.
• Learning rate: 0.02 helps at all four scales on small objects; 0.005 hurts at S, L and X [supp. tab. 1] §4.5.
• Combination versus best single change: YOLO-Z S has a lower small-object mAP@0.5 than S_XS_ex (0.925 against 0.9295), so the combination is not always better than its best part tab. 2 [supp. tab. 1].
• Dataset-level caveat: the authors say the feature-map gains likely depend on the dataset's high density of small objects §4.3, and they call for testing on other datasets §4.7.

Paper vs code

No official code.

What was searched:

• The paper text, the arXiv metadata and the bibliography: no link to a repository for YOLO-Z. The only code link in the bibliography is the upstream YOLOv5 repository, which is the base model and not the authors' code for these changes [glenn_jocher_2021_4679653].
• A project page: none linked.
• GitHub repository search with the queries "yolo-z", "yolo-zoomed", "benjumea", and "cone yolo racing". One repository under the first author's account, abenju/Scale-invariance-in-object-detection, is described as an undergraduate final-year project. It has no LICENSE file and no mention of YOLO-Z in its metadata. It is not used here, because it is not clearly the code for this paper and it has no license.

Only the paper's text, tables and figures are used.

Pitfalls

• Headline numbers in the abstract do not match Table 2. The abstract claims up to 6.9% mAP gain on small objects and "just a 3 ms" extra inference time. Table 2 gives 5.46% to 7.73% relative small-object gains and +0.9 to +3.7 ms tab. 2. The Conclusion says "close to 6%" §5. The 6.9% figure matches single-change runs, not the YOLO-Z models: S_XS_ex gives 6.96% and M_XS_ex 6.88% [supp. tab. 1]. Implement from Table 2, not from the abstract.
• Table 2 uses relative percentages; the text averages use absolute points. The averages in §4.6 (2.7 points on all objects, 5.9 on small objects) are consistent with Table 2 when computed in absolute points §4.6 tab. 2. Do not mix the two.
• Combined models are not always better than their parts. YOLO-Z S has a lower small-object mAP@0.5 than the single XS_ex change at S tab. 2 [supp. tab. 1].
• ResNet50 inference time is not always higher. The text says ResNet's inference time is "significantly higher" §4.1. At M, the ResNet50 run is faster than the YOLOv5 baseline: 10.1 ms against 11.6 ms [supp. tab. 1].
• Single runs. No seeds, repeats or variance are reported. Many differences in [supp. tab. 1] are 0.5 to 1 point, which may be within run-to-run noise. The size of the test split is not stated; the total is "close to 4,000" images §3.2.
• Which split Table 2 uses is not stated explicitly. §3.2 names a test split and says the validation split guides training §3.2. The tables do not say which split they use.
• Only blue and yellow classes are reported. The dataset has four classes §3.2, and results are given for two §4. The text says the score is an average over "the most prominent classes" §3.2, but the averaging is not defined.
• Metric is a re-implementation. The COCO-style metric was re-implemented in the authors' own testing code, and the authors say it "may be slightly inaccurate in extreme cases" §3.1. Whether area thresholds use input pixels or original image pixels is not stated. No equations describe the metric.
• Timing is limited to one setup. Timing uses one GTX 1080 Ti at batch size 1 with one CPU §3.2. Whether the reported times include pre-processing and non-maximum suppression is not stated.
• Most training hyperparameters are missing. Not stated: optimizer, epochs, training batch size, input resolution, learning-rate schedule, baseline learning rate, augmentation, pretrained weights, NMS and confidence thresholds. The paper says the best learning rate depends on the number of epochs §4.5, but the epoch count is not given.
• Anchor generation is unclear. The paper says anchors are "generated" from the dataset with 3 or 5 per scale fig. 6 §4.4. The algorithm and the resulting anchor sizes are not stated. Anchor sizes must change when the feature maps change §3.3.3.
• Feature-map resolutions are not stated. XS_ex replaces the lowest-resolution map with a higher-resolution one, but the paper gives no resolution values §3.3.3 fig. 5.
• DenseNet scaling is not specified. The paper says DenseNet was "downscaled proportionally" to keep the layer count close to the original §3.3.1. The factor is not given.
• The neck sentence is ambiguous. §3.3.2 describes simplifying PAN-Net to FPN and "replacing it" with bi-FPN, in one sentence. The results tables show both variants tab. 1 [supp. tab. 1].
• The dataset is not named, and its availability is not stated. The paper describes the dataset but gives no name, source link or license statement §3.2. Reproduction from the paper alone is not possible.
• Supplementary results are in a separate file. [supp. tab. 1] is in the arXiv source as results.tex, and the paper refers to it as supplementary material §4. Readers of the main PDF may not have it.
• The findings may not generalise. The authors say the gains likely depend on the dataset's high density of small objects, and they ask for tests on other datasets such as traffic signs §4.3 §4.7.

Lineage

• Builds on: YOLOv5, the base detector and its S, M, L and X scales §3 [glenn_jocher_2021_4679653]. The model is described in the paper and in the software release, not with an arXiv id.
• Builds on: DenseNet, arXiv:1608.06993, used as the backbone after scaling down §3.3.1.
• Builds on: ResNet50, the alternative backbone §3.3.1. No arXiv id in the source's bibliography.
• Builds on: FPN and bi-FPN, the neck variants §3.3.2. PAN-Net, the YOLOv5 default neck that is simplified here §3.3.2. No arXiv ids in the source's bibliography.
• Compared against: the unmodified YOLOv5 at the S, M, L and X scales tab. 2 [supp. tab. 1].