A Survey of Modern Deep Learning based Object Detection Models

In plain words

Object detection means finding every instance of a set of object types in a picture and drawing a box around each one. People do this without effort; for computers it was hard until deep neural networks arrived.

This survey walks through the main approaches of the last decade. It explains how detection is scored and which image collections are used for testing. It then goes through the image-classifier networks that detectors borrow as their "eyes", the detectors themselves, and the small networks meant for phones and other edge devices. The authors finish with a side-by-side table of accuracy and speed. Those numbers come from the original papers, not from a shared test, so they are a guide rather than a ranking.

TL;DR

• A survey, not a method: it covers datasets and metrics, backbones, two-stage and single-stage detectors, and lightweight networks §1.
• Two-stage detectors (R-CNN family) are generally more accurate; single-stage detectors (YOLO, SSD family) are faster §9.
• Author-reported headline: DetectoRS reaches 53.3 AP on COCO at 1333 input, about 4 FPS tab. 3. YOLOv4 reaches 43.0 AP at 512 input and about 31 FPS tab. 3.
• The compiled numbers come from different papers, hardware and input sizes, so they are not directly comparable §7.

Problem

Object detection must find all instances of predefined classes and localize each one with an axis-aligned box. It is treated as supervised learning on labelled images §2.1. The paper names three practical difficulties: intra-class variation (occlusion, illumination, pose, viewpoint, deformation), the large number of categories with scarce annotation, and computational cost on mobile and edge hardware §2.2.

Method

This is a survey, so the "method" is its structure and the evaluation settings it reports. The steps below follow the paper's sections.

1. Datasets and metrics §3. Four benchmarks are described: PASCAL VOC, ILSVRC detection, MS-COCO and Open Images §3.1. Precision and recall are defined from true positives, false positives and false negatives eq. 1 eq. 2. Average precision is computed per class, and mAP is the mean over classes §3.2. The benchmark summary is in tab. 1.

2. Backbone architectures §4. AlexNet, VGG, GoogLeNet/Inception, ResNet, ResNeXt, CSPNet and EfficientNet are reviewed as feature extractors for detectors §4.1–§4.7. The comparison is in tab. 2.

3. Object detectors §5. Detectors are grouped into pioneer work (Viola-Jones, HOG, DPM) §5.1, two-stage detectors §5.2, and single-stage detectors §5.3. Two-stage detectors first propose candidate regions and then classify and localize them. Single-stage detectors predict in one dense pass over predefined boxes or points §5 intro.

4. Lightweight networks §6. SqueezeNet, MobileNet, ShuffleNet, MobileNetv2, PeleeNet, ShuffleNetv2, MnasNet, MobileNetv3 and Once-for-All are described §6.1–§6.9. Their comparison is in tab. 4.

5. Comparison §7. Detector accuracy and speed are collected in tab. 3, with a COCO chart of AP against release date in fig. 10.

The two generic detection pipelines, as the survey describes them, are sketched below. This is a summary of the text, not code from any paper.

# Two-stage detector, R-CNN family (generic)  [§5.2]
# Input: image (3, H, W); Output: boxes, classes, scores
def detect_two_stage(image):
    # 1. Compute shared convolutional features with a backbone  [§5.2] [§4]
    feature_maps = backbone(image)
    # 2. Propose candidate windows with objectness scores  [§5.2]
    #    R-CNN used Selective Search; Faster R-CNN uses an RPN.
    proposals = region_proposals(feature_maps)
    # 3. Pool one fixed-size feature per proposal  [§5.2]
    #    Fast R-CNN uses RoI pooling; Mask R-CNN uses RoIAlign.
    roi_features = roi_pool(feature_maps, proposals)
    # 4. Classify each proposal and regress its box  [§5.2]
    class_scores, box_deltas = detection_head(roi_features)
    # 5. Suppress duplicate detections by IoU and class  [§5.2]
    return non_maximum_suppression(proposals, class_scores, box_deltas)

# Single-stage detector, dense prediction (YOLO and SSD families)  [§5.3]
# Input: image (3, H, W); Output: boxes, classes, scores
def detect_single_stage(image):
    # 1. Run the backbone once to get one or more feature maps  [§5.3]
    feature_maps = backbone(image)
    # 2. Each location predicts boxes and class scores  [§5.3]
    #    SSD predicts offsets from predefined default boxes  [§5.3]
    raw_boxes, raw_scores = dense_head(feature_maps)
    # 3. Remove duplicates with NMS  [§5.3]
    #    CenterNet predicts a heatmap peak per object and skips NMS  [§5.3]
    return non_maximum_suppression(raw_boxes, raw_scores)

Key equations, as stated in §3.2:

\mathrm{Precision} = \frac{TP}{TP + FP} eq. 1

\mathrm{Recall} = \frac{TP}{TP + FN} eq. 2

Dataset summary tab. 1:

Hyperparameters

The survey trains no model, so it has no training hyperparameters. The values below are the evaluation settings the text and tables state.

Results

Author-reported. The survey collects numbers from each source paper; it does not rerun them. Hardware, input size and training data differ per row §7. Most rows are on MS-COCO; rows marked * are on PASCAL VOC 2012. The split used for each row is not stated tab. 3.

Detector accuracy and speed tab. 3:

Lightweight classifiers, ImageNet top-1 tab. 4:

Backbone comparison tab. 2:

Headline text claims, also author-reported: Fast R-CNN is 146x faster than R-CNN §5.2; Faster R-CNN takes about 0.2 s per image for proposals, against 2 s for Fast R-CNN's selective search §5.2; R-FCN is 2.5 to 20 times faster than its counterpart §5.2; MobileNetv3 is 35% faster than earlier iterations at higher mAP §6.8; OFA reaches 80% ImageNet top-1 §6.9; SqueezeNet with Deep Compression is 510x smaller than AlexNet at baseline accuracy §6.1.

Paper vs code

No official code. The LaTeX source and arXiv metadata contain no code link. The only URLs in the source are tool footnotes for drawing figures (netron.app and cnn-explainer). No GitHub search was run: a survey has no method implementation, and repositories found for surveys of this kind are usually third-party lists.

Pitfalls

• Numbers are copied, not reproduced. Each row comes from its source paper, with its own hardware, input size and training set §7. The FPS column mixes devices and is not comparable across rows.
• Table 3 mixes two benchmarks. Rows marked * use PASCAL VOC 2012; the rest use MS-COCO. Both sit in the same AP columns tab. 3.
• COCO IoU step. The text says AP averages IoU thresholds from 0.5 to 0.95 "in steps of 0.5" §3.1. Ten thresholds over that range imply steps of 0.05, which is the standard COCO setting. Use 0.05 and check the source paper.
• COCO size conflicts. The text says 91 object categories and "more than two million instances" §3.1. tab. 1 lists 80 classes and about 0.9 million objects across train, validation and test. The paper does not explain this.
• Open Images counts conflict. The text gives 9.2 million images, 16 million boxes and 1.9 million detection images §3.1. tab. 1 gives 14.6 million train objects over 1.74 million train images. The split used is not stated.
• VOC counts are train plus validation. The text gives 11k images and 27k objects for VOC12 §3.1. These match the sums of train and validation in tab. 1 (11,540 images, 27,450 objects). The text does not say so.
• AlexNet layer count. The text says eight learnable layers §4.1. tab. 2 gives 7.
• ResNet-34 description. The text describes a 34-layer ResNet with 16 bottleneck modules §4.4. The paper gives no further detail, so the block type is unclear.
• Latency is unexplained. tab. 4 has no device, batch size or framework. OFA and MobileNetv3 both show 58 ms with no stated setup.
• Unsupported claims. "Lightweight networks still lack detection accuracy by more than 50%" §8 and "models with MFLOPs below 600 perform adequately on mobile devices" §7 have no table or source behind them.
• Swin Transformer placement. Swin is a backbone, but the text places it under single-stage detectors §5.3. Its row in tab. 3 has no input size or FPS.
• Numbering in the source. LaTeX labels in the source do not match the compiled order. Tables are cited here in compiled order: 1 dataset, 2 backbone, 3 detectors, 4 lightweight. In the compiled paper, fig. 10 is the COCO AP-by-release chart §7.

Lineage

• Builds on: prior surveys named in the introduction, Zou et al. (2019), Liu et al. (2018), Chahal et al. (2018) and Jiao et al. (2019). The authors say these lack recent developments §1.
• Compared against: the same prior reviews, and the source papers of each row in tab. 3 and tab. 4.
• Superseded by: not stated.