Car Detection using Unmanned Aerial Vehicles: Comparison between Faster R-CNN and YOLOv3

In plain words

Drones flying over roads can take pictures, and software can count the cars in them. This paper asks which of two well-known detection programs does that job better: one that first proposes candidate boxes and then checks them, and one that looks at the whole picture in a single pass.

The authors trained both on a small set of photos taken from a drone and from an open dataset, then tested them on held-out photos. Both programs rarely claimed something was a car when it was not. The single-pass program missed far fewer real cars and ran much faster per image. The paper concludes that this newer single-pass approach suits traffic monitoring from drones, though the test set is small and the timing setup is only partly described.

TL;DR

On 52 test images with 737 labelled cars, YOLOv3 misses far fewer cars than Faster R-CNN (7 vs 150 false negatives in tab. 1), and both have almost the same precision. The paper reports YOLOv3 as far faster per image §4.4. Some reported values are internally inconsistent (see Pitfalls), so check them before relying on the table.

Problem

Traffic monitoring from UAV imagery needs car detection and counting in near real time §1. The authors argue that deep detectors vary in performance depending on the scene [Abstract] §1, and they compare two state-of-the-art families: region-based (Faster R-CNN) and single-pass (YOLOv3) §1 §2. The paper assumes a single class ("car") on aerial images from altitudes of about 55 m and above 80 m §4.1. It does not test other vehicles, lighting or weather, which the authors list as future work §5.

Method

This is a comparison study; it proposes no new model. It has three parts.

Data §4.1. A training set of 218 images with 3,365 car instances, and a test set of 52 images with 737 car instances §4.1. Images come from a UAV flown over the Prince Sultan University campus and from an open GitHub dataset (cited as aerial-car-dataset, github.com/jekhor/aerial-cars-dataset) §4.1 [bib: aerial-car-dataset]. Image resolution, the number of images from each source, how images were split, and the annotation procedure are not stated.

Faster R-CNN §3.1 fig. 1. Two modules share one set of convolutional layers: a region proposal network (RPN) that outputs boxes with objectness scores at several scales and aspect ratios, and a Fast R-CNN detector. The detector pools a fixed-size feature per proposal (ROI pooling), passes it through fully connected layers, and outputs class probabilities (here "car" and "background") and box coordinates §3.1. In these experiments the shared convolutional network is Inception-ResNet v2 §3.1 §4.2. The model is the "provided" Faster R-CNN model of the TensorFlow Object Detection API; the exact configuration file is not named §4.2.

YOLOv3 §3.2 fig. 2. A single convolutional network sees the whole image once and predicts boxes at three scales with a feature-pyramid-style design §3.2. Each scale gives an N×N×3×(4+1+C) tensor, holding box offsets, an objectness score and class scores for each of three anchors §3.2. Class scores use a logistic per label (multi-label), trained with binary cross-entropy instead of softmax §3.2. The backbone is Darknet-53 §3.2. The authors describe the training assignment: the anchor that overlaps a ground-truth box most gets objectness 1, and others are ignored above a threshold of 0.7 §3.2. The paper also describes YOLO v1 and v2 as background §3.2.

# Faster R-CNN forward pass, as described  [§3.1]
# Input: image (H, W, 3); output: boxes and class probabilities

def detect(image):
    # 1. Shared convolutional features (Inception-ResNet v2 here)  [§3.1]
    features = shared_backbone(image)
    # 2. The RPN proposes boxes with objectness scores  [§3.1]
    proposals, objectness = rpn(features)
    # 3. Pool a fixed-size feature for each proposal (ROI pooling)  [§3.1]
    pooled = roi_pool(features, proposals)
    # 4. Fully connected layers give class probabilities and boxes  [§3.1]
    class_probs, boxes = fc_head(pooled)
    return boxes, class_probs

# YOLOv3 anchor assignment for one ground-truth box  [§3.2]
# Input: anchors (A, 4), box (4,), threshold (0.7 as stated)
# Output: objectness target and ignore mask per anchor

def assign_anchors(anchors, box, threshold):
    # 1. Overlap (IoU) of every anchor with the ground-truth box  [§3.2]
    overlaps = iou(anchors, box)                        # (A,)
    # 2. The best-overlapping anchor is responsible  [§3.2]
    best = argmax(overlaps)
    objectness = zeros(A)
    objectness[best] = 1
    # 3. Other anchors past the threshold are ignored; direction disputed,
    #    see Pitfalls  [§3.2] [§4.2]
    ignore = overlaps > threshold
    ignore[best] = False
    return objectness, ignore

Metrics §4.3. Counts are TP (cars detected), FP (non-cars reported as cars) and FN (cars missed). The paper defines precision = TP/(TP+FP), recall = sensitivity = TP/(TP+FN), F1 as the harmonic mean of precision and recall, and quality = TP/(TP+FP+FN) §4.3. The matching rule (IoU threshold for a true positive), the score threshold and the non-maximum-suppression setting are not stated. The equations are inline in §4.3 and not numbered.

# Metrics and timing for the comparison  [§4.3] [§4.4]
# Input: detector, test images with car boxes
# Output: TP, FP, FN, four metrics, mean time per image

def evaluate(detector, test_images, ground_truth):
    # 1. Count detections against ground truth; matching rule not stated  [§4.3]
    tp, fp, fn = match_counts(detector, test_images, ground_truth)
    # 2. Count-based metrics  [§4.3]
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f1 = 2 * precision * recall / (precision + recall)
    quality = tp / (tp + fp + fn)
    # 3. Time one detection on squares of 100 to 1500 px, step 100  [§4.4]
    times = []
    for size in range(100, 1600, 100):
        times.append(time_one_detection(detector, size))
    return tp, fp, fn, precision, recall, f1, quality, mean(times)

Hardware and software §4.2. The paper lists one machine: Intel Core i9-8950HK (six cores), NVIDIA GTX 1080 (8 GB), 32 GB RAM and Ubuntu 16.04. It does not say whether training and timing used the GPU; the GTX 1080 is the only accelerator named. Training frameworks are the TensorFlow Object Detection API for Faster R-CNN and "the YOLO v3 provided code" for YOLOv3 §4.2. The paper does not name the YOLOv3 framework or version.

Hyperparameters

Results

Author-reported, single test split (52 images, 737 car instances) §4.1. Faster R-CNN is the baseline column. Processing time is for one detection per image, averaged over 15 square sizes §4.4.

The paper states that YOLOv3 has higher recall and processing speed, and that precision is comparable §4.4 §5. Processing time for Faster R-CNN does not depend on image size; the paper reports the same average across sizes §4.4.

Paper vs code

No official code. The paper links no code repository. Its only link is to the open car dataset (github.com/jekhor/aerial-cars-dataset) [bib: aerial-car-dataset], which is not the authors' code. Its training code is the TensorFlow Object Detection API and "the YOLO v3 provided code" §4.2, both third-party.

Searches run: the abstract, the arXiv metadata and the paper text, for links; GitHub repository search for benjdira, uav car detection, and the GitHub user search for benjdira. The user bilel-bj has repositories, one of them a UAV traffic-analytics project (bilel-bj/TAU, GNU GPL v3.0). It shows no full name and cannot be verified as this paper's author, and it is copyleft, so it was not used. No code is recorded on this card.

Pitfalls

• Test-set counts do not add up. The paper says the test set has 737 car instances §4.1. TP + FN is 728 for Faster R-CNN (578 + 150) and 758 for YOLOv3 (751 + 7) tab. 1. Either the ground-truth count, the TP/FN values or the matching rule differ from what the paper describes. Recall values agree with TP/(TP+FN) on those sums.
• YOLOv3 F1 looks wrong. From the reported precision (99.73%) and recall (99.07%), the harmonic mean is about 99.40%, not 99.94% tab. 1 §4.3. The Faster R-CNN F1 (88.38%) does match its precision and recall. Quality values also differ slightly from TP/(TP+FP+FN) on the table's counts (e.g., 79.18% vs 79.17% for Faster R-CNN); this is small rounding-level noise.
• Ignore-threshold direction contradicts itself. §3.2 says anchors that overlap a ground-truth box "more than" the threshold (0.7) are ignored §3.2. §4.2 says anchors that overlap "less than" 0.7 are ignored §4.2. The assignment code is the only way to settle it, and no code is recorded here.
• Grid size is unresolved. §4.2 says the input is subdivided into 16×16 grids §4.2. §3.2 describes predictions on three scales with N×N grids §3.2, and does not give N for the 608 × 608 input. The paper does not reconcile these.
• Timing is not comparable as reported. The two detectors ran in different frameworks (TF Object Detection API and the YOLOv3 code) §4.2. The paper does not say whether timing includes image loading, pre- or post-processing, non-maximum suppression, warm-up, batching or GPU synchronisation. Faster R-CNN resizes inputs to a minimum of 600 and a maximum of 1024 §4.2, so the size-independence in §4.4 may come from that resizing; this is an inference, not something the paper states. The "real-time" framing §1 rests on the YOLOv3 figure alone.
• Training is not matched. Batch size is 1 for Faster R-CNN and 64 for YOLOv3 §4.2. Learning rates differ (0.00019 vs 0.001), only YOLOv3 states weight decay, and iteration counts are stated for Faster R-CNN (200K steps) but not for YOLOv3 §4.2. Differences in the results cannot be attributed to architecture alone.
• Single run, no variance. The paper reports one number per metric with no seeds, repeats or confidence intervals tab. 1. Whether the 52-image test set was used for model selection is not stated.
• Dataset provenance. Part of the data comes from an open GitHub dataset §4.1. Its license and how it was annotated are not stated in the paper. The per-source image counts and the split rule are not stated either.
• Terminology errors. The table row labelled "Precision (TPR)" is precision, not the true positive rate §4.3 tab. 1. The conclusion calls recall "accuracy" (99.07%) §5. The paper says "five" metrics but defines four plus time §4.3. The F1 formula in the text is garbled ("2PecisionRecall (Precision + Recall)") §4.3; the standard form is used above.
• Figure reference. The YOLO v1 text says the general architecture is shown in "Fig 2" §3.2, but Fig. 2 is captioned "YOLO v3 Architecture" fig. 2.
• Mixed units in the table. Times are in seconds for one row and milliseconds for the other tab. 1. Compare them after conversion, and only with the caveats above.
• Citation mismatch. The reference keyed Faster_R-CNN_conf is Girshick's Fast R-CNN paper, but the text cites it for Faster R-CNN. The Faster R-CNN reference is a journal entry (Faster_R-CNN_journal) with no arXiv id [bib].

Lineage

• Builds on: Faster R-CNN (Ren et al., IEEE TPAMI journal version in the paper's bibliography; no arXiv id given); Fast R-CNN and R-CNN as background §2. YOLO v1 and v2 and YOLOv3 are cited as the YOLO line §3.2 [bib].
• Compared against: each other. The paper also notes earlier car detectors (HDNN, a mean-shift plus VGG16 pipeline) as slower alternatives §2.
• YOLOv3: arXiv:1804.02767 [bib: YOLOv3].
• Superseded by: not stated in this wiki.