xView: Objects in Context in Overhead Imagery

In plain words

Satellites photograph the Earth every day, but almost none of those photos come with labels that say where the cars, buildings, ships and cranes are. Researchers who want a computer to find such things need a labeled collection to learn from, and most public collections cover only a few kinds of objects or a few places. This paper introduces xView, a large collection of satellite photos in which people drew boxes around more than a million objects and tagged each one with one of 60 types.

The labeling was done in stages. Crowd workers drew boxes, other workers checked them, and expert reviewers compared a sample against a carefully labeled reference set before a batch was accepted. Objects range from small vehicles to large facilities, and many classes are rare, so the data is also a test of how well a detector copes with imbalance and with objects at very different sizes.

To show the data is hard, the authors trained a standard detector on three versions of the photos. The version that cut each image into tiles of several sizes did clearly better than the plain tiling, and simple image augmentation did not help. The results are the authors' own, and the detector setup is described only loosely, so the paper is most useful as a dataset reference and a starting benchmark.

TL;DR

xView is a satellite object-detection dataset of WorldView-3 imagery at 0.3 m ground sample distance, with 60 classes, over 1 million axis-aligned boxes and more than 1,400 km² of coverage [abstract] §3.4. Each 1 km² chip was annotated in QGIS and passed through worker, supervisory and expert quality control §3.2 §3.3. The paper's SSD baseline reports total mAP 0.1456 on 300×300 chips, 0.2590 with multi-resolution chips (300, 400, 500 px), and 0.1549 with augmentation on top of multi-resolution tab. 1 §4. Use it if you need an overhead detection benchmark, but note that the training recipe is not stated, and the mAP protocol is only in the released scoring code.

Problem

Public overhead datasets were, in the authors' view, limited by low class count, poor geographic diversity, few training instances or narrow class scope §1. Small objects are a particular problem: objects range from about 3 m (10 px at 0.3 m GSD) to more than 3,000 m §1. Existing datasets are compared in §1 and §2: COWC has cars only from aerial imagery in six locations, SpaceNet focuses on buildings and roads, and FMoW has no vehicles §2. The paper states the goal as a general-purpose, multi-class overhead detection benchmark that is familiar to both computer-vision and remote-sensing communities §1.

Method

The contribution has two parts: a dataset-building pipeline and a baseline experiment.

Dataset construction §3.

1. Select areas of interest (AOIs) from lat-long points of relevant sites, such as mines, ports and airfields, and draw polygons around them where imagery confirms the objects §3.1.
2. Cut the imagery into 1 km² chips, one per UTM-derived grid cell that intersects an AOI polygon. Strips with different weather can be chosen for the same AOI §3.1.
3. Annotate each chip with axis-aligned bounding boxes in QGIS using an in-house plugin. Annotators label each object individually, use the child class when confident and the parent class otherwise, and skip objects that are roughly 20% visible or less §3.2.
4. Run three-stage quality control: worker review on a rotating basis, supervisory checks for duplicates, invalid labels or geometries, and unlabeled objects, and expert review against a gold set §3.3.

The 60 classes sit under seven parent classes, with some classes (such as helipad) having no parent §3.1 tab. 2.

Expert quality control §3.3. Six 1 km² chips per batch are labeled by experts to form the gold set. A batch passes only if its precision is at least 0.75 and its recall at least 0.95 at IoU 0.5, compared with the gold set. Failing batches are remediated and resubmitted.

# Expert quality control of one annotation batch  [§3.3]
# Input:  batch boxes; gold boxes from six expert-labeled 1 km^2 chips
# Output: True if the batch is accepted

def passes_expert_qc(batch, gold):
    # Match batch boxes to gold boxes at IoU >= 0.5  [§3.3]
    precision, recall = match_at_iou(batch, gold, iou=0.5)
    # Accept only if both thresholds are met  [§3.3]
    return precision >= 0.75 and recall >= 0.95

Baseline experiments §4. The detector is SSD, which predicts from several feature layers for multi-scale detection §4. Three training sets are built from the same data:

• vanilla: non-overlapping 300×300 chips. Boxes that cross a chip edge are cropped to the chip §4.
• multires: chips of 300², 400² and 500² pixels, the three sizes combined into one dataset. How they are combined is not stated §4.
• aug: the multires set with added shifting, rotation, noise and blurring. Rotations are bounded in both directions, but the bounds are not stated §4 fig. 7.

All three are evaluated on the vanilla test set, and the experimental split is 70% train and 30% test of the whole dataset §4.

# Build the three baseline training sets  [§4]
# Input:  full xView scenes (about 3000 x 3000 px per 1 km^2)
# Output: vanilla, multires and aug training sets

def cut_chips(scene, chip_size):
    # Non-overlapping square chips; boxes are cropped at the edge  [§4]
    return non_overlapping_grid(scene, chip_size)

def build_datasets(scenes):
    # Vanilla: one chip size  [§4]
    vanilla = [cut_chips(s, 300) for s in scenes]
    # Multires: 300, 400 and 500 px chips, combined  [§4]
    multires = [cut_chips(s, n) for s in scenes for n in (300, 400, 500)]
    # Aug: multires plus shift, rotate, noise and blur  [§4] [fig. 7]
    aug = augment(multires)
    return vanilla, multires, aug

Note that augment stands for the four augmentations listed in §4, whose parameters are not stated.

Hyperparameters

Results

Author-reported. Setup: xView with 60 classes; SSD detector; the three training variants (vanilla, multires, aug) are all tested on the vanilla test set; four M60 GPUs with a seven-day cap §4. Baseline for comparisons: vanilla §4. The paper's text names the metric as total mean average precision §4.

Per-class average precision, as printed in the paper:

Source: tab. 1 §4.

Points the paper makes about these numbers §4: multires beats both vanilla and aug on total mAP; the best classes are large or uniformly backed (for example planes on runways); small cars, despite a high instance count, are detected poorly; pickup truck and hut/tent score under 1% AP in every run; and construction site rises from 0.0172 (vanilla) to 0.1711 (multires). The authors state that the augmented run was still improving when training stopped and expect it to improve with more iterations; this is a claim, not a tested result.

Paper vs code

Official code: DIUx-xView/xView1_baseline at 8efe3407861046458427ac6c47b424a62d5079db, Apache-2.0 README.md. The repository is published under the DIUx-xView organisation, and its README links the first author's blog post for the baseline models README.md. The paper does not link the repository, so the attribution rests on that organisation and README link.

Several differences and gaps:

• Training code is absent. The repository contains only inference (inference/) and scoring (scoring/) code and pre-trained frozen models provided as release assets README.md. None of the SSD training hyperparameters in the paper can be checked against code.
• Inference chip size. The code defaults to 300 px and accepts any size through -cs create_detections.py. The paper does not say which chip size each released model expects.
• Box decoding is implicit. The code reads the model's boxes in (y, x) order, swaps them to (x, y), and multiplies by the chip size, implying the model outputs boxes normalised to the chip create_detections.py. The paper says nothing about this.
• Fixed output count. The code hard-codes 250 predictions per chip create_detections.py. The paper does not state it.
• Partial borders dropped. Chip counts use integer division, so pixels beyond the last full chip are ignored create_detections.py. The paper does not mention this.
• Scoring defaults are not in the paper. score() uses IoU 0.5 by default score.py and matches greedily in prediction order, with each prediction assigned only to its highest-IoU ground-truth box matching.py. Predictions with confidence 0 are dropped score.py. AP follows the PASCAL VOC envelope method score.py. Classes 75, 82 and 87 are excluded score.py, and a class with no ground truth scores 0.0 score.py. The paper gives none of these.
• Split metrics. The code also reports small/medium/large and common/rare splits score.py, which the paper does not use.

No other discrepancies were found in the parts the code covers.

Pitfalls

• The training recipe is missing. SSD backbone, learning rate, batch size, optimiser, number of iterations, anchors and normalisation are not stated, and the released repository has no training code. An implementer cannot reproduce the 0.2590 number from this card alone.
• Two different splits. §3.4 describes a release split (train, test, val) with per-image splitting. §4 uses a 70/30 train/test split of the entire dataset, and says all three runs are scored on the vanilla test set. How the two relate, and how the 70/30 split was drawn, is not stated.
• Multires is underspecified. The paper does not say whether the 300, 400 and 500 px chips are mixed at random, sampled per iteration, or stored in separate files §4.
• Augmentation parameters are missing. Shift size, rotation range, noise level and blur kernel are not stated. Rotation is "bounded in both directions" with no number fig. 7 §4.
• Edge cropping. Boxes that straddle a chip edge are cut at the edge, which the authors say can cut large objects such as facilities and stadiums §4. No minimum visible fraction is given for training, while the annotation rule is about 20% visibility §3.2.
• mAP protocol. The paper reports mAP without naming the IoU threshold or the AP method. The scoring code uses IoU 0.5 and VOC-style AP, and treats classes with no ground truth as 0 (see Paper vs code). Compare numbers only under the same protocol.
• Figure numbering. The text cites Figure 8 for small-car extracts, but the source contains no figure numbered 8 in that position §4. The introduction cites Figure 11 for geographic locations §1, while in source order Figure 11 is the chip comparison and Figure 12 is the location map §6.2. Numbers here follow source order, because no compiled PDF was available.
• Costs the abstract hides. Training was limited to seven days on four M60 GPUs for every run, so the augmented run may be under-trained by the authors' own account §4.
• Label noise is acknowledged. Clusters of objects, such as tightly packed buildings, were sometimes labeled as one object, and the authors say this matches real-world errors §3.2.

Lineage

• Builds on: SSD (Single Shot MultiBox Detector), cited as §4 without an arXiv id in the paper.
• Compared against: PASCAL VOC, COCO, ImageNet detection, OpenImages V2, COWC, SpaceNet and FMoW §1 §2.
• Superseded by: not known from content already in this wiki.