SeaDronesSee: A Maritime Benchmark for Detecting Humans in Open Water

In plain words

Drones can help find people who have fallen into the sea, but the computer vision software that finds them has mostly been trained on roads and traffic. This paper provides a large set of drone videos and photos of people in open water, with every person, life jacket and boat marked by hand. It also records where the drone was (its height and the angle of the camera) for every frame, which earlier drone datasets mostly did not do.

The set comes with three test tasks: finding objects in single photos, following one object through a video, and following several objects at once. The test labels are kept back and scored on a server, so that results can be compared fairly. The authors also tried several existing detection and tracking models on the data as a starting point. The baseline numbers are low for several classes, especially life jackets, and they drop for some camera angles.

TL;DR

• A UAV benchmark of people and objects in open water, with per-frame altitude, viewing angle and other sensor metadata §1 §3.1.
• Three tasks: object detection, single-object tracking (SOT) and multi-object tracking (MOT), scored on an evaluation server; test labels are withheld §1 §3.3.
• Detection baselines reach AP 30.4 (Faster R-CNN, ResNeXt-101-FPN); the smaller EfficientDet-D0 reaches AP 20.8 tab. 4.
• Tracking baselines: Tracktor++ beats FairMOT on both MOT tasks tab. 7 tab. 8; the paper attributes this partly to the detector used §5.3.
• All numbers are author-reported. The training settings for the models are in an appendix that is not in the LaTeX source §5.1.

Problem

Search-and-rescue drones need detectors and trackers for small people in open water, seen from many heights and angles, with reflections, waves and foam §1 §4. Existing UAV datasets come mostly from land traffic scenes, and maritime datasets are mostly satellite radar images, which do not give the resolution needed for search-and-rescue §2.1 §2.2. UAV datasets with rich metadata are scarce, and the authors argue this blocks multi-modal methods that use metadata to improve accuracy or speed §1.

Method

This is a dataset and benchmark paper. It contains no new model. The method is the collection, annotation, split and evaluation protocol.

Capture §3. Footage was recorded over several days with over 20 test subjects in open water. Boats carried subjects to the area, quadcopters flew at a safe distance, and a fixed-wing UAV flew from the shore along waypoints §3. Subjects could wear life jackets of several colours §3. Five cameras were mounted on four drones; the list is in tab. 2. Video was recorded at 30 fps, and for detection at most three frames per second were extracted §3. A multi-spectral camera recorded top-down imagery at 1 fps §3.

Metadata §3.1. Each frame carries a metadata record logged at 10 Hz from the clock, barometer, IMU, GPS and gimbal. Each video frame is matched to the nearest metadata stamp §3.1. The fields are listed in tab. 3. The camera pitch is the viewing angle: 90° is a top-down view and 0° a horizontal view §3.1. The gimbal yaw is not recorded; it is fixed to the UAV yaw §3.1. The authors state that the metadata lie within sensor error thresholds, but they do not analyse the error further §3.1.

Annotation §3.2. All images and frames were labelled by hand with a bounding box for these classes: swimmer, floater (swimmer with life jacket), life jacket, swimmer† and floater† (the same two on a boat), and boat §3.2. Boats on land and other objects are marked as ignored regions. Driftwood, birds and the coast are not labelled, so detectors learn to ignore them §3.2. Checks were done by experts in aerial vision §3.2. The labelling tool, DarkLabel, is described by the authors as non-commercial §3.2.

Object detection split §3.3. Individual images are assigned at random: 4/7 to training, 1/7 to validation and 2/7 to test. The authors aim for similar altitude and angle distributions in each subset §3.3. Each video is cut at random into three parts of 4/7, 1/7 and 2/7 of its length, and every 10th frame of each part is kept, to avoid near-duplicate frames across subsets §3.3.

# Split one video into train, val and test frames  [§3.3]
# Input:  frames of one video, in time order
# Output: three lists of frames

def split_video(frames):
    # Cut the video at random into parts of 4/7, 1/7 and 2/7 of its length  [§3.3]
    train_part, val_part, test_part = random_cut(frames, [4, 1, 2])
    # Keep every 10th frame of each part  [§3.3]
    train = train_part[::10]
    val = val_part[::10]
    test = test_part[::10]
    return train, val, test

Tracking split §3.3. Tracking clips are split 4/7, 1/7 and 2/7 in the same way, and the test-set annotations are withheld §3.3. The set-level counts are in §4.2 and §4.3.

Tasks §4. The object detection set has 5,630 images §4.1. The single-object tracking set has 208 clips longer than four seconds, with 393,295 frames counting duplicates §4.2. The multi-object tracking set has 22 clips with 54,105 frames and 403,192 annotated instances §4.3. Two MOT variants are defined: MOT-Swimmer tracks floaters and swimmers; MOT-All-Objects-In-Water also tracks boats, not people on boats. All tracked objects share one class §4.3. A multi-spectral subset has 432 images with 1,901 instances; only its RGB channels are used for detection §4.4.

Evaluation §5. Detectors are scored in COCO json format: AP, AP50 and AP75, and AR1 and AR10, averaged over categories except the ignored region §5.1. Per-class AP50 is also reported. The domain analysis bins AP50 by altitude and by viewing angle §5.1. Single-object trackers are scored by a success score and shown as success and precision plots §5.2. MOT is scored with the MOT benchmark protocol §5.3.

Hyperparameters

Results

Author-reported. The paper does not say on which split the tables were computed; see Pitfalls. The models are baselines, not state of the art on this data.

Object detection tab. 4. Models: Faster R-CNN with ResNeXt-101-FPN ("F. ResNeXt-101-FPN"), Faster R-CNN with ResNet-50-FPN, CenterNet with Hourglass104, ResNet101 and ResNet18, and EfficientDet-D0. S = swimmer, F = floater, S† and F† = swimmer and floater on boats, B = boat, LJ = life jacket. The per-class columns are AP50.

Source: tab. 4; FPS on a single RTX 2080 Ti tab. 4.

Altitude domains, AP50 tab. 5.

Viewing-angle domains, AP50 tab. 6.

The "avg" column is the mean of the five domain values, which the paper calls AP50avg §5.1. The printed averages match the means of the printed domain values, within rounding.

Single-object tracking fig. 5. The success and precision curves are only plotted; no numeric table exists. The paper says DiMP and PrDiMP families outperform Atom in success and precision, that PrDiMP50 slightly trails DiMP50, and that all trackers score similar to or worse than on UAV123 §5.2 fig. 5. The only number in the text is Atom's 65.0 success on UAV123, cited from prior work §5.2.

Multi-object tracking, swimmer task tab. 7. Metrics named in §5.3: MOTA, IDF1, MOTP, mostly tracked (MT), mostly lost (ML), false positives (FP), false negatives (FN), recall, precision, ID switches and fragmentations.

Multi-object tracking, all-objects-in-water task tab. 8.

Meta-data-aware detector tab. 9. A Faster R-CNN with ResNet-50-FPN baseline is compared with "5×Altitude@3" and "5×Angle@3" experts from prior work §5.4. The experts use the altitude or angle metadata to choose domain-specific features §5.4.

The paper reports +3.1 AP50avg for 5×Angle@3 over the baseline, and +9.2 and +6.4 AP50avg for the acute (A) and medium (M) angle domains §5.4. The table gives +9.2 for A and +6.6 for M; see Pitfalls.

Paper vs code

Not verified. The paper names the code as available on the project page at seadronessee.cs.uni-tuebingen.de [abstract, footnote 1], and says all experiments can be reproduced with the provided code §5. This run could not open that page, so no official repository was confirmed and no code field is set.

What was searched: a GitHub repository search for "SeaDronesSee" returned 13 repositories. None is listed as official, and none of the four authors' names matched an owner, except one account whose GitHub profile name is "Benjamin Kiefer" (repository "Ben93kie/SeaDronesSee", described as a vision benchmark for maritime search and rescue). That repository has no license, so it was not cloned or used. Ownership was not confirmed from the project page. The other repositories are third-party projects using the dataset.

Next step for a reviewer: open the project page, and if it links a repository owned by the authors with a license, add code, code_commit and code_license and compare its configs with the tables above.

Pitfalls

• The appendix is missing from the source. The paper refers to an appendix for detector training configurations, the metric definitions and the upload format §5.1 §3.3. The LaTeX source in this run does not include it, and the training settings in the Hyperparameters table are therefore "not stated".
• Which split the tables use is not stated. The test-set annotations are withheld and results are obtained by uploading predictions to the server §3.3 §5. The tables could be from the validation split or from the server. Check before comparing against them.
• The altitude range is inconsistent. The abstract and the detection row of tab. 1 say 5–260 m. The single-object and multi-object tracking rows say 5–150 m tab. 1. The highest altitude domain is 208–259 m §5.1. The paper does not say which clips reach which heights.
• The altitude bins overlap. L = 5–56 m and LM = 55–106 m share the interval 55–56 m. The angle bins start at 7°, although the viewing angles are 0–90° §3.1 §5.1. Samples outside the bins are not discussed.
• The medium-right angle bin has two ranges. The text gives 56–73° §5.1; the caption of tab. 6 gives 57–73°.
• The split counts do not match the stated ratio. Images are 2,975 / 859 / 1,796, which is not 4:1:2 §4.1. Video frames are added to the subsets, so the total per subset need not match exactly, but the paper does not give the per-subset video-frame counts. Use the released split instead of re-deriving it.
• The random cut is not reproducible from the text. The seed and cut positions for the videos are not stated §3.3.
• One text number disagrees with its table. The text says +6.4 AP50avg for the medium angle §5.4. tab. 9 gives 39.3 − 32.7 = 6.6.
• Frame and instance counts differ by section. The abstract gives over 54,000 frames and 400,000 instances. The multi-object tracking section gives 54,105 frames and 403,192 instances §4.3. The single-object tracking section gives 393,295 frames counting duplicates §4.2. Each figure is correct for its own subset, but they cannot be summed.
• Detector speed is not measured on embedded hardware. Real-time claims for EfficientDet-D0 on an Nvidia Xavier are supported by a citation to prior work, not by measurements in this paper §5.1. All FPS numbers here were measured on one RTX 2080 Ti tab. 4.
• The tracker comparison is confounded by the detector. FairMOT uses a CenterNet detector with DLA34 or ResNet34, while Tracktor++ uses Faster R-CNN with ResNet50. The paper says the detector may explain Tracktor++'s advantage, but did not test this §5.3.
• Tracker training is not described. The compiled paper does not state whether the single-object trackers were trained on SeaDronesSee or used pre-trained weights §5.2.
• The success score is not defined in the paper. The paper refers to VisDrone for the success and precision protocol, and gives no IoU thresholds for the curves §5.2.
• Life jackets are very rare and very hard. Life-jacket AP50 is between 0 and 0.6 for every model in tab. 4 §5.1.
• Licence of the data is not stated. The card carries the paper's licence, CC BY 4.0, which covers the paper text. The paper does not state a licence for the images, the annotations or the evaluation code. Check the dataset's own terms before using its images commercially.
• The annotation tool is non-commercial. DarkLabel is described as non-commercial §3.2. This card does not claim anything about the tool's licence for the data it produced.
• The figures are not reproduced. No image is included in this card. Figure numbers follow the order of figure environments in the LaTeX source. The compiled PDF was not checked.
• Commented-out text is not used. The source contains commented-out sentences, for example one about pre-trained trackers. None of this is part of the compiled text, so this card does not use it.

Lineage

• Builds on: prior UAV and maritime datasets. Traffic datasets include VisDrone and UAVDT; the maritime satellite and radar sets include the Airbus ship dataset; the UAV-based human-detection work named includes Gallego et al. and Lygouras et al. §2 tab. 1. None of these has a card in this wiki.
• Metadata-aware methods: the 5×Altitude@3 and 5×Angle@3 experts are from Kiefer et al. §5.4 tab. 9. No card for that work exists in this wiki.
• Compared against, in the dataset comparison: DOTA, Airbus Ship, AU-AIR, UAV123, DTB70, UAVDT, VisDrone tab. 1.
• Baseline models: Faster R-CNN §5.1, CenterNet §5.1, EfficientDet §5.1, FairMOT §5.3, Tracktor++ §5.3, DiMP, PrDiMP and ATOM §5.2.
• Superseded by: not stated.