A Dataset And Benchmark Of Underwater Object Detection For Robot Picking

In plain words

A robot that picks sea creatures from underwater farm nets needs a camera model that finds each creature. The public training data for this task had three problems: the test labels were withheld, many photos were near copies of each other, and some labels were wrong or missing.

The authors merged six of these datasets, removed the near copies, and relabelled every remaining image with a detector that was then corrected by hand. The result is a single dataset with one agreed test split, so different methods can be compared on the same images. They also trained twelve common detectors on it and measured both accuracy and speed on an embedded board of the kind a robot could carry.

TL;DR

DUO is a cleaned and relabelled underwater dataset of 7,782 images with four creature classes, plus a benchmark of twelve detectors on three ResNet depths under MMDetection. Accuracy is reported as COCO AP, and speed as FPS on a Jetson AGX Xavier. Use it when you need a public underwater detection baseline with a fixed test split. Check the dataset's own terms before reuse, because the paper's CC BY 4.0 license covers the paper, not the released data.

Problem

Existing underwater datasets for robot picking, the URPC series from 2017 to 2020, do not publish test-set annotations. Researchers therefore carve their own test sets out of the training data, and results cannot be compared across papers §1 tab. 1. Within the URPC data, many images are near duplicates: URPC2017 keeps only 15% of its images after similar images are removed §1 tab. 1. Labels are also incomplete, since some sets lack the starfish class and errors and misses are common §1.

Method

The paper has two parts: a dataset (DUO) and a benchmark.

Source data. Six sources are merged: URPC2017, URPC2018, URPC2019, URPC2020 (two releases, labelled ZJ and DL) and UDD §2 tab. 1. Table 1 gives the size of each source, the number of classes it labels and its retention after near-duplicate removal tab. 1.

Table 1 marks the URPC test sets with an asterisk, meaning their annotations are unavailable tab. 1. The "Classes" column counts holothurian, echinus and scallop; the four-class sources add starfish tab. 1 §2.

Near-duplicate removal (§3.1). The authors use perceptual hashing (PHash), whose hash changes little when an image's content changes only slightly. Within one scene, images with close hashes are treated as duplicates and one is removed §3.1. The paper reports the result as 7,782 images, 6,671 for training and 1,111 for testing §3.1. The hash-distance threshold and the rule for choosing which copy to keep are not stated.

Re-annotation (§3.2). The labels are rebuilt in three stages.

1. Train a GFL detector on the original labels §3.2.
2. Predict boxes on all 7,782 images, then retrain GFL with those predictions as ground truth. The output of this model is the coarse annotation §3.2.
3. Manually correct the coarse annotation. The result is the fine annotation, stored in COCO format §3.2.

# DUO construction  [§3.1] [§3.2]
# Input:  images and labels from six source datasets
# Output: unique images and fine labels (COCO format)

def build_duo(source_images, source_labels):
    # 1. Remove near-duplicate images with perceptual hashes  [§3.1]
    # (distance threshold not stated)
    unique_images = deduplicate_phash(source_images)

    # 2. Train GFL on the original labels  [§3.2]
    detector = train_gfl(unique_images, source_labels)

    # 3. Predict on all images; the predictions become ground truth  [§3.2]
    pseudo_labels = detector.predict(unique_images)

    # 4. Retrain GFL on the predictions; this output is the coarse set  [§3.2]
    detector = train_gfl(unique_images, pseudo_labels)
    coarse_labels = detector.predict(unique_images)

    # 5. Manual correction gives the fine labels  [§3.2]
    fine_labels = manual_correct(coarse_labels)
    return unique_images, fine_labels

Dataset statistics (§3.3). DUO has 74,515 objects: 7,887 holothurian, 50,156 echinus, 1,924 scallop and 14,548 starfish §3.3. Echinus is 67.3% of objects, and the paper calls the class distribution long-tailed §3.3 fig. 2. Most objects cover 0.3% to 1.5% of the image area, and most images contain 5 to 15 objects, with a mean of 9.57 per image §3.3 fig. 3. The paper compares these numbers with PASCAL VOC and COCO, citing COCO at 7.7 and VOC at 3 objects per image §3.3. Figure 1 shows example scenes fig. 1.

Benchmark protocol (§4). The benchmark uses MMDetection V2.5.0 §4.2. Each detector is trained with the settings in the pseudocode below, then scored on the test split with COCO AP over IoU 0.50 to 0.95, AP at 0.50 and 0.75, AP for small, medium and large objects, and AP per class §4.1. Efficiency is reported as parameter count, FLOPs at 512x512 input, and FPS on a Jetson AGX Xavier in its 30 W mode §4.1 tab. 2.

# Benchmark training  [§4.2]
# Input:  train split (6,671 images), 512 x 512 after resizing
# Output: trained detector, scored on the test split (1,111 images)

def train_benchmark(detector, train_set):
    # 1. Start the backbone from ImageNet-pretrained weights  [§4.2]
    detector.backbone.load_imagenet_weights()

    # 2. Run 12 epochs with batch size 4 on one GTX 1080Ti  [§4.2]
    for epoch in range(12):
        # 3. Learning rate 0.005, divided by 10 after epochs 8 and 11  [§4.2]
        learning_rate = lr_schedule(epoch, base=0.005, steps=(8, 11))
        for images, boxes in batches(train_set, size=4):
            # 4. Resize to 512 x 512; flip horizontally with p = 0.5  [§4.2]
            images = resize(images, (512, 512))
            images = random_horizontal_flip(images, probability=0.5)

            # 5. Normalise RGB by channel mean and std  [§4.2]
            images = normalise_rgb(images)

            # 6. SGD step; linear warmup over the first 500 iterations  [§4.2]
            # (warmup form not stated)
            loss = detector.loss(images, boxes)
            sgd_step(loss, learning_rate)
    return detector

Step 5 uses the values in the hyperparameter table below. Testing uses no flip and no multi-scale augmentation §4.2.

Hyperparameters

Values from the paper's text, not from code. The code repository has no training configs (see "Paper vs code").

Results

Author-reported. Setup: the DUO test split of 1,111 images §3.1; each detector is a single model at a single scale with no test-time augmentation, 512x512 input, MMDetection V2.5.0 tab. 2 §4.2. Full table below, all rows from tab. 2. Bold marks in the source were not reproduced, because some are misplaced (see Pitfalls).

Headline numbers: the highest AP is GFL with ResNet-50, at 58.6 with 5.5 FPS and 52.35G FLOPs tab. 2. ATSS with ResNet-50 reaches 58.2 AP at 5.2 FPS tab. 2. The fastest row is FSAF with ResNet-18, at 7.4 FPS and 49.6 AP tab. 2. The paper's reading is that the multi-stage Cascade R-CNN is more accurate but slower than RetinaNet, and that ATSS and GFL combine accuracy with speed §4.3 tab. 2. It also finds that ResNet-101 is not clearly better than ResNet-50 §4.3 tab. 2, and that AP_Sc, the scallop class, is the weakest class in the paper's reading §4.3 tab. 2.

Paper vs code

No official code. The paper links a GitHub user page for the authors, and the repository it names, chongweiliu/DUO (default branch master, shallow clone at commit faad1b7caa35b32c02ca42aa420c0e3613ff324f), contains only a README. It has no LICENSE file, no training or model code, no configs and no annotation files. Its download links point to Baidu Yun and Google Drive README.md. The README reproduces the abstract and the BibTeX entry README.md. Searched: the GitHub link given in the paper's introduction §1, the authors' repository list, and a GitHub repository search for the title words and "DUO"; the search found no other repository. The benchmark's MMDetection configs and the GFL re-annotation code are not published, so the paper's settings cannot be compared with any code.

Pitfalls

• Retention figure does not reconcile. The paper gives 95% retention for DUO §3.1. Summing the train and test counts in Table 1 gives 46,872 images, and 7,782 is about 17% of that. The paper does not say what the 95% is measured against tab. 1 §3.1.
• Split rule is not in the paper. The text gives 6,671 training and 1,111 test images §3.1, but the rule for picking the test set appears only in a commented-out passage of the LaTeX source, not in the compiled paper. The released split file is not in the repository. An implementer cannot reproduce the exact split from the paper.
• Near-duplicate removal is underspecified. The PHash distance threshold and the rule for keeping one of two duplicates are not stated §3.1. Re-running the dedup will not give 7,782 images unless these are guessed.
• Re-annotation is partly self-labelled. The coarse labels come from a GFL model trained on labels it then predicts on its own training images §3.2. The paper does not report how many labels manual correction changed, how many annotators worked, or any agreement measure §3.2. Treat the "fine" labels as unverified.
• Misplaced bold marks in Table 2. The bold AP_S value for Cascade R-CNN ResNet-18 is 49.0 tab. 2, but Faster R-CNN ResNet-50 has 53.0 tab. 2. The bold AP50 for ATSS ResNet-50 is 80.1 tab. 2, but RepPoints ResNet-50 has 80.2 tab. 2. Compare values directly, not the bold marks.
• Possible transcription error. FSAF ResNet-101 lists 55.01M parameters and 55.01G FLOPs, the same number in both columns tab. 2. The paper gives no check.
• FLOPs definition. The paper defines FLOPs as "floating-point operations per second" §4.1. That is a rate, but Table 2 gives counts in G. Treat the column as an operation count at 512x512 input. The counting tool and whether multiply-accumulates count as two operations are not stated §4.1 tab. 2.
• FPS depends on setup. FPS is measured on a Jetson AGX Xavier in MODE_30W_ALL §4.1, at batch size 1 "whenever possible" tab. 2. Exceptions are not listed. Figures from other hardware, power modes or batch sizes are not comparable.
• Small-object thresholds. AP_S, AP_M and AP_L use absolute areas of 32² and 96² §4.1. The paper does not say whether areas are measured at 512x512 or at the original resolution. Source images range from 720x405 to 3,840x2,160 §2.
• Resize without padding. All images are resized to 512x512 §4.2. For non-square inputs this changes the aspect ratio, and no padding is mentioned. The paper does not say whether this is intended.
• Single runs, no variance. No seeds, repeat runs or confidence intervals are reported §4.2 tab. 2. Gaps of a few tenths of a point, such as GFL ResNet-50 at 58.6 against GFL ResNet-101 at 58.3, cannot be judged from the paper.
• Per-detector settings are missing. The benchmark says "standard configurations" §4.2. Anchors, label assigners, NMS settings and neck details are not stated per detector, so the 36 configurations cannot be rebuilt from the paper alone.
• Warmup is vague. "WarmUp" is applied for the first 500 iterations §4.2, but its form and starting factor are not stated.
• Dataset licence is separate from the paper's. The paper is CC BY 4.0. The dataset is not licensed in the paper, and the repository has no LICENSE file README.md. Its images come from the URPC and UDD sources §2. Do not assume the dataset may be used commercially until its terms are checked.

Lineage

• Builds on: the URPC challenge datasets for 2017 to 2020 and UDD, as source data and the baseline for re-annotation §2 tab. 1.
• Uses: GFL for the coarse re-annotation step §3.2; MMDetection V2.5.0 for the benchmark §4.2; ImageNet pretraining for the backbones §4.2; the WarmUp schedule §4.2.
• Compared against: Faster R-CNN, Cascade R-CNN, Grid R-CNN, RepPoints, RetinaNet, FreeAnchor, FoveaBox, PAA, FSAF, FCOS, ATSS and GFL, each with ResNet-18, 50 and 101 backbones tab. 2.
• Related datasets named but not used: COCO and PASCAL VOC, for the statistics comparison only §3.3.