RELLIS-3D Dataset: Data, Benchmarks and Analysis

In plain words

Most self-driving research uses city streets, so the data and tools are built for roads. A robot that drives over dirt trails, grass, bushes and puddles needs to know what each of those is, and whether it can drive over it or must avoid it. RELLIS-3D is a dataset recorded by a ground robot on a university campus. It holds camera images, laser scans (LiDAR), stereo images and GPS and motion sensor readings. People labelled each pixel of 6,235 photos and each point of 13,556 laser scans into 20 kinds of terrain and objects.

The authors ran two well-known segmentation models on the labelled data. Both did much worse than on city datasets. They think the main reason is that a few classes are rare. The raw recordings are published too, so others can build new methods without driving the robot again.

TL;DR

RELLIS-3D is an off-road dataset with camera, LiDAR, stereo and GPS/INS data. It has pixel labels for 6,235 images and point labels for 13,556 LiDAR scans, in 20 classes §IV-B §IV-A. The authors report 51.55 (HRNet+OCR) and 52.92 (Gated-SCNN) mIoU on images, and 43.07 (SalsaNext) and 19.97 (KPConv) on point clouds tab. 2. Check the split and class-count inconsistencies under Pitfalls before quoting numbers. The repository states non-commercial terms for the data and code README.md.

Problem

LiDAR and inertial navigation alone perform poorly off-road. They mark bushes as obstacles even where a larger vehicle could drive through them §I. Semantic labels help, but most segmentation datasets are urban. The off-road datasets that exist either have one sensor type or few labels §I §II tab. 1. Off-road scenes also have unclear class boundaries, uneven terrain, and class frequencies that differ between sites §I §V-B.

Method

This is a dataset and benchmark paper. It has no new model. It describes three things: the sensor rig and its calibration §III-A–§III-D, the recording and labelling pipeline §IV-A–§IV-B, and reference segmentation runs with their metric §V.

1. Record five sequences by teleoperating the robot along three unpaved trails. Each sequence lasts around five minutes §IV-A.
2. Synchronise the sensors with PTP. The GPS/IMU unit cannot be PTP-synced, so each LiDAR or camera frame takes the GPS/IMU reading with the closest timestamp §III-B.
3. Calibrate the cameras with a pinhole model plus distortion, using the ROS camera calibrator built on OpenCV §III-C. Estimate LiDAR–camera extrinsics with three cited methods, and check them with the mean line re-projection error, since no ground truth exists §III-D.
4. Label images. Trained annotators label pixels, with one annotator per sequence, followed by in-house checks. The camera stream is downsampled to 5 Hz, and frames where the robot is stationary are not duplicated §IV-B.
5. Label LiDAR scans from the image labels, as in the pseudocode below §IV-B.
6. Use the 20-class ontology, which includes void. It borrows from RUGD and adds classes such as mud, barriers and rubble. Puddle and deep water are separate classes §IV-A.
7. Evaluate with mIoU on fixed splits, for two image models and two point-cloud models §V-A–§V-C.

# Label LiDAR scans from labelled images  [§IV-B] [§III-D]
# Input:  labelled images, camera-LiDAR calibration, scans of one sequence
# Output: one class per point, for every scan
def label_sequence(images, calibration, scans):
    # Start from the image labels projected into each scan  [§IV-B]
    initial = project_image_labels(images, calibration, scans)
    # Register and loop-close the sequence to get one pose per scan  [§IV-B]
    poses = slam_register(scans)
    # Refine by hand over overlapping scans in the stacked cloud  [§IV-B]
    return refine_labels(initial, scans, poses)

# Mean IoU over classes  [eq. 1] [§V-A]
# Input:  predicted, truth: class id per pixel or point (flat arrays)
# Output: scalar mIoU
def mean_iou(predicted, truth, num_classes):
    ious = []
    # One class at a time, counting pixels or points  [§V-A]
    for c in range(num_classes):
        true_pos = count(predicted == c and truth == c)
        false_pos = count(predicted == c and truth != c)
        false_neg = count(predicted != c and truth == c)
        # Division fails when a class has no pixels; handling not stated
        ious.append(true_pos / (true_pos + false_pos + false_neg))
    # Average over the C classes; whether void is counted is not stated
    return mean(ious)

Key equation:

mIoU = \frac{1}{C}\sum_{c=1}^{C}\frac{TP_c}{TP_c+FP_c+FN_c} eq. 1

Here TP, FP and FN are true positives, false positives and false negatives for class c, and C is the number of classes §V-A.

Hyperparameters

The paper states almost no training settings. The rows marked code come from the official repository, not from the paper. They describe the setup the authors used, but the paper does not confirm it.

Results

Author-reported. Setup: RELLIS-3D with the splits above; image models are 2D and score 19 classes, point-cloud models are 3D and score 15 classes. Hardware is not stated. The paper compares its numbers against published urban results: Cityscapes and SemanticKITTI §V-B §V-C. Those datasets differ, so the comparison is not like-for-like.

Per-class IoU tab. 2. A dash means the class is not in that model's class list. Mean is computed over the listed classes (see Pitfalls).

Other reported results:

• Image models, on the same split, with urban references the paper cites: Gated-SCNN 52.92 vs 74.7 on Cityscapes; HRNet+OCR 51.55 vs 81.1 on Cityscapes §V-B.
• Point clouds: SalsaNext 43.07 mIoU; KPConv 19.07 mIoU in the text, but 19.97 in the table §V-C tab. 2. The paper gives SemanticKITTI references of 59.5 and 58.8 "respectively" §V-C.
• Label-distribution sampling for KPConv adds 0.6 mIoU points §V-C.
• Fence visibility: a fence hidden behind bushes in images and in a single scan appears only after several LiDAR scans are registered §V-D fig. 7.
• Largest errors in images, per the paper: log, pole, water and building §V-B tab. 2.

Paper vs code

Repository: https://github.com/unmannedlab/RELLIS-3D at commit c17a118fcaed1559f03cc32cc3a91dedc557f8b8. It belongs to the authors' lab; the paper's abstract links it README.md.

• Splits. The paper's image split (3,302 / 983 / 1,672) and scan split (7,800 / 2,413 / 3,343) are not in the repo. Configs point to train.lst, val.lst, pt_train.lst and similar files seg_hrnet_ocr_w48_train_512x1024_sgd_lr1e-3_wd5e-4_bs_12_epoch484.yaml Rellis.py. The counts cannot be checked.
• "Same splits" for points. The paper says point experiments use "the same data splits as for the image data" §V-C, but the counts differ. The code loads separate point lists. The paper does not say whether this means the same sequences or the same frames.
• Image class mapping. The HRNet loader keeps sky, object, asphalt and building as classes, and maps raw ids to indices 5, 7, 8 and 9 rellis.py. The SalsaNext map sends sky, object, asphalt and building to void rellis.yaml. The paper's point table agrees with the SalsaNext map tab. 2.
• Void in HRNet. HRNet maps raw void (id 0) and dirt (id 1) to class 0, and its configured ignore value is 255, which this map never produces rellis.py seg_hrnet_ocr_w48_train_512x1024_sgd_lr1e-3_wd5e-4_bs_12_epoch484.yaml. Whether class 0 enters the loss or the mIoU was not checked.
• Which HRNet config produced the numbers. Both LR variants exist. The README and train.sh use the 0.001 file README.md. The paper does not say which one was used.
• Batch size. The file name says bs_12, but the config gives 4 per GPU on 2 GPUs (8 total) seg_hrnet_ocr_w48_train_512x1024_sgd_lr1e-3_wd5e-4_bs_12_epoch484.yaml.
• SalsaNext config. The run script uses the Ouster config run_salsanext.sh. Its FOV matches the paper's 45° §III-A. A different config, benchmarks/SalsaNext/salsanext.yml, names an HDL-64 sensor and is not the one the run script uses salsanext.yml.
• KPConv class weights. The paper describes label-based sampling §V-C. The config's class-weight lines are commented out train_Rellis.py. How the label-based sampling is implemented was not checked.
• Evaluation code. The notebooks in utils/ (Evaluate_img.ipynb, Evaluate_pt.ipynb) were not reviewed. They are the metric code the README points to README.md. Check them before reproducing the mIoU numbers.

Pitfalls

• Numbering and version. Citations follow the IEEE conference template's Roman-numeral section style (§III-A and so on) in arXiv v4 source. A different version may renumber them.
• Image split does not add up. 3,302 + 983 + 1,672 = 5,957, but 6,235 images are annotated. The 278 missing images are not explained §IV-B §V-B.
• Class counts disagree. The text says 19 image classes and 15 point classes §V-B §V-C. The table caption says 20 and 16 tab. 2. The 20-class ontology includes void §IV-A.
• KPConv mIoU disagrees. 19.07 in the text, 19.97 in the table. The table value is the one that matches the arithmetic (see next point).
• Means. Arithmetic check on the table: the image means (51.55, 52.92) equal the average of the 18 listed class columns, and the point means (43.07, 19.97) equal the average of the 14 listed columns. So void is not in the mean. The paper does not say this.
• Accumulation and empty classes. Whether TP, FP and FN are summed over the test set or averaged per image is not stated. Handling of classes with no pixels is not stated §V-A.
• Point count. Scans are said to contain "up to 13,056 points" §IV-B, while the scan count is 13,556. The point count may be a typo. Not verified.
• GPS rate. §III-A lists 300 Hz GPS and 100 Hz IMU. §III-B says the GPS/IMU system updates at 100 Hz. Which stream is matched to frames is not stated.
• Label quality. Labels come from crowdsourcing with in-house checks §IV-B. The paper reports no inter-annotator agreement or error rate. Point labels are propagated from image labels, then refined, and scans can disagree where SLAM registration is imperfect §IV-B.
• Class imbalance is the stated cause. The paper attributes the weak image results mainly to imbalance and unclear boundaries §V-B, and the weak KPConv result to imbalance and unstructured data §V-C. These are the authors' beliefs; no ablation isolates them.
• Cross-dataset comparison. The Cityscapes and SemanticKITTI numbers come from other papers and other setups. Do not rank them against these results.
• Single site. All five sequences come from one facility §VI. The paper says diversity is a challenge for splits.
• Licences. This card is published under the paper's CC BY 4.0. The dataset and repository are not covered by that licence. The README states CC BY-NC-SA 3.0 for both README.md. Bundled GSCNN code is CC BY-NC-SA 4.0 LICENSE. No snippet from these repositories is quoted here; the code references are citations only. Anyone using the data or code must check the non-commercial terms first.
• Dataset availability. The README links Google Drive and Baidu Pan downloads, some limited in quota and one requiring email access README.md. Not all files are verified here.

Lineage

• Builds on: the RUGD ontology (not a card) for its class design §IV-A.
• Benchmarks: HRNet+OCR (Wang et al. and Yuan et al., arXiv:1909.11065); Gated-SCNN (arXiv:1907.05740); SalsaNext (arXiv:2003.03653); KPConv (arXiv:1904.08889). None has a card, so no content is taken from them §V-B §V-C.
• Compared against (named only): RUGD, Freiburg Forest / DeepScene, YCOR, the NREC person-detection set, and Dabbiru et al. §II tab. 1. Urban references: Cityscapes, SemanticKITTI and others §V.
• Superseded by: not stated.