M2DGR: A Multi-sensor and Multi-scenario SLAM Dataset for Ground Robots

In plain words

Robots that drive around indoors and outdoors need to know where they are and build a map as they go. This is called SLAM. Most public test data for SLAM comes from cars or drones, and it rarely covers what a delivery or service robot meets: entering a lift, moving from outdoors into a building, or working in a room with the lights off.

The authors drove a ground robot through such places and recorded everything it could sense at the same time. The robot carried many cameras (including a thermal camera and an event camera), a laser scanner, an inertial sensor, and satellite receivers. Each recording also comes with a reference trajectory from a precise tracker, so an algorithm's path can be compared with the true one.

The authors then ran several well-known SLAM systems on part of the data. Most of them struggled in at least one of the hard scenes. The dataset is meant as a test field for improving them.

TL;DR

M2DGR is a public SLAM dataset: 36 sequences recorded by a ground robot with a rich sensor suite, indoors, outdoors, in lifts and in darkness. Reference trajectories come from an RTK/INS system outdoors and from motion-capture or a laser tracker indoors. Authors evaluate visual and LiDAR SLAM systems on seven sequences and report that existing solutions perform poorly in some scenarios. Use it to test SLAM on ground robots; it is not a method you implement.

Problem

Most public SLAM datasets are designed for autonomous driving or aerial robots §1. The few for ground robots have a limited sensor set or limited scenes, and the cited examples lack a rich sensor set or challenging scenes such as lifts and darkness §2.2. The authors add that LiDAR SLAM was used to create the ground truth of one of those datasets, which may make it unreliable §2.2. They also note that no public SLAM dataset then had infrared data for indoor scenes §2.2, and none had both GNSS raw measurements and sky-pointing camera images §2.2.

Method

M2DGR is a dataset and benchmark, not an algorithm. The steps below cover how it was recorded and how the paper evaluates SLAM on it.

1. Platform. The robot has three layers: the bottom holds power, computer and display, the middle and top layers hold sensors §3.1. The LiDAR is connected by Ethernet and the other sensors by USB 3.0. Recording goes to a laptop with an NVMe SSD §3.1.
2. Sensors. Six fish-eye cameras cover a 360° view, arranged as two stereo pairs (forward and rear) plus two side cameras §3.2. A sky-pointing fish-eye camera monitors satellites §3.2. The other sensors are a 32-beam LiDAR, an infrared camera, a VI-sensor (color camera and IMU), an event camera, a consumer IMU, and a GNSS receiver §3.2. Specifications are in tab. 2. The paper's text and figure captions describe the sensors in words fig. 1 fig. 2.
3. Reference trajectories. Outdoors, a GNSS-IMU navigation system (Xsens Mti 680G) tracks the robot §3.2. In the room, a motion-capture system with twelve cameras at 50 Hz tracks it §3.4. Elsewhere indoors, a laser tracker follows a prism reflector mounted on the robot §3.4. tab. 3 gives the ground-truth source per scenario.
4. Calibration §3.3.
   • Pinhole cameras: MATLAB camera calibration toolbox.
   • Fish-eye cameras: Kannala-Brandt, omnidirectional and MEI models.
   • Thermal camera: a heated checkerboard whose black and white blocks are made of materials with different heat capacities, so the infrared camera can see the pattern.
   • IMU noise (white noise and random walk of gyroscope and accelerometer): a separate calibration toolbox §3.3.
   • The LiDAR is the reference frame. IMU to LiDAR uses a LiDAR-IMU calibration toolbox, cameras to IMU uses Kalibr, LiDAR to cameras uses Autoware §3.3.
5. Synchronization §3.3. There are no hardware triggers. All sensors are stamped with the same system clock. The six fish-eye cameras and the sky-pointing camera are triggered in software by an API call at the same instant. The authors report under 10 ms between cameras in their tests.
6. Sequences. The 36 sequences cover nine scenario types tab. 3. Scenes are the SJTU campus (outdoors), halls and rooms, door transitions, a lift between two floors, and a complete-darkness room §3.4 fig. 3 fig. 5. Totals in tab. 3: 13,688 s and 10,708.67 m, 1220.6 GB.
7. Data format §3.5. Data is in ROS bags. Topics are listed in §3.5. Scripts export to other formats such as EuRoC §3.5.
8. Evaluation §4. Seven sequences are used tab. 4. The metric is Absolute Trajectory Error (ATE) §4, computed with the EVO tool after alignment §4. Every method runs with its default settings §4.
   • Visual: ORB-SLAM3 in monocular mode without IMU, with pinhole, fish-eye and thermal cameras; CubemapSLAM; Multicol-SLAM; VINS-Mono §4.
   • LiDAR: A-LOAM, LeGO-LOAM, LINS, LIO-SAM §4.

The ATE protocol, as the paper and the repository describe it:

# ATE protocol  [§4]  [code: README.md#L409-L426]
# Input:  estimated poses, ground-truth poses, timestamps per sequence
# Output: ATE value in metres for one sequence

def evaluate_sequence(estimated, ground_truth):
    # Pair each estimated pose with the ground-truth pose at the same time
    # The paper does not say how pairs are matched  [§4]
    pairs = match_by_timestamp(estimated, ground_truth)
    # Align the estimate to the ground truth with EVO  [§4]
    # Flags differ by sensor type, see Hyperparameters  [code: README.md#L417]
    aligned = align_with_evo(pairs.estimated, pairs.ground_truth)
    # Aggregate the per-pose translation errors into one value  [§4]
    # The paper names ATE but does not restate the aggregate
    return ate(aligned, pairs.ground_truth)

Hyperparameters

Sensor specifications, as stated in the paper tab. 2:

Other values:

Results

Author-reported. Setup: seven sequences from the dataset tab. 4, ATE in metres, default settings of each method §4. The table is copied in full from the paper. "X" means the method failed to initialize, or tracked fewer than half of the frames (visual), or failed to initialize (GNSS) tab. 5.

Source: tab. 5. Multicol-SLAM is discussed in the text §4 but has no row here.

Headline points from the text §4:

• LiDAR-based methods generally beat vision-based methods, most clearly in large outdoor scenes. Both kinds struggle in some cases.
• Low light: ORB-SLAM3 with pinhole and fish-eye cameras fails on Roomdark06 and Street07. With the thermal camera it is much more robust on the same scene §4 tab. 5.
• Lift04: none of the tested systems tracks the whole trajectory or builds a complete map. LIO-SAM drifts badly after the robot enters the lift §4 tab. 5.
• Door01: GNSS signals degrade near the door. The visual systems produce large errors, while the GNSS result still looks normal in ATE but has failed indoors §4 fig. 4.

Paper vs code

Official code: the abstract links https://github.com/SJTU-ViSYS/M2DGR. The repository is owned by the SJTU-ViSYS organisation and its README names the first author README.md. It was cloned at commit 5db59c1fe38d8f1d2fb3a71f1f8c5581b8de00e5. The repository holds the dataset's description, download links, evaluation commands and calibration results. It does not contain the recordings, which are linked from the README README.md.

Differences between the paper and the repository:

• Scenario durations. tab. 3 gives 7958 s for Street and 1224 s for Lift. The per-sequence durations listed in the README sum to 7667 s for the ten Street sequences README.md and to 1264 s for the four Lift sequences README.md. The other seven scenario totals match. The overall total is 13688 s in tab. 3 and 13437 s from the README list.
• Street06 description. tab. 4 describes Street06 as a "straight line" sequence. In the README the straight-line sequence is street_05, and street_06 is "one turn" README.md. Street07 is "night" in tab. 4 and §4, but "dawn" in the README README.md.
• Event topic. §3.5 lists /dvs/events0 §3.5. The README lists /dvs/events README.md.
• Evaluation alignment. §4 says trajectories are aligned with EVO §4. The README uses a different flag set for each sensor family [code: README.md#L417] README.md README.md. The paper does not state the flags it used.
• Configuration files. §4 says every method runs with default settings §4. The repository root holds LVI-SAM parameter files and launch files module_sam.launch my_params_lidar.yaml. LVI-SAM is not one of the paper's evaluated methods. The per-method configurations named in the README live in an external toolkit, not in this repository [code: README.md#L390].
• Head-camera calibration. The calibration file marks the head camera's extrinsics "to be changed" calibration_results.txt.
• Sensor table. No differences found between tab. 2 and the README's sensor parameters README.md.
• License. The paper is CC BY 4.0. The repository is MIT [code: LICENSE#L1], but the README says the work is provided for academic purposes and asks commercial users to contact the authors README.md.

Pitfalls

• The data has no stated license. CC BY 4.0 covers the paper. The repository's MIT license is conditioned by the README's academic-use statement. Neither the paper nor the README states a license for the recordings. Do not ship the data in a commercial product without written permission from the authors.
• Dataset size. The abstract says "about 1TB". tab. 3 gives 1220.6 GB. Plan storage from the table figure.
• Downloads. The README warns that the file host has been unstable under load and asks users to retry later README.md. The per-sequence links point to a university SharePoint and OneDrive. They may move.
• Sync accuracy is the authors' own test. The under-10 ms figure §3.3 is not backed by timing data in the repository. Since the cameras are triggered by software calls, their timing may depend on call latency (inference, not stated in the paper).
• Ground truth source changes by scene. The Door01 and Lift04 ground truth comes from the laser tracker, not RTK tab. 3 §3.4. The paper states that GNSS signals degrade near the door §4. Compare systems on the same reference, and check the reference before drawing conclusions on door or lift sequences.
• Visual SLAM runs without IMU. ORB-SLAM3 is run in monocular mode without IMU §4. Results do not show the inertially aided modes.
• Failure marks hide magnitude. An "X" in tab. 5 means the run failed under the footnote's rule tab. 5. A finite value can still be a poor estimate. Read the magnitudes with the failures in mind.
• Multicol-SLAM has no row in tab. 5. The text says it lost track in almost every sequence §4, but no numbers are given.
• Thermal is not always better. The authors note that some scenes look texture-less in thermal images, such as a flat colored curtain §4 fig. 5. Choose the sensor per scene.
• The ATE aggregate is not spelled out. §4 cites the ATE definition §4 and the table reports values in metres tab. 5, but the text does not define how per-pose errors are combined. Recompute the statistic yourself before comparing with other papers.

Lineage

• Builds on: nothing stated. §3.5 says the scripts export to EuRoC format §3.5.
• Compared against: the datasets in tab. 1 (KITTI, UrbanLoco, Brno Urban, KAIST D/N, Pit30M, USVinland, EuRoC, UZH-FPV, TUM VI, LaFiDa, NCLT, OpenLORIS) §2.2.
• Extended by: the repository README links a follow-up dataset, M2DGR-plus README.md. This card does not cover it.