ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data

In plain words

Teaching a computer to understand a room in 3D takes many examples: colour images, depth (how far each point is from the camera) and labels saying where the furniture is. Most of the public collections were recorded with older depth cameras, and phones and tablets with built-in depth sensors were hard to use for this kind of research.

This paper releases a dataset recorded with a tablet that has a built-in LiDAR depth sensor, inside real homes. Each capture has the tablet's colour video and its low-resolution depth. A stationary laser scanner also provides a much more precise depth map of each room, aligned to the video. People drew 3D boxes around the large pieces of furniture.

The paper uses the data for two jobs: finding furniture in 3D, and sharpening a low-resolution depth map with the help of a colour photo. It trains existing methods on the data and reports the results. The data is not released under the same licence as the paper, so check its terms before using it.

TL;DR

• A mobile-LiDAR RGB-D dataset from homes, with laser-scanner depth as high-resolution ground truth and 3D oriented boxes for 17 room-defining furniture classes §1 §3.2.
• The laser-scan-to-video alignment is an offline pipeline that renders the scans from the camera's viewpoint and estimates the camera pose by feature matching and photometric refinement §3.2.
• Benchmark 1: 3D detection with three published indoor detectors, whole-scene and per-frame; whole-scene mAP 0.358 (VoteNet), 0.383 (H3DNet), 0.419 (MLCVNet) tab. 2 tab. 3 (author-reported).
• Benchmark 2: colour-guided depth upsampling at x2, x4, x8; the learned MSG and MSPF beat bilinear, JBU and FGI on L1 and RMSE tab. 4 (author-reported).
• Licence: the paper is CC BY 4.0. The dataset and code are under Apple's LICENSE file in the companion repository, which is non-commercial. See Pitfalls.

Problem

Most indoor RGB-D datasets were recorded with Kinect-class sensors or earlier hardware, and depth sensors now in consumer devices differ from those. Outdoor detection methods often assume a bird's-eye projection that does not transfer to indoor scenes. Indoor 3D detection datasets lacked either scene-level labels or oriented boxes, and most indoor depth ground truth came from a few images or from downscaled sensor depth. Depth upsampling methods were therefore tested on synthetic or pre-downscaled inputs, not on the artefacts a phone's depth sensor produces §1 §2 §4.2.

Method

Data acquisition §3.1

• Operators rent homes for a full day, in London, Newcastle and Warsaw, chosen across urban, suburban and rural locations and three socio-economic tiers. Owners consented to public release, and operators removed personal information before capture §3.1.
• Per room, the authors place a Faro Focus S70 stationary laser scanner on a tripod at about four locations on average, and record up to three video sequences with a 2020 iPad Pro, each with a different motion pattern §3.1.
• The iPad sequences hold wide and ultra-wide RGB, LiDAR depth, IMU data, ARKit world-tracking poses and an ARKit scene reconstruction. The app shows the reconstruction live so operators can check coverage §3.1 fig. 1.
• Scenes were kept static, but a venue took six hours on average, so lighting can change between sequences §3.1.

Ground truth §3.2

• Scan registration. Scanner point clouds are placed in one venue coordinate system with the vendor's software, which estimates a rigid transform per scan automatically for most scenes §3.2.
• Scan triangulation. Each point cloud is projected to 2D by stereographic projection around the scanner's nodal point and triangulated with a 2D Delaunay step. The result is a watertight mesh with an equirectangular colour texture §3.2.
• Occlusion split. Triangles whose normal deviates from the ray to the scanner by more than a threshold become occlusion geometry; the rest are foreground. The threshold value is not stated §3.2.
• Rendering. Foreground and occlusion triangles are drawn in separate OpenGL passes that both write depth. Only front-facing foreground fragments set the stencil, so the stencil marks pixels with unobstructed line of sight. Several scans are merged in screen space using their depth and stencil buffers §3.2 fig. 3.
• Camera pose. For keyframes of each iPad sequence the authors extract local features. Renderings of the scans give reference features with 3D positions. Each query feature is matched to its most similar reference, and RANSAC with PnP gives initial poses. The poses are then refined by jointly minimising a dense photometric error across keyframes and the scan renderings, and across neighbouring keyframes that see the same surface. Registration runs once offline per sequence. The optimiser and its parameters are not stated §3.2.
• Depth maps. For each refined keyframe pose, the rendered scans give a per-pixel orthographic depth map, shipped with the pose §3.2 fig. 3.
• 3D boxes. Annotators draw oriented 3D boxes for 17 furniture categories on the ARKit reconstruction. The tool projects the boxes onto video frames in real time §3.2 fig. 2.
• Splits. Venues, not frames, are assigned 80% train, 10% validation and 10% test. The test bin is not released. The same split is used for all tasks §3.2.

Task 1: 3D object detection §4.1

• Settings. Whole-scene detection works on the accumulated reconstruction of a whole scan. Single-frame detection works on one RGB-D frame at a time §4.1.
• Labels. Whole-scene labels come directly from the reconstruction boxes. For single-frame labels the authors follow KITTI's convention of keeping boxes visible in the camera frustum. They also drop boxes with too few points. 65% of frames end up with no box. Scans are capped at 300 frames, and one third of the data is subsampled, giving 365,007 frames: 323,868 for training and 41,139 for testing §4.1.
• Models. VoteNet, H3DNet and MLCVNet, all built on a PointNet++ backbone with Hough voting. A backbone with set-abstraction and feature-propagation layers gives seed points. Each seed votes for a bounding-box centre. A proposal module with a shared PointNet predicts centre, size and class §4.1.
• Single-frame variant. To fit a budget of about two weeks per model, the authors cut the output width of the four SA layers from 256 to 128, use two-layer instead of three-layer MLPs, and train for fewer epochs §4.1.

# Per-frame box flag in the released labels
# [code: threedod/benchmark_scripts/data_prepare_online.py#L102-L107]
# points (N, 3) in the upright camera frame; boxes_corners (M, 8, 3)
# Output: boolean (M,); the boxes stay in the file, only flagged

def per_frame_box_flags(points, boxes_corners):
    # Test every point against every oriented box
    # [code: threedod/benchmark_scripts/utils/box_utils.py#L127]
    inside = points_in_boxes(points, boxes_corners)  # (N, M)
    # Count the points inside each box  [§4.1 says fewer than 10 to drop]
    counts = inside.sum(axis=0)  # (M,)
    # The code flags boxes with more than 20 points
    # [code: threedod/benchmark_scripts/data_prepare_online.py#L107]
    return counts > 20

Task 2: colour-guided depth upsampling §4.2

• The task maps a low-resolution depth map to a high-resolution one using a high-resolution colour image as a guide. The input depth comes from the iPad LiDAR, so it contains real sensor artefacts and holes §4.2.
• Baselines: bilinear interpolation, Joint Bilateral Upsampling (JBU) and Fast Guided Global Interpolation (FGI), reimplemented by the authors. Learned methods: Multi-Scale Guided Network (MSG) and Multi-Scale Progressive Fusion (MSPF), both trained on ARKitScenes §4.2.
• Changes for holes. The SSIM term used by MSPF is not used, because SSIM needs every pixel. The MSPF edge loss runs over the whole image, which the authors judge not robust enough, so they replace it with a loss following MegaDepth §4.2.
• Training frames are kept only if missing depth covers at most 40% of the high-resolution map, and the LR depth agrees with the downscaled HR depth (RMSE within 7 cm; at most 20% of pixels off by more than 5 cm). Training sees one frame every two seconds: 39k frames, validated on 5.6k frames that were manually filtered for transparent and specular objects, from different houses §4.2.

The paper gives no pseudocode for training. The step below follows the released code and the loss the paper describes in words; the code is the source for the exact form.

# One training step, after a random crop and horizontal flip
# [code: depth_upsampling/train.py#L35-L38] [code: depth_upsampling/train.py#L91-L103]
# color (B, 3, H, W) in 0..255; low_res_depth (B, 1, h, w) in metres
# high_res_depth (B, 1, H, W) in metres; h = H / factor
# Output: scalar loss

def training_step(color, low_res_depth, high_res_depth, network, loss_name):
    # Keep pixels with a valid ground-truth depth above 0.01 m
    # [code: depth_upsampling/transfroms/valid_depth_mask.py#L16-L17]
    valid_mask = high_res_depth > 0.01
    # Predict the high-resolution depth from colour and LR depth
    # [code: depth_upsampling/models/mspf/mspf.py#L33-L39]
    prediction = network(color, low_res_depth)
    if loss_name == "MSG":
        # MSG: RMSE over valid pixels
        # [code: depth_upsampling/losses/__init__.py#L10-L11]
        loss = masked_rmse(prediction, high_res_depth, valid_mask)
    else:
        # MSPF: L1 plus twice a multi-scale gradient loss on log depth
        # [code: depth_upsampling/losses/__init__.py#L6-L7]
        loss = masked_l1(prediction, high_res_depth, valid_mask)
        loss = loss + 2 * masked_log_gradient(
            prediction, high_res_depth, valid_mask)
    return loss

Hyperparameters

Detection (paper text). The paper states no hyperparameters beyond these; the training code for VoteNet, H3DNet and MLCVNet is not in the companion repository.

Depth upsampling. The paper text gives no optimiser or learning rate. Values below come from the released training code, which the paper does not describe.

Results

Author-reported. Dataset benchmarks; no independent replication.

Table 2 is whole-scene 3D detection, IoU threshold 0.25 according to the companion README tab. 2 README.md. The first block lists per-class AP; the last column, "Overall", is the mAP that the text quotes.

Table 3 is per-frame (single-frame) detection tab. 3.

Table 4 is depth upsampling over ARKitScenes, with L1 and RMSE per factor. The paper gives no units; the loader uses metres dataset.py tab. 4.

Other reported numbers: the dataset has 5,048 sequences, 1,661 scenes, and 2,257 HR-depth scans over 841 venues §1 tab. 1 [fn.]. Paper text says single-frame and whole-scene results follow a similar trend, but the overall per-frame ranking differs from whole-scene (see Pitfalls).

Paper vs code

Paths are relative to the companion repository at commit 474aaa83d7b1c9080619d8485de2d7c885c01fc5 (apple-aiml-research/ARKitScenes). Its README says it accompanies this paper, and the owner is Apple's research organisation; the authors are listed with Apple. The source of the paper has no link to it. The repository contains threedod/ (data preparation and evaluation) and depth_upsampling/ (training). It contains no ground-truth registration code and no detection training code.

• Single-frame box filter. Paper: drop boxes with fewer than 10 points §4.1. Code: counts points per box and stores bboxes_mask = count > 20, without removing boxes data_prepare_online.py.
• Frustum rule. Paper: keep boxes with at least five corners in the camera frustum §4.1. No such filter is in the released code.
• Frames per scan. Paper: at most 300 frames §4.1. Code: frame stride max(2, n // 300) over n frames data_prepare_online.py. For scans with 1,000 frames that is every third frame, giving 334.
• Box size. The code's whole-scan boxes are full size, not half size as in the VoteNet code README.md.
• Detection training. Paper gives the optimiser and epochs §4.1. The training code lives in external repositories linked from the README README.md. The IoU threshold 0.25 is in the code and not in the paper text.
• Frame selection for upsampling. Paper: 40% and RMSE rules and one frame per two seconds §4.2. Code: the loader enumerates every wide/*.png in the training fold of metadata.csv dataset.py. The filters may have been applied before release; this cannot be checked from the code.
• Resolution. Paper gives no resizing. Code: for x2 and x4 the HR target is resized to 384x512 and 768x1024; for x8 the LR input is resized to 180x240 and HR stays native 1440x1920 dataset.py. Depth is resized with nearest neighbour, colour linearly dataset.py.
• Rotation. The loader rotates each frame to upright using the metadata sky direction dataset.py dataset.py. The paper does not mention this.
• Loss. Paper: no formulas. MSPF uses L1 plus twice a four-scale log-depth gradient loss [code: depth_upsampling/losses/init.py#L6-L7] gradient_loss.py. MSG uses RMSE [code: depth_upsampling/losses/init.py#L10-L11].
• SSIM. The code has no SSIM term, which agrees with the paper's statement that SSIM was dropped §4.2 [code: depth_upsampling/losses/init.py#L6-L11].
• MSG normalisation. The code min-max normalises the LR depth per image before the network and undoes it on the output msg.py msg.py. Paper does not describe this.
• Metric averaging. The code averages per-image L1 and per-image RMSE over the validation set, which is not the RMSE over all pixels of the set eval.py eval.py.

The released split files list only Training and Validation folds, so the 80/10/10 venue split and the unreleased test bin cannot be checked from the code.

Pitfalls

• Licence. The paper is CC BY 4.0. The companion repository's LICENSE is Apple's, applies to the code and the dataset (see the README's Licence section), and grants a non-commercial licence. It also has a second grant for entities below 700 million monthly active users in every month before August 2024 LICENSE README.md. The paper's source contains a commented-out checklist saying the planned dataset licence is non-commercial § checklist, not in the compiled PDF. Do not treat the dataset or the code as CC BY, and do not ship them in a commercial product without reading the LICENSE file. Questions go to the address in the README.
• Count mismatch. The text says 5,048 sequences §1. Table 1, the README and the split CSV say 5,047 tab. 1 README.md 3dod_train_val_splits.csv.
• Test split. The 10% test bin is not released §3.2. Published numbers are on the validation bin README.md. Do not compare against the paper's "test" numbers without checking.
• Single-frame labels. Two rules are unclear in the paper and differ in the code: the point threshold (10 in the paper, 20 in the code) and the frustum rule (not in the code). Implementers should follow the code's mask only if they check it first.
• Epoch budget. The sentence giving 200 epochs is not clearly scoped to single-frame training, which has its own epoch counts §4.1.
• Ordering of results. The text says the per-frame results show "a similar trend" to whole-scene §4.1. In the overall column, per-frame places VoteNet (0.427) above H3DNet (0.403) and MLCVNet far below (0.271). Whole-scene places MLCVNet first (0.419).
• Box convention. The code saves full-size boxes, and the VoteNet code uses half sizes README.md. Converting between them is a common bug source.
• Upsampling inputs. For x2 and x4 the high-resolution target is a downscaled 384x512 or 768x1024 map, not the native 1440x1920, so the reported L1 and RMSE refer to that resolution. This is not in the paper dataset.py.
• Units. The paper's table gives no unit. The code uses metres, and the frame-selection thresholds in the paper are in centimetres §4.2.
• Missing ground-truth code. The occlusion threshold, the photometric optimiser and its weights are not stated. The registration pipeline is not in the repository, so the exact ground truth cannot be rebuilt from this card.
• Cost. Each detector took about two weeks of training §4.1. The upsampling paper states no training budget; the code defaults are 200,000 steps at batch 16 train.py. Hardware is not stated for either task.
• Sensor scope. Upsampling is evaluated only on ARKitScenes, with the LiDAR artefacts of the iPad Pro §4.2. Nothing on the card shows transfer to other sensors.

Lineage

• Builds on: VoteNet (Hough voting on PointNet++), H3DNet and MLCVNet, the three detectors evaluated here. MLCVNet adds attention and H3DNet adds primitive supervision §4.1.
• Builds on: MSG and MSPF for upsampling. JBU and FGI are classical baselines. The edge loss comes from the MegaDepth work §4.2.
• Compared against: bilinear interpolation, JBU and FGI tab. 4.
• Superseded by: not known from content already in this wiki.