The Replica Dataset: A Digital Replica of Indoor Spaces

In plain words

Replica is a set of 18 digital copies of real indoor spaces. They range from single rooms to a multi-room apartment and a hotel room. Each copy is a detailed 3D surface with very bright and very dark light levels kept in its textures (this is what "high dynamic range" means). Every surface piece is labelled with what it belongs to, such as a chair or a book, and mirrors and glass panes are marked. The authors built the copies with a custom camera rig, camera tracking, and manual clean-up. The aim is to train and test AI systems, such as virtual robots that move around, in places that look close to real ones.

TL;DR

Replica is a data release, not a method: 18 indoor 3D scenes with dense meshes, HDR textures, per-primitive semantic labels and reflector files §1 §4. The paper compares its resolution and labels with four other reconstructed datasets tab. 1 and reports no learning benchmark. The scene data falls under the repository's non-commercial research terms, not the paper's CC BY 4.0 license, so check those terms before any commercial use.

Problem

Synthetic indoor datasets look unrealistic and simplify the semantics. Scans of real rooms show artifacts, but scanner-based datasets have geometry artifacts, lighting problems and missing surfaces §2 fig. 3. The paper also says that the semantic labels of two of those datasets follow a segmentation that is inaccurate at object boundaries §2. Image-only datasets cannot support active perception, where an agent moves through the scene §1. The authors argue that a faithful 3D replica is needed for embodied-agent research and for systems that may transfer from simulation to real images §1 [abstract].

Method

The paper describes how the dataset was built §3 and what each scene contains §4 §4.1. It gives no equations, no algorithms and no code for the capture pipeline. The capture, reconstruction and annotation tools are not in the released repository (see Paper vs code).

Capture §3 fig. 4. A custom RGB-D rig with an IR projector records time-aligned IMU, RGB, IR and wide-angle greyscale video. An in-house SLAM system uses the greyscale video and the IMU to estimate 6-DoF camera poses; the paper says it is similar to the state-of-the-art SLAM systems it cites §3. Depth comes from the IR stream, using the structured-light pattern the rig projects §3.

Reconstruction §3. Depth frames are fused into a truncated signed distance function (TSDF) using the poses, in the style of KinectFusion §3. A Marching Cubes surface is extracted, simplified with Instant Meshes, and textured with a PTex-like scheme §3. Mirrors and reflective surfaces are extracted in a separate step §3.

HDR textures §3. The texture camera cycles through exposure times. Using the SLAM poses, the measured radiance for each texel is fused into 16-bit float RGB values §3. The paper states a dynamic range of about 85,000:1, which it equates to more than 16 f-stops §3.

Mesh and reflector fix-up §3.1 fig. 5. Planar reflectors are annotated by hand as polygons on the mesh §3.1. Holes are found automatically as closed cycles of boundary edges. An annotator selects each hole, and the tool fills it by triangulating the boundary (the paper names CGAL and the method of Liepa), then refines and smooths the patch §3.1.

Semantic annotation §3.2 fig. 6. Views are rendered until every mesh primitive is seen at least once. Those views are annotated in parallel with 2D instance masks, and the masks are fused back onto the mesh by a voting scheme §3.2. A superpixel-like step fills small holes in the fused labels §3.2. Last, an annotator paints corrections on the 3D mesh down to single primitives §3.2. Areas that must be anonymised (blurred or pixelated) are labelled too §3.2.

Labels are stored as a segmentation forest. Leaves are mesh primitives, the next level groups them into segments, and each root is one object instance. A class segmentation renders every instance of a class in one colour. Rendering at different tree levels gives different segmentations §3.2 fig. 6.

# Build one Replica-style scene  (summary of [§3]; the paper has no code)
# Input:  raw RGB, IR, IMU and greyscale video from the rig [§3]
# Output: mesh, HDR textures, reflectors, segmentation forest

def build_scene(raw_video):
    # Estimate 6-DoF poses from greyscale video and IMU [§3]
    poses = slam(raw_video.greyscale, raw_video.imu)

    # Compute depth from the IR structured-light stream [§3]
    depth = depth_from_ir(raw_video.ir)

    # Fuse depth frames into a TSDF, KinectFusion-style [§3]
    volume = fuse_tsdf(depth, poses)

    # Extract the surface, then simplify it [§3]
    mesh = simplify(marching_cubes(volume))

    # Fuse exposure-bracketed radiance into 16-bit HDR textures [§3]
    textures = fuse_hdr_radiance(raw_video.rgb, poses, mesh)

    # Mark planar mirrors and glass by hand [§3.1]
    reflectors = annotate_planar_reflectors(mesh)

    # Fill holes with an annotator-selected triangulated patch [§3.1]
    mesh = fill_holes(mesh)

    # Vote 2D instance masks onto primitives, then refine [§3.2]
    forest = refine(vote_masks_onto_mesh(render_views(mesh)))

    return mesh, textures, reflectors, forest

Export §4 §4.1. Each scene ships a mesh with vertex colours (low resolution and not HDR, and the paper advises against using them), HDR PTex texture files, a reflector file glass.sur, the segmentation forest as semantic.json and semantic.bin, a planar and non-planar segmentation as preseg.json and preseg.bin, and a habitat folder for AI Habitat. The habitat folder holds a semantic PLY mesh, a navigation mesh and the instance mapping §4.1. The paper recommends AI Habitat for ML use and states that it renders RGB, depth, and semantic segmentation at up to 10k frames per second §4.

Hyperparameters

The paper gives no training hyperparameters, since it contains no learned model. The table lists the dataset and capture values it does state, and marks the reconstruction settings it leaves out.

Results

Author-reported. The paper has no learning benchmark. Its only quantitative results are the dataset statistics and the comparison in Table 1 tab. 1. The Table 1 values are the authors' own estimates. Resolution is the number of pixels or mesh primitives per square metre, using median values over the semantically annotated meshes tab. 1 caption.

The paper also reports the qualitative realism tests. In a side-by-side test, raw captures and renders from the same camera pose are hard to tell apart, though the authors note artifacts and missing motion blur in the renders §1 fig. 2 §4. The renders also show glass and mirror reflectors §4 fig. 9. The paper gives no numbers for these tests.

Paper vs code

The official repository is linked from the paper's title and from §4 §4. It contains the Replica SDK (a viewer and a headless renderer) and download scripts README.md download.sh. The repository holds no capture, SLAM, reconstruction or annotation code, so the pipeline in §3 cannot be re-run from it.

• Reflectors. The paper says a reflectance of 1 means a mirror and any other value means partially transparent glass §4.1. The SDK has no branch on the value. It applies one Fresnel-weighted alpha blend to every reflector, with the stored value as the base reflectance mirror.frag MirrorRenderer.h MirrorRenderer.h. The key is named reflectivity in the code, not reflectance MirrorSurface.cpp.
• File names. §4.1 lists habitat/semantic.json and a single mesh_semantic.ply §4.1. The README lists habitat/info_semantic.json, a preseg mesh, a stage config and sorted_faces.bin README.md. Use the README names for the release as published.
• Render defaults. When a scene is HDR, the SDK sets exposure 0.025, gamma 1.6969 and saturation 1.5 PTexLib.cpp. The viewer UI starts at exposure 0.01 viewer.cpp. The paper describes no display or tone-mapping settings §3 §4.
• Scene count. The README's 18 scenes match the paper README.md.

Pitfalls

• Licence of the data. The paper is CC BY 4.0, but the repository's LICENSE is "Replica Dataset Research Terms". It allows use only for research or education that is non-commercial or not-for-profit. Employees of commercial organisations may use it only for such purposes LICENSE. Using the scenes in a commercial product needs separate permission.
• No learning benchmark. The paper does not show that models trained on Replica transfer to real images. The abstract states this only as a hope [abstract].
• Realism is qualitative. The "Turing test" fig. 2 has no reported accuracy or sample size.
• Resolution claims versus Table 1. The text calls the resolution high §2. Table 1 gives Matterport3D a higher colour resolution (97k vs 92k) and ScanNet a higher geometry resolution (20k vs 6k) tab. 1.
• Labels. Replica's semantic labels are painted by hand on the 3D mesh down to single primitives §3.2. The paper says Matterport3D and ScanNet were labelled from a Felsenszwalb-based segmentation §2. The annotation effort in person-hours is not stated.
• Pipeline not reproducible from the release. The steps in §3 (SLAM, TSDF, texture fusion, annotation) are described in words, with several parameters not stated (see Hyperparameters).
• Vertex colours are not for use. The paper advises against using mesh vertex colours §4.1. Use the PTex textures for rendering.
• Habitat throughput is the paper's claim. The 10k frames-per-second figure for AI Habitat comes from the paper and was not measured in this card §4.
• Reflector values. The mirror and glass values are stored per reflector and are not listed in the paper. Read them from glass.sur for each scene.

Lineage

• Builds on (cited, no card here yet): AI Habitat §4, KinectFusion for the TSDF step §3, Marching Cubes and Instant Meshes §3, PTex-like texturing §3, CGAL and the hole-filling method of Liepa §3.1, and an in-house SLAM system close to published direct and feature-based methods §3.
• Compared against: Matterport3D, ScanNet, Stanford 2D-3D-S and Gibson tab. 1. The related work also names SUNCG, SceneNet, RobotriX and InteriorNet as synthetic alternatives §2.
• Superseded by: not stated in the paper, and no later card in this wiki supersedes it.