SceneNet: Understanding Real World Indoor Scenes With Synthetic Data

In plain words

Teaching a computer to label every pixel of a room picture (bed, chair, wall, and so on) needs many labelled examples, and drawing those labels by hand is slow and expensive. The authors wanted a way to get as many labelled examples as they liked without a human labeller.

Their answer is to build the rooms on a computer. They take furniture models from 3D libraries, place them in a room with a few rules (bounding boxes may not overlap, a bed sits next to a nightstand, a TV can be seen from the sofa), and then take pictures of the room from random camera positions. Because the computer knows what every object is, the labels come for free. The fake depth pictures are also made to look a bit like the noisy output of a real depth sensor.

A network trained only on these synthetic depth pictures does poorly on real rooms. After a short extra training round on a small set of real labelled rooms, it gets close to methods that also use colour. The authors say depth works best for functional object categories, and that TVs and windows suffer because colour cues are missing.

TL;DR

• Synthetic indoor scenes are built from furniture models placed by simulated annealing over geometric constraints, so every rendered pixel gets a free label §3 §3.1.
• Depth is rendered from random viewpoints, given Kinect-style noise, and fed to a VGG-based segmenter as a three-channel depth encoding §3.2 §4.
• Fine-tuning on the NYUv2 training split after synthetic pre-training improves depth-only accuracy; the authors report 68.0 global and 59.9 class accuracy on 11 classes tab. 4.
• Author-reported. The paper's RGB-D comparison (Eigen and Fergus) is higher and uses colour and normals, so it is not a like-for-like comparison §4.
• The authors' code releases the rendering and noise parts. The scene-synthesis step is not released, and the code is non-commercial README.md.

Problem

Per-pixel labels for indoor depth data are scarce. NYUv2 has 795 labelled training images for as many as 894 object classes, and SUN RGB-D has 5,285 training images for 37 classes, according to the introduction §1. Human labelling is costly and produces missing or wrong labels §2, fig. 1. Existing object repositories are not directly usable for scene layouts, and the small scene sets that exist are limited in size and object count §2 §3.

The paper assumes that synthetic data can replace a large share of the labelled data if its depth looks like real sensor depth; the noise model is therefore central §3.2.

Method

1. Basis scenes. The authors compile annotated 3D models from online 3D repositories, grouped into five categories (bedrooms, offices, kitchens, living rooms, bathrooms), with at least 10 scenes per category. The per-category counts of models and objects are in tab. 1; a scene has 15 to 250 objects §3. Models are metrically accurate and labelled in 3D §3. Examples are in fig. 2 and fig. 3.
2. Scene synthesis. New scenes are made by placing objects from object repositories (ModelNet, Archive3D, the Stanford database; sizes in tab. 2) into a basis room. Bounding boxes must not intersect §3.1. Placement minimises an energy that is the weighted sum of the constraint terms in tab. 3 by simulated annealing. Objects are placed hierarchically: object-level constraints first, then groups of objects §3.1. The energy is given in eq. 2. Effects of the individual terms are shown in fig. 5.
3. Pairwise term. Object pairs that co-occur in real rooms (for example beds and nightstands, chairs and tables) get a term based on bounding-box distance, with exponent α = 2 §3.1. Co-occurrence statistics come from NYUv2 and the basis scenes §3.1 fig. 4.
4. Visibility term. An object must be visible from its partner, so boxes of other objects that block the line between them are penalised eq. 1. The paper's example is a TV that should be visible from a sofa §3.1.
5. Rendering. The authors render 10,030 depth images from random camera poses, keeping views with enough visible objects, rather than from trajectories §4. Scene labels are rendered alongside the depth §3.
6. Noise. The clean rendered depth is perturbed with a simulated Kinect noise model after two earlier works (Gschwandtner et al. and Handa et al.); the paper gives no parameters for it §3.2 fig. 6.
7. Inpainting. The noisy depth is in-painted with the NYUv2 MATLAB colourisation code, using a 3×3 kernel; 7×7 and larger kernels changed the final results little §3.2.
8. Input encoding (DHA). Each depth image is turned into three channels: depth, height above the ground plane, and angle with the gravity vector. For synthetic data the camera pose gives height and angle directly; for NYUv2 and SUN RGB-D the authors compute them (SUN RGB-D supplies tilt and rotation) §3 §4. The values are kept as floats and are not quantised to 0 to 255 §4.
9. Network. A segmentation encoder-decoder built on VGG. The paper cites Noh et al. and Badrinarayanan et al. for the architecture and uses the public code of Noh et al. §4. Weights are initialised from pretrained VGG, and the input is the three-channel DHA image at 224×224 §4. Dropout 0.5 is used in the fully connected layers in the "DO" variants §4.
10. Training schedule. Pre-train on the synthetic set (21K iterations, about 14 epochs), then fine-tune on the 795 NYUv2 training images (4K iterations, about 30 epochs). The NYU-only baseline runs 7.5K iterations (about 56 epochs) §4.
11. Evaluation. Global and per-class accuracy on the NYUv2 test set with 11 classes (paintings and books are not in the synthetic label set) and with 13 classes; SUN RGB-D is evaluated with 13 classes §4 tab. 4 tab. 7 tab. 9.

Two energy equations are the core of step 2.

\nu(v_{o},v_{n},v_{m}) = \sum_{m=1}^{N} w_{on,m}\,\max(0,\,bb_{on,m}-d_{on,m}) eq. 1

\mathcal{E}(\mathcal{C}) = \sum_{o \in \mathcal{O}} \bigg\{ \sum_{n \in \mathcal{O}} \big\{ w_{bb}\max(0,\,bb_{o,n}-d_{o,n}) + w_{pw}\,\rho(bb_{o,n},d_{o,n},M_{o,n},\alpha) + w_{\theta}\,\psi(\theta_{o,n}-\theta'_{o,n}) + \sum_{m \in \mathcal{O}} w_{o,n,m}\max(0,\,bb_{on,m}-d_{on,m}) \big\} + w_{o,w}\,\psi(d_{o,w}-d'_{o,w}) + w_{\theta}\,\psi(\theta_{o,w}-\theta'_{o,w}) \bigg\} eq. 2

Here ψ(x) = x², bb is the sum of half-diagonals of two bounding boxes, d is the distance between box centres, and 𝒞 holds the position and angle of each object §3.1. The pairwise term ρ is given as a three-case display after the "pairwise distances" paragraph §3.1. Feasibility means d ≥ bb §3.1.

# Pairwise term rho  [§3.1, display after "Pairwise distances"]
def rho(bb, d, M, alpha):
    # overlap of boxes: penalise by the ratio of box size to distance
    if d < bb:
        return (bb / d) ** alpha
    # far apart beyond the recommended maximum distance M
    if d > M:
        return (d / M) ** alpha
    # in the preferred band between bb and M
    return 0

# Scene synthesis by simulated annealing  [§3.1] [eq. 2]
# Input:  object models with bounding boxes and class labels
# Output: layout C = {p_o, theta_o} for every object o

def synthesise_scene(objects, weights, temperatures):
    # 1. Place objects, object-level constraints first, then groups  [§3.1]
    #    The initial placement is not stated.
    layout = initial_placement(objects)
    # 2. Score the layout with the weighted sum of the constraint terms  [eq. 2]
    energy = scene_energy(layout, weights)
    # 3. Propose moves and accept or reject them by annealing  [§3.1]
    #    The schedule, step count and acceptance rule are not stated.
    for temperature in temperatures:
        candidate = perturb_position_or_angle(layout)
        candidate_energy = scene_energy(candidate, weights)
        if accept(candidate_energy, energy, temperature):
            layout, energy = candidate, candidate_energy
    # 4. Return the layout; the object labels are already known  [§3.1]
    return layout

Kinect-style noise is only partly described in the paper. The released code contains the following lateral and quantisation steps; they are not stated in the paper, so treat them as code-only.

# Lateral Kinect-style noise, code only  [§3.2 cites the model]
# Input: per-pixel points, unit normals, focal length (px), theta_1, theta_2
# Output: noisy points

def add_lateral_noise(points, normals, focal, theta_1, theta_2):
    for each pixel with point p, normal n:
        # angle between the view ray and the surface normal
        # [code: utils/noise/add_kinect_noise.cu#L170-L178]
        theta = abs(acos(-(dot(n, p) - 1) / norm(p)))
        # noise std grows with theta and with depth
        # [code: utils/noise/add_kinect_noise.cu#L180-L182]
        sigma = theta_1 + theta_2 * theta / (pi_approx / 2 - theta)
        sigma = sigma * depth(p) / focal
        # displace along the normal; the axial part is zero
        # [code: utils/noise/add_kinect_noise.cu#L184-L190]
        p = p + normal_sample(0, sigma) * n
        # grazing rays (|pi/2 - theta| <= 8 degrees) get z = 0
        # [code: utils/noise/add_kinect_noise.cu#L192-L195]
        if abs(pi_approx / 2 - theta) <= 8 * pi_approx / 180:
            p.z = 0
    return points

# Depth quantisation, code only  [code: utils/noise/add_kinect_noise.cu#L336]
# Not in the paper. Metres assumed: the code multiplies by 100 for centimetres.

def quantise_depth(depth):
    # depth to centimetres, then to a disparity-like integer
    cm = round(depth * 100)
    disparity = round(35130 / cm + normal_sample(0, 1) * 0.1 + 0.5)
    # back to depth in metres
    return (35130 / disparity) / 100

Hyperparameters

Results

Author-reported. Setup: depth-only input (DHA, 224×224), NYUv2 test set, global and class accuracy. Eigen and Fergus use RGB, normals and depth, and their accuracies were recomputed by the authors on resized labels tab. 4 tab. 7.

11 classes, NYUv2 test tab. 4

13 classes, NYUv2 test tab. 7

13 classes, SUN RGB-D test tab. 9

The paper's headline claims: fine-tuning after synthetic pre-training raises NYU-DHA by 3.6 points global and 5.4 class accuracy §4; dropout adds 1.9 global and 3.2 class for NYU-only §4; on SUN RGB-D, synthetic pre-training plus fine-tuning adds 1.0 global and 3.5 class §4. The authors call this a benchmark for depth-only segmentation on SUN RGB-D §4.

Per-class, 11 classes, NYUv2 test tab. 5

Per-class, 13 classes, NYUv2 test tab. 6

Per-class, 13 classes, SUN RGB-D test tab. 8

Confidence: the paper uses the ratio of second-best to best class probability as an uncertainty map; uncertainty concentrates at object boundaries §4.1 fig. 9. Dropout-based variation ratios were tried and were not more informative §4.1.

Paper vs code

Searched: the paper's own links (the project page at robotvault.bitbucket.org, which returned HTTP 400 when fetched, and the bitbucket dataset links listed in the README); a GitHub user search for the first author, which found ankurhanda/SceneNetv1.0 (the rendering code, owner = first author). The ankurhanda/simkinect repository (also owned by the first author; no LICENSE file) implements a related noise model in Python; it is not the release tied to this paper and is not used here.

Code at 8194273afaec2039aff6a9a06e04b7b45b53e02c. Licence: the README links CC BY-NC 4.0 and says the code and data are for research use only. There is no LICENSE file. Treat as non-commercial; cite only, no verbatim quotes.

• Scene synthesis is not released. The README lists simulated annealing as a to-do item README.md, and the only annealing call in the dynamic-room program is commented out main_living_room_dynamic.cpp. The energy terms, weights and schedule of eq. 2 therefore cannot be checked against code.
• DHA conversion is not released. The README lists depth-to-DHA conversion as a to-do item README.md; the paper's C++ GPU port of the MATLAB height and angle code is not in this repository §3.
• Training is not in this repository. The segmenter is the public code of Noh et al., which the paper names §4. The hyperparameters in the Hyperparameters table therefore cannot be checked against the authors' code; the repository contains evaluation only.
• Evaluation script. getAccuracyNYU.m uses 13 classes, ignores void label 0, and resizes the labels to 224×224 with nearest-neighbour sampling before scoring getAccuracyNYU.m getAccuracyNYU.m. It reads predictions and labels from hard-coded local paths.
• Noise constants are in the code, not in the paper. Defaults are θ₁ = 0.138, θ₂ = 0.035 and focal length 480 px main_add_kinect_noise.cpp. The paper gives no values §3.2.
• Noise steps beyond the paper. The code also applies a Gaussian pixel warp (σ = 0.5, default) main_add_kinect_noise.cpp main_add_kinect_noise.cpp, the quantisation above add_kinect_noise.cu, and zeroes grazing-angle depth add_kinect_noise.cu. The paper describes none of these, only the two cited noise models §3.2.
• Axial noise is zero in the code path add_kinect_noise.cu; the paper does not say so.
• π is approximated as 22/7 in the noise code add_kinect_noise.cu, which shifts the grazing threshold and the sigma formula slightly.
• Unused parameter. A kernel-size control set to 3 is declared main_add_kinect_noise.cpp; grep found no other use in the repository. The paper's 3×3 in-painting is done with NYU MATLAB code that is not in the repository §3.2.

Pitfalls

• The energy is underdetermined. The weights, the maximum distance M and the annealing schedule are not stated §3.1 tab. 3. Eq. 2 uses the same weight symbol w_θ for object-to-object and wall-angle terms, so the paper does not say whether they are tied. Re-implementing the generator means choosing these values yourself; the code that would show them is not released.
• Pairwise distance is a variant. The paper says it uses "a slight variation" of a published pairwise term §3.1 but does not list the differences. Implement the three-case form as written in the paper.
• Visibility sum index. Eq. 1 reuses the index m both as the summation index and as the object argument. Read it as a sum over all other objects m in the scene.
• Noise is only half described. Without the code, an implementer would not know about quantisation, grazing-angle zeroing or the warp. Those are in the code as summarised above. The paper says the noise is what lets synthetic training transfer §3.2, so if you reproduce it, use the code's steps and mark them as not in the paper.
• Derived channels need pose. Height above floor and angle with gravity need camera pose and a ground-plane estimate; for NYUv2 the authors used a ported version of an external tool §3. Skipping this step changes the input the segmenter sees.
• Iteration comparison is partial. The claim that fine-tuning converges "twice as fast" compares 4K fine-tuning iterations with 7.5K for NYU-only training §4. Including the 21K synthetic pre-training iterations, the total is 25K.
• Counts disagree inside the paper. The introduction gives 5,285 SUN RGB-D training images §1; the results give 5,825 training and 5,050 test images §4. The text says 10,030 rendered images §4; the figure caption says 9K fig. 8.
• Comparisons are not like-for-like. Eigen and Fergus use colour and surface normals, and the authors' accuracies for that system were recomputed on 224×224 resized labels; Hermans et al. annotations are not public and that row is not directly comparable tab. 4 tab. 7.
• Label set changes with the synthetic data. Synthetic scenes use ModelNet10 models, which do not have painting or books, so the 11-class results drop those two classes; the 13-class results fine-tune directly on NYU labels, and the paper notes few synthetic examples for paintings and books §4.
• Thin and flat classes fail on depth alone. Synthetic-only training gives 0 for TV and near 0 for window on NYU tab. 5 tab. 6; the paper attributes this to limited synthetic examples of those classes §4 tab. 6.
• Learning-rate schedule is ambiguous. "Multiplied by 0.9 every 3 to 4 epochs" does not say whether the same schedule applies to every run, including the SUN RGB-D runs §4.
• Test-time dropout was tried; the paper reports similar gains without it §4.
• Generalisation is untested. The authors say test-time dropout might help with out-of-domain data and leave it for future work §4. Results are on NYUv2 and SUN RGB-D only.
• Scenes are static. Only static images are evaluated; video or trajectory rendering is described as future work §3 §5.

Lineage

• Builds on: the constraint-based scene layout and simulated annealing of earlier scene-synthesis work, with a pairwise term adapted from a published layout method §3.1; the Kinect noise models of Gschwandtner et al. and Handa et al. §3.2; the VGG-based encoder-decoder of Noh et al. and Badrinarayanan et al. §4.
• Compared against: Eigen and Fergus (RGB-D plus normals); Hermans et al. (RGB-D with CRF smoothing) tab. 4 tab. 7.
• Superseded by: not stated in this wiki.