Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering

In plain words

A neural radiance field is a learned model that can draw a scene from new camera angles. Most of them learn the whole room as one blob, so the model does not know where the chair ends and the wall begins. That makes it hard to move a chair, duplicate a toy or add furniture afterwards.

This paper splits the job in two. One part of the network learns the whole room, including the background. A second part learns each object on its own, guided by rough outlines of the objects in the photos. Each object gets a small learned label that tells the second part which object to draw. Once trained, you can switch the label to draw a single object alone, or move and copy it, and then put the scene back together.

The tricky part is objects that are partly hidden behind other things. The outline says "not this object" for the hidden pixels, which is wrong, so the method works out where the hidden space is and leaves it out of training. The authors report that this keeps rendered objects intact in cluttered rooms.

TL;DR

• Scene plus per-object neural radiance fields, trained jointly from posed images and rough 2D instance masks §3.1 §3.2.
• Object branch is conditioned on a learnable activation code per object; one object is sampled per ray during training §3.3.
• Two occlusion fixes: scene-guided sampling and a 3D guard mask that blocks gradients in occluded space §3.3.
• Author-reported: scene rendering on par with NeRF and sparse voxels on ToyDesk and ScanNet, and clean single-object renders for editing tab. 1 §4.4 §4.5.
• Official code is MIT, but several defaults differ from the paper (see Paper vs code).

Problem

Monolithic neural radiance fields encode a scene as one network or one volume. They are not aware of object identity, so high-level edits such as moving or adding furniture are not supported §1. The closest object-level method, OSF, needs training images captured for each object in isolation, which the authors say is not feasible for cluttered real rooms §1.

Applying a rough instance mask directly is also wrong for occluded objects. A zero in the mask means either "the ray misses the object" or "the ray is blocked by something in front", and the mask cannot tell these apart. Training on it treats the hidden part as empty and yields a shattered object. Simply ignoring empty pixels leaves unsupervised floaters §3.3.

Method

Overview. Two pathways share one hybrid input. The scene branch encodes geometry and appearance of the whole scene and renders the background. The object branch encodes each object, conditioned on its activation code, and renders that object alone with everything else removed §3.1 fig. 2.

Hybrid embedding. For each sample point, the model interpolates a scene voxel feature from the eight nearest vertices and applies positional encoding to that feature and to the coordinate. Ray direction is encoded separately. The scene branch outputs density and colour from this embedding and the direction. The object branch adds an encoded object voxel feature and the object code §3.2.

Object supervision. For each ray, one object k is picked as the target and its code is fed to the object branch. The object's colour and opacity are volume-rendered along the ray eq. 1. The loss has two terms: colour is fitted only where the mask is set, and opacity is fitted everywhere, with a balance weight between mask and background pixels eq. 2 §3.3.

Scene-guided occlusion identification. Samples for the object branch are drawn with a distribution taken from the scene branch's transmittance. This reduces samples inside occluded regions §3.3.

3D guard mask. The scene depth is rendered online and pushed forward by a small distance eps along the camera direction. Object samples beyond that pushed depth are removed from the object branch's training, except in the visible instance region. Removing them blocks gradients into occluded space, while visible pixels are still supervised §3.3 fig. 3.

# Object-compositional training step  [§3.2] [§3.3] [§3.4]
# Input:  rays, colours C, instance masks M, balance weights w, codes L
# Output: scalar loss L_obj + L_scn  [eq. 2] [eq. 4]

def training_step(rays, colours, masks, weights, codes):
    # Pick one object k per ray; its code conditions the object branch  [§3.3]
    object_ids = random_object_per_ray(rays)            # (N_r,)
    object_code = codes[object_ids]                     # (N_r, C_code)

    # Scene branch: samples, density and colour along each ray  [§3.2]
    samples = sample_points(rays)                       # (N_r, N, 3)
    scene_sigma, scene_colour = scene_branch(samples)

    # Scene guidance: draw object samples from scene transmittance  [§3.3]
    object_samples = resample_with_scene_weights(scene_sigma, samples)

    # 3D guard mask: beyond the pushed scene depth, no object gradient  [§3.3]
    scene_depth = render_depth(scene_sigma, samples)
    guard = is_beyond(object_samples, scene_depth + eps)
    # Visible target pixels are exempt from the guard  [§3.3]
    guard = guard and not_on_target(masks, object_ids)

    object_sigma, object_colour = object_branch(object_samples, object_code)
    object_sigma = where(guard, 0.0, object_sigma)

    # Rendered object colour and opacity along the ray  [eq. 1]
    object_rgb, object_opacity = volume_render(object_sigma, object_colour)

    # Colour loss on mask pixels; opacity loss on valid pixels  [eq. 2]
    mask_k = masks[object_ids]
    colour_term = mask_k * squared_error(object_rgb, colours)
    opacity_term = weights[object_ids] * squared_error(object_opacity, mask_k)
    loss_obj = lambda1 * colour_term + lambda2 * opacity_term

    # Scene branch: plain colour loss over all rays  [§3.4]
    loss_scn = squared_error(scene_rgb(scene_colour, samples), colours)
    return loss_obj + loss_scn                          # [eq. 4]

Joint optimisation. Scene and object branches are trained together with the summed loss §3.4 eq. 4.

Editable rendering. Each object is rendered by switching its code. The background is rendered from the scene branch with the target region pruned. Each object is then transformed, and all K+1 sample sets are composited by distance along the ray §3.5 eq. 5. Rendering therefore evaluates K+1 times as many samples as a single-branch render.

# Editable rendering  [§3.5] [eq. 5]
# Input:  camera ray, object transforms, codes; K objects
# Output: pixel colour

def render_edited_pixel(ray, transforms, codes):
    # Background from the scene branch, target region pruned  [§3.5]
    samples = [scene_samples_outside_targets(ray)]
    for k in range(K):
        # Each object is shifted with its own transform  [§3.5]
        samples.append(object_samples(ray, transforms[k], codes[k]))
    # Sort all samples by depth, then composite  [eq. 5]
    merged = sort_by_depth(concatenate(samples))
    return volume_render(merged)

Hyperparameters

The paper's main text gives few numbers. The LaTeX source contains no appendix, so the values below that come from code are defaults or per-scene configs, not values the paper states. Read them with that in mind.

Results

Author-reported. Scene-branch comparison on ToyDesk and ScanNet, PSNR, SSIM and LPIPS tab. 1 §4.3. "Sparse Voxel*" is the authors' own re-implementation, because NSVF released no ScanNet training code §4.3. NPCR is not evaluated on ToyDesk because it needs sensor depth §4.2 tab. 1.

Ablation of the object branch on ScanNet 0033 and 0038, ten test views each, with the background masked out tab. 2 §4.6:

Sweep of eps for the 3D guard mask tab. 3 §4.6:

Qualitative results: single-object renders against a bounding-box crop and a mesh-masked point sampling fig. 6 §4.4; editing by rotating, moving and duplicating ToyDesk objects fig. 7 §4.5; editing against NPCR on ScanNet fig. 8 §4.5; the effect of the two occlusion fixes fig. 9 §4.6; and rendered masks that are smoother than the rough input masks fig. 10 §4.7.

Paper vs code

Repository: https://github.com/zju3dv/object_nerf, at code_commit above. It is linked from the paper's footnote project page, and its owner is the Zhejiang University 3D vision group, to which the authors Hujun Bao and Guofeng Zhang belong. Licence: MIT LICENSE.

Discrepancies, each checked against the code:

• Colour loss scale. The paper sums squared L2 error over rays §3.4. Code uses element-wise mean squared error over rays and the three channels, so the value is divided by 3 and normalised by the count of valid entries losses.py.
• Opacity weight. The paper leaves lambda2 unstated. The code's value is 1.0 by default and 10 or 100 in several scene configs default_conf.yml scannet_base_0038.yml toy_desk_1.yml.
• Object weights. The paper calls w(r) a balanced weight §3.3. The code uses fixed values, foreground 1.0 and background 0.05, from the default config default_conf.yml image_utils.py.
• Depth losses. The paper adds a depth loss only for ScanNet, to all compared methods §4.2. The code has depth terms for the scene and the object branch, both weighted 0.1, which apply whenever depth targets are present losses.py default_conf.yml default_conf.yml. The object-branch depth term is not in the paper.
• Guard mask mechanism. The paper says "prune ray samples" §3.3. The code instead sets object alpha to zero beyond the scene depth plus threshold, and skips that for pixels of the target and of background instances rendering.py generic_dataset.py. The background exemption is not described in the paper.
• Guard threshold units. The paper states eps = 0.05 without units tab. 3. The code divides the config value by scale_factor before using it train.py, so the effective value in scaled coordinates is 0.025 for ScanNet 0038 (scale 2.0). ToyDesk configs disable the guard entirely toy_desk_1.yml toy_desk_2.yml.
• Scene guidance. The paper describes biased sampling from scene transmittance §3.3. The code does this only when a fine pass exists (N_importance > 0), using the coarse scene weights rendering.py. There is no separate flag for it.
• Object branch input. The object branch receives the scene's embedded coordinate, the object voxel feature and the object code nerf_model.py. This matches §3.2.
• Object sampling. One object is drawn per ray at random in the training set generic_dataset.py. This matches §3.3.
• Object-branch last interval. The object branch sets the last sample interval to zero, while the scene branch uses a very large one rendering.py. The paper does not say how the last sample is handled.
• Object background. Object renders always add a white background, whereas the scene branch uses the dataset setting rendering.py. The paper does not state this.
• Voxel features. The scene voxel feature is trilinearly interpolated from 8 corners embedding_helper.py, as the paper says §3.2. The code also concatenates a 10-frequency encoding of xyz to it embedding_helper.py, which the paper does not mention.
• Inference noise. Sigma noise and stratified jitter are applied at evaluation as well as training rendering.py rendering.py. Validation output is therefore not deterministic.
• Mask border. The code crops a 20-pixel border from each image as invalid generic_dataset.py. The paper does not mention it.

Pitfalls

• Appendix is not in the source. The main text defers dataset details and more results to a supplement, and the LaTeX source contains no appendix. Lambdas, network sizes, sampling counts, learning rate and schedule are not stated in the paper. The code supplies them, but it is not the paper's reported run.
• eps units. Use the code's convention, scaled by scale_factor, when you reproduce the ablation, and say so. The paper's 0.05 is not self-describing.
• Occlusion depends on the coarse scene depth. The guard uses the coarse pass's depth rendering.py. The fine pass therefore does not refine the guard.
• Editing cost. Each rendered pixel needs K+1 sample sets for K objects §3.5 eq. 5. Latency grows with the number of objects, and the paper does not give timings.
• Depth is optional but used. §4.2 says depth is not needed, but the ScanNet comparison adds depth loss, and the code applies it when depth is present. Compare methods with the same depth setting.
• Point cloud for voxel initialisation. Both datasets need a point cloud for the voxel grid: SfM points on ToyDesk, and the NPCR point clouds on ScanNet §4.2. Without them the voxel features cannot be built.
• Rough masks. The ScanNet masks are described as fairly rough §4.1. The paper reports that the rendered masks become smoother than the inputs §4.7, but gives no quantitative mask-quality measure.
• Sparse Voxel is not NSVF. The authors' "Sparse Voxel*" is their own re-implementation, and NSVF released no ScanNet training code §4.3. Do not read those rows as NSVF numbers.
• Few results per scene. The ablation uses two ScanNet scenes and ten test views each §4.6. Treat the differences as indicative.
• Frame selection thresholds (40 degrees, 3 metres) follow the paper for ScanNet, but the ToyDesk config 1 disables the observation check toy_desk_1.yml.

Lineage

• Builds on NeRF §2 and NSVF's sparse voxel features §2, both named by the paper without arXiv ids here. The scene branch uses voxel features and positional encodings in that tradition §3.2.
• Compared against NPCR (neural point clouds) §4.3, NeRF and sparse voxels tab. 1.
• Related prior work: OSF, a per-object field that needs a separate capture of each object §1 §2.
• Superseded by: not known from this wiki.