Panoptic Lifting for 3D Scene Understanding with Neural Fields

In plain words

Imagine walking around a room with a camera and getting a 2D labelling tool that marks every pixel as wall, chair or sofa, and which chair is which. The labels are good but not perfect, and the same chair can get a different ID in every photo. Panoptic Lifting builds one 3D model of the room from those photos and their noisy labels. Because the model has to explain all views with one shared geometry, the labels that disagree between views get averaged out.

The model also draws the scene from new camera positions, with a class and an object ID for each pixel that stay the same across views. Two tricks keep it honest. The labels are first made less overconfident by running the labeller on altered copies of each photo and merging the results. The model's geometry is also shielded from the labels, so a bad label cannot bend the 3D shape to match it.

TL;DR

• Input: posed RGB images plus 2D panoptic masks from a pretrained Mask2Former; no ground-truth 3D labels are used for training §4.
• Model: a TensoRF density and colour field, plus two small MLPs for class distribution and an instance-ID distribution, rendered per ray §3.2.
• Key pieces: rendered probabilities (not logits), segment consistency loss, injective ID assignment by linear assignment, bounded segmentation output, and stopped gradients from semantics to density.
• Author-reported ScanNet results: 65.2 mIoU and 58.9 PQ-scene, against 46.7 mIoU for Mask2Former on the same images tab. 1.
• Cost: about 450k iterations (about 10 hours on one A6000 GPU) and an offline pipeline with pose estimation and label generation §3.3 §4.1.3.

Problem

Per-frame 2D panoptic labels from a pretrained network are noisy and disagree between views. Object IDs are not preserved across frames, so a 2D model cannot give consistent object identities in a 3D scene. Earlier neural-field methods either need 3D inputs or fit semantic logits that can disagree with the geometry, which the authors show lets the model cheat on inconsistent labels §3.3 §4.1.1.

Method

Inputs. Each training ray carries an RGB colour, a class distribution from test-time augmentation, a confidence, and, for thing pixels only, a 2D instance ID from the labeller §3.1. The test-time augmentation procedure runs the labeller on several transformed copies of each image, takes the outputs before panoptic fusion, groups overlapping candidate segments by a soft-IoU graph, averages masks and probabilities per group, and fuses the result into one panoptic map with a per-pixel confidence app. TTA. The paper does not state the augmentation list's exact per-transform parameters.

Field. The field Φ(x, d) gives density σ, colour c, a class distribution κ over classes K, and a distribution π over surrogate instance identifiers J §3.2. The 3D object identity is the pair (class, surrogate), with probability κ(k)π(j) §3.2.

Rendering. Colour, κ and π are rendered along each ray with the standard volume rendering integral and transmittance eq. 1 eq. 2. Rendering is done on the probabilities themselves, not on logits, so that unbounded logits cannot create semantic signal in empty space §3.3, "Logits vs. Probabilities".

Novel-view panoptic output. For a pixel, take the most probable class k* from κ. If k* is a thing class, take the most probable surrogate j* from π. The pair (k*, j*) is the instance ID and is consistent across views §3.2.

Losses. Ray batches are a random set R_S over the scene for colour and semantics, plus a batch R_I from a single image for instance and consistency terms §3.3, "Training objective". The terms are:

1. Colour: squared error between rendered and ground-truth colour eq. 3.
2. Semantic: cross entropy of the rendered κ against the test-time-augmented distribution, weighted by the ray's confidence eq. 4.
3. Instance: negative log-probability of π for the ID assigned to each ray, weighted by confidence. The assignment is an injective map from 2D instances to surrogate IDs, chosen to maximise the summed mean probability of each instance's rays, solved as a linear assignment eq. 5 eq. 6.
4. Segment consistency: within an image, rays are grouped by their 2D segment. Each group gets one target class (the most probable class over the confidence-weighted group sum). Each ray's log-probability of that class is penalised, weighted by confidence eq. 7.

Gradient stopping. Gradients from the semantic and instance branches are blocked before the density, so label noise cannot change the geometry §3.3.

Weights. The total objective is written only in a commented-out block of the source. The weights stated in the text are λ_con = 1.35 and λ_ins = λ_sem = λ_RGB = 1 §3.3, "Implementation Details".

# Panoptic Lifting: per-ray panoptic output  [§3.2]
# Input:  rendered class probabilities kappa (K,), instance probs pi (J,)
# Output: class id, instance id (or none for stuff)

def panoptic_label(kappa, pi, thing_classes):
    # Most probable class for this pixel  [§3.2]
    k_star = argmax(kappa)
    # Stuff pixels get no instance id  [§3.2]
    if k_star not in thing_classes:
        return k_star, None
    # Surrogate id from the instance distribution  [§3.2]
    j_star = argmax(pi)
    return k_star, j_star

# Injective instance assignment for one image  [§3.3] [eq. 6]
# Input:  pi_rays (R, J) per-ray instance probs; ids (R,) 2D instance ids
# Output: mapping from 2D instance id to surrogate id

def assign_instances(pi_rays, ids):
    # Cost is the negative mean probability of each surrogate per instance
    # (mean over the instance's rays)  [§3.3] [eq. 6]
    cost = -mean_over_rays_of_each_instance(pi_rays)  # (H, J)
    # Injective mapping that minimises total cost  [§3.3] [eq. 6]
    rows, cols = linear_sum_assignment(cost)
    return dict(zip(rows, cols))

Hyperparameters

Results

Author-reported. Setup: Mask2Former labels for all methods, held-out test views, with the scene-level PQ metric §4 tab. 1. Values are mIoU, PQ-scene and PSNR. A dash means the table does not report that cell. Values in brackets in the source table are gaps to the authors' method and are omitted here.

Source: tab. 1 (tables/results.tex). PNF + GT boxes is given ground-truth 3D detections and is not a fair comparison tab. 1 caption. The authors re-implemented PNF because it has no public code app. impl..

Conventional PQ on novel views (not scene-level), from the appendix: Mask2Former 50.52 / 50.10 / 43.6 and Panoptic Lifting 66.84 / 63.79 / 60.4 on HyperSim / Replica / ScanNet app. results. The appendix also reports SQ-scene and RQ-scene for DM-NeRF, PNF and this paper app. results.

Ablations on ScanNet tab. 2 §4.1.1. Each row removes components from the full method:

Rendering speed, from the appendix: 2048 rays take 13.1 ms here, against 119.7 ms (PNF), 66.5 ms (DM-NeRF) and 65.7 ms (Semantic-NeRF) on an RTX A6000 app. render.

Paper vs code

Searched: the paper's text, then the project page at nihalsid.github.io/panoptic-lifting, which links the GitHub repository nihalsid/panoptic-lifting and the nihalsid GitHub profile. The page lists the paper's seven authors. No GitHub search was needed. The code is at commit b8ad4f5. The LICENSE file is CC BY-NC-SA 4.0 and the README says the majority is CC BY-NC, so the repository is cite-only here. Only the paper is used for the method text.

Differences found, each with both citations:

• Segment loss target. The paper's target is the most probable class of the confidence-weighted sum of rendered probabilities eq. 7. The training step computes an unweighted mean of log-probabilities per segment and takes its argmax train_panopli_tensorf.py. A weighted variant exists but is never called from training train_panopli_tensorf.py.
• Segment loss sum. As written, eq. 7 sums over all m segments of the batch for each ray. The code uses one target per ray, the target of the ray's own segment train_panopli_tensorf.py.
• Loss weights. The paper uses λ_con 1.35 and λ_sem 1 §3.3. The code uses a semantic weight of 0.1 and a segment weight of 0.12 panopli.yaml train_panopli_tensorf.py. The README examples override the segment weight per scene, to 0.5 for HyperSim and 0.75 for Replica README.md README.md.
• Instance weight. The paper's λ_ins is 1. The config defines lambda_instances as 0.1, but the code never reads it. The instance loss is applied with its own optimiser and no weight train_panopli_tensorf.py.
• Unstated regularisers. The code adds total-variation terms on density, appearance and semantics (weights 0.1, 0.01, 0.02) and a distortion loss weighted 0.005 and ramped by 1 − exp(−0.25 · epoch) panopli.yaml panopli.yaml train_panopli_tensorf.py train_panopli_tensorf.py. The paper names none of them.
• Instance head. The paper's instance head is a 3-layer MLP with at most 50 IDs app. impl.. The code builds 4 linear layers with 25 output IDs by default tensoRF.py tensoRF.py panopli.yaml.
• Instance rendering. The paper renders distributions for both semantics and instances and argues that bounded probabilities avoid cheating §3.3. The code renders raw instance logits, since the instance head uses an identity output activation, and only applies softmax inside the loss tensoRF.py tensoRF.py panopli_tensoRF_renderer.py panopli_tensoRF_renderer.py. Only the semantic branch renders probabilities.
• Semantic normalisation. The code divides the rendered class probabilities by their sum and takes the log before the loss panopli_tensoRF_renderer.py. The paper does not mention this step.
• Instance loss gating. The code computes the Hungarian assignment but backpropagates only if the assigned labels differ from the current argmax. Correct predictions are not reinforced train_panopli_tensorf.py. The paper has no such gate.
• Class 0. Class 0 gets zero weight in the semantic cross entropy train_panopli_tensorf.py.
• Density activation. Density is softplus of the field output minus 10, and rays are scaled by 25 in the alpha computation tensoRF.py panopli_tensoRF_renderer.py panopli_tensoRF_renderer.py. The paper states neither.
• Background. During training, a random white background is applied to half the rays panopli_tensoRF_renderer.py. Not in the paper.
• Schedules. The paper gives 450k iterations. The code uses epochs: grid upscaling at epochs 1 to 4, AABB resets at epochs 1 to 3, instance loss from epoch 3, segment loss from epoch 6, and a learning-rate step (× 0.5) at epochs 9 and 10 panopli.yaml panopli.yaml [code: trainer/init.py#L146] train_panopli_tensorf.py. The README example for HyperSim uses 34 epochs and different gates README.md. No conversion from iterations to epochs is given.
• Instance MLP weight decay. 1e-3 tensoRF.py.
• Test-time augmentation. The augmentation and fusion code is in a separate fork named in the README, not in this repository README.md. It was not checked.
• Confirmed matches. The grid range 128³ to 192³, learning rates of 5e-4 and 1e-2 (code: 20 × 5e-4), batch of 2048 rays, density and appearance components, the semantic MLP shape, and the stopped gradient from semantics to density all match train_panopli_tensorf.py panopli_tensoRF_renderer.py.

Pitfalls

• Test-time augmentation is load-bearing. Without it the ablation loses 2.1 mIoU and 3.7 PQ-scene points tab. 2. Reproducing it requires the separate Mask2Former fork and one labeller pass per augmentation. The appendix lists nine transform types, with no per-transform parameters, and gives θ = 0.5 only as an example threshold app. TTA.
• Stop-gradient position matters. Blocking the gradient to density is what keeps geometry clean tab. 2 §3.3. The code applies this block to the semantic and instance maps only; the colour map is not detached panopli_tensoRF_renderer.py.
• Probabilities, not logits, for semantics. Rendering logits lets large logits in empty space satisfy the loss §3.3. The code's instance branch renders logits, so the paper's argument does not carry to the instance head.
• Instance IDs are truncated. The assignment keeps only the first k instance labels (sorted) and gives the rest no assignment train_panopli_tensorf.py. With k = 25 or 50, scenes with many thing instances per image lose labels.
• Linear assignment costs. Per-image assignment is solved with SciPy linear_sum_assignment on a cost built from mean probabilities; NaNs are replaced by zero train_panopli_tensorf.py.
• Per-ray loss scale. The RGB loss and TV terms are summed under λ_RGB, so the effective balance with the semantic term depends on the TV and distortion weights the paper omits train_panopli_tensorf.py.
• Unfair baselines. PNF with ground-truth boxes uses 3D information the method does not get. The authors' own PNF is a re-implementation without its prior-based initialisation app. impl..
• Poses matter for baselines. The PNF detector needed ScanNet-like pose scale and centring, which the authors computed from ground-truth or NeRF-estimated depth app. impl.. Pose scale is a hidden cost for any method that reuses such detectors.
• Offline pipeline. Pose estimation, label generation, and optimisation are all offline §4.1.3. Reported timings cover rendering only app. render.
• Mismatch in the headline. The abstract gives 8.4% for HyperSim, while the table gives 60.1 − 51.6 = 8.5 points. The other two datasets match.
• Not every PSNR is best. On Replica, PNF + GT boxes (31.6) and PNF (29.8) have higher PSNR than the method (29.6) tab. 1. The paper's claim of no loss in view synthesis quality holds only loosely.
• Scene-level PQ needs tiling. The metric can be computed by tiling each scene's predictions and ground truth and running standard PQ §4, footnote 1.
• Label set limits. The method is limited to classes the 2D model was trained on §4.1.3.
• Evaluation labels are mapped. COCO classes and ground truth from other datasets are mapped to shared ScanNet classes; the mapping changes scores §4 app. data.

Lineage

• Builds on: TensoRF for the radiance and density grids [Chen2022ECCV, cited in the paper; not a card in this wiki].
• Compared against: Panoptic Neural Fields (re-implemented by the authors, with a multi-view detector for boxes), SemanticNeRF and DM-NeRF (both cited in the paper; not cards in this wiki), and Mask2Former as the 2D labeller §4.1.
• Labeller: Mask2Former [cheng2022masked], with test-time augmentation described in the appendix app. TTA.