JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields

In plain words

A 3D scan of a room is a cloud of points. For each point, the method has to say two things: what kind of thing it sits on (a chair, a wall) and which individual object it belongs to (this chair, not the one next to it). Most earlier work handled these as separate jobs, or found objects only after the classes were known.

The method trains one network with two outputs. One output guesses the class of each point. The other maps each point to a short list of numbers, arranged so that points on the same object land close together and points on different objects land apart. Clustering those numbers gives a first guess at the objects. A final step treats the points as a graph in which every pair of points is linked, with stronger links between points that look alike. It then adjusts the class and object guesses together so that they agree: a point's class should match the class of its object, and similar points should share a label. That joint adjustment is the paper's main idea. It matters for anyone who needs to know both what a 3D scene contains and where each separate item is.

TL;DR

• Joint semantic and instance segmentation of 3D point clouds, with one PointNet-style network that outputs class scores and point embeddings §3.1.
• The labels are refined by a multi-value conditional random field, solved with mean-field inference §3.2 §3.3.
• Author-reported: S3DIS semantic mAcc 87.4 (best of the compared methods) and instance mAP@0.5 36.3, below SGPN's 54.4 tab. 2 tab. 3.
• The official PyTorch code covers S3DIS only, and its CRF and evaluation differ from the paper's equations in several places (see Paper vs code).

Problem

Semantic segmentation gives a class to every point. Instance segmentation groups points into objects. The paper argues that the two are mutually dependent, and that treating instance segmentation as a post-processing step of semantic segmentation ignores that §1. Scenes are large and noisy, so the method works on overlapping windows of a scan rather than on the whole scene at once §4.1.

The paper assumes the input is a reconstructed point cloud with 3D location, normal and colour for each vertex §3.2. For S3DIS, which has no normals, the instance pairwise term is simplified to location and colour only §4.2.

Method

Overview. The scene is scanned with overlapping 3D windows. Each window goes through the multi-task network (MT-PNet), which outputs a class and an embedding for every point. Embeddings from all windows are merged, and the joint optimisation runs over the whole scene. A final non-maximal suppression step produces the output §3 §4.1 fig. 1.

3.1 Multi-task pointwise network. The backbone is PointNet-style. The network computes a per-point feature map, then splits into two heads §3.1 fig. 2. The semantic head predicts class scores; the embedding head outputs a vector per point. The training loss is the sum of the two branch losses eq. 1:

• Prediction loss: cross-entropy over classes §3.1.
• Embedding loss: a discriminative loss with three terms eq. 2. The pull term eq. 3 draws each point towards its instance centroid, with a hinge at margin δ_v. The push term eq. 4 pushes distinct centroids apart, with a hinge at margin 2·δ_d. The regularisation term eq. 5 draws centroids towards the origin. The weights are α = β = 1 and γ = 0.001 §3.1.

With δ_d > 2·δ_v, each point ends up closer to its own centroid than to any other centroid §3.1.

3.2 Multi-value conditional random field. Each point is a graph node holding two labels at once: a semantic label from the set S and an instance label from a set I that is not known in advance §3.2. The energy sums the following terms eq. 6:

• Semantic unary eq. 7: the negative log-probability of the class from the network.
• Semantic pairwise eq. 8: a Gaussian on the difference of the two points' class probabilities, weighted by a Potts-style compatibility ω eq. 9. ω is −1 when the labels agree and +1 otherwise.
• Instance unary eq. 10: the negative log of a Gaussian density of the point's embedding under the mean and covariance of its current instance, minus the log of the instance's size. The size term favours large instances, which the paper says removes tiny noisy instances §3.2.
• Instance pairwise eq. 11: the same Potts-style weight ω times a Gaussian over location, normal and colour, with one bandwidth per feature.
• Consistency term eq. 12: for each instance i and class s, a term based on the entropy of the class histogram h_i of that instance, so that a consistent labelling of an instance is favoured.

The paper stresses that this differs from higher-order CRFs in which detections act as prior knowledge. Here both label sets are unknown and optimised together §3.2.

3.3 Variational inference. The paper uses mean-field inference. The variational distribution first factorises the semantic and instance labels independently eq. 13. An auxiliary per-point term m_j is the entropy term of its instance, divided by the instance size eq. 14. The text rewrites the consistency sum as the sum of m_j over all points eq. 15. Mean-field updates follow, one for the semantic distribution eq. 16 and one for the instance distribution eq. 17. Gaussian pairwise terms are evaluated with fast filtering on downsampled distributions, following the DenseCRF approach, and truncated Gaussians can approximate the filters further §3.3.

Training step (the paper has no algorithm box; this is composed from §3.1 and eq. 1-eq. 5):

# Training loss for one window  [§3.1] [eq. 1]
# Input:  logits (N, C) log-probabilities; embeddings (N, d)
#         labels (N,) semantic ids; instances (N,) instance ids
# Output: scalar loss

def training_loss(logits, embeddings, labels, instances):
    # Cross-entropy on the semantic branch  [§3.1]
    loss_prediction = cross_entropy(logits, labels)
    # Discriminative loss on the embedding branch  [§3.1] [eq. 2]
    loss_embedding = embedding_loss(embeddings, instances)
    return loss_prediction + loss_embedding

# Discriminative embedding loss  [§3.1] [eq. 2]-[eq. 5]
# Weights: alpha = beta = 1, gamma = 0.001  [§3.1]
# Input:  embeddings (N, d); instances (N,) ids of K instances
# Output: scalar loss

def embedding_loss(embeddings, instances, delta_v, delta_d):
    # Centroid mu_k of each instance  [§3.1] [eq. 3]
    centroids = instance_means(embeddings, instances)  # (K, d)
    # Pull: squared hinge at margin delta_v, averaged within each
    # instance first, then over instances  [§3.1] [eq. 3]
    pull = pull_loss(embeddings, instances, centroids, delta_v)
    # Push: squared hinge at margin 2 * delta_d over every ordered
    # pair of distinct centroids  [§3.1] [eq. 4]
    push = push_loss(centroids, 2.0 * delta_d)
    # Regularisation: mean norm of the centroids  [§3.1] [eq. 5]
    reg = mean(norm(centroids, axis=1))
    return alpha * pull + beta * push + gamma * reg

Mean-field loop as printed in the paper §3.3 eq. 13-eq. 17:

# Mean-field loop for the MV-CRF  [§3.3] [eq. 13]-[eq. 17]
# Input:  Q_S (N, |S|), Q_I (N, |I|); unary terms; pairwise messages
# Output: semantic labels (N,), instance labels (N,)

def mean_field(Q_S, Q_I, unary_S, unary_I, num_iters):
    # Number of iterations is not stated in the paper  [§3.3]
    for step in range(num_iters):
        # Per-point term m_j from the current instance histograms  [eq. 14]
        m = entropy_terms(Q_S, Q_I)
        # Semantic update; Z_j normalises over labels  [eq. 16]
        Q_S = normalise(exp(-unary_S - message_S(Q_S) - m))
        # Instance update  [eq. 17]
        Q_I = normalise(exp(-unary_I - message_I(Q_I) - m))
    return argmax(Q_S), argmax(Q_I)

Hyperparameters

Values in the paper are stated in the text unless marked otherwise. Code values come from the official repository at the commit in the frontmatter.

Results

Author-reported throughout. Setup: S3DIS with 13 classes, SceneNN with the NYU-D v2 category set (40 classes; the tables show a subset). Per-class accuracy and micro-mean accuracy for semantic segmentation; AP at IoU 0.5 for instance segmentation §4.3. Baselines: PointNet, Pointwise CNN and SEGCloud for semantic; SGPN and Armeni et al. for instance. Armeni et al. report on 3D bounding boxes, not point clouds tab. 3 caption.

Ablation on S3DIS tab. 1

The unary-only semantic row equals the MT-PNet row in tab. 2. The unary-only instance row is the mean-shift result on the embeddings.

S3DIS semantic, per-class accuracy tab. 2. The printed table omits beam and column. The source data file lists them, and those values are given in brackets in the note below.

The source data for this table (figures/csv/s3dis/semantic.csv in the LaTeX source, tab. 2) also holds the beam and column values, which are not printed. The other values match the printed table. Column keys: Ceil. is ceiling and Col. is column; the instance table uses the same keys.

S3DIS instance, mAP@0.5 per class tab. 3. The printed table omits beam and column; the source data lists them as: Armeni et al. 66.7 and 91.8; SGPN 78.0 and 60.7; MT-PNet 0 and 0; MV-CRF 0 and 0.

SceneNN semantic, per-class accuracy tab. 4. The printed table omits door and window; the values in the source data are shown here.

The source row is labelled "Ours (MLS-CRF)", while the text calls the method MV-CRF tab. 4. No overall mAcc is reported for SceneNN.

SceneNN instance, AP@0.5 per class tab. 5. The metric is not labelled in the table; the text defines it as AP at IoU 0.5 §4.3. No overall mean is reported.

No SceneNN baseline is reported for instance segmentation. The MT-PNet ablation with the embedding branch disabled is described in text only: it matches vanilla PointNet on semantic segmentation §4.3. No numbers are given.

Paper vs code

Official code: pqhieu/jsis3d (MIT; copyright Quang-Hieu Pham, the first author), commit b6b8aaecf120e9e80910f897b50094885ef7e307. The repository is S3DIS only; there is no SceneNN loader or config. It needs a separate C++ build for the CRF step.

• Embedding pull normalisation. Paper averages within each instance, then across instances eq. 3. Code divides the summed hinge over all points by the total point count in the window, so large instances carry more weight discriminative.py.
• Learning-rate schedule. Paper halves the rate every 50 epochs §4.1. Code trains 40 epochs and halves the rate every 20 scheduler steps; the scheduler is stepped at the start of each epoch, so the first halving takes effect at the start of zero-based epoch 19 s3dis.json s3dis.json train.py train.py.
• Unspecified constants. Embedding size 32 and push/pull margins are code-only s3dis.json s3dis.json. Momentum 0.9 and weight decay 5e-4 are code-only s3dis.json s3dis.json.
• Semantic unary. Paper uses the network's log-probability eq. 7. The CRF uses only the argmax label, with a fixed confidence of 0.5 for that label and the rest spread evenly main.cpp pred.py.
• Instance unary. Paper uses the Gaussian density of the embedding with an instance covariance and a size term eq. 10. Code uses the MeanShift label with the same fixed confidence, and no density or size term main.cpp.
• Pairwise terms. Paper's semantic pairwise uses classification scores eq. 8; paper's instance pairwise uses location, normal and colour eq. 11. Code uses one DenseCRF Potts kernel on location (σ 0.2) and one on location plus colour, both shared by the semantic and instance CRFs, with a spatial Potts term of 3.0 that the paper does not mention main.cpp.
• Consistency term. Paper's term is the entropy of the instance's class histogram eq. 12. Code uses a leave-one-out soft frequency of the class within the MAP instance, weighted by 20 higherorder.cpp main.cpp.
• Coupling in inference. Paper's mean-field updates run on both label sets jointly eq. 13-eq. 17. Code runs two separate CRFs for 5 alternating iterations. Each iteration updates the instance CRF first, then sets the semantic CRF's consistency term from the current instance MAP. The instance update never sees the semantic labels main.cpp.
• Confidence and AP. Paper scores each instance with a confidence f_j eq. 18 and reports AP. Code does not compute f_j. eval.py counts a proposal as a true positive when its IoU with any ground-truth instance is at least 0.5, with no one-to-one matching and no ranking, and it reports per-class precision and recall eval.py eval.py. The repository does not produce the AP numbers in the tables.
• Accuracy names. Paper's footnote says micro-mean accuracy equals overall accuracy, and the table column is mAcc §4.3. Code prints overall accuracy as one number and "mean accuracy" as the average of per-class accuracies eval.py eval.py.
• Test split. Paper says it followed the original split of Armeni et al. §4.2. The README says the paper's S3DIS numbers are on Area 6, and the repository keeps Area 5 as the test set README.md. The test list has 68 rooms from Area 5 test.txt.
• Window merging and NMS. Paper says it uses SGPN's BlockMerging and a final non-maximal suppression step §4.1. Code has its own block_merge with fixed constants merge.py. No suppression step is in the repository.
• Architecture. Paper calls the backbone a feed-forward PointNet §3.1. Code adds input and feature T-Nets (3×3 and 64×64 transforms), then a 1024-wide max pool and a 128-wide shared feature pointnet.py pointnet.py.
• Window sampling. Paper says each window holds about 4,096 points §4.1. Code's room_to_blocks samples a fixed count from 1.0 m columns with stride 0.5 m and a 100-point minimum prepare_h5.py. The count is not defined in the script; see Pitfalls.
• Table naming. Paper's text calls the method MV-CRF, while the SceneNN source tables call it MLS-CRF tab. 4 tab. 5.
• Other checks. Mean-shift bandwidth 1.5 matches s3dis.json pred.py. The 13-class CRF matches main.cpp. Cross-entropy and the α, β, γ weights match train.py discriminative.py. The centroid push margin 2·δ_d matches discriminative.py. The regularisation term matches discriminative.py.

Pitfalls

• Eqs. 16-17 as printed have no semantic-instance coupling. The term −m_j is the same for every label of point j, so it cancels in the normalising constant Z_j. The updates as written do not depend on the consistency term. The official code couples the two labels by hard MAP instances instead (see Paper vs code). An implementer following the printed equations will not reproduce the paper's coupling; decide which one you want and note it.
• Density in the instance unary. Eq. 10 needs the inverse and determinant of a d×d covariance per instance (d = 32 in the code). The paper says nothing about regularising the covariance or about instances with fewer than d points. Compute the log-density in log space.
• Pairwise terms inside label sums. Eqs. 8 and 11 write φ(l_j, l_k), but the mean-field sums in eqs. 16-17 need the value for each pair of labels. The paper does not say how the expectation is computed.
• Pull weighting. Per-point averaging (code) gives large instances more weight than per-instance averaging (paper eq. 3). Results may shift on rooms with uneven instance sizes.
• Over-segmentation. The paper says mean shift over-segments, that the cluster count may be much larger than the true count, and that low bandwidth hurts §4.1. Plan for a merge step; the code's merge is written for S3DIS blocks.
• Instance accuracy is not a clear win. The text says MV-CRF clearly improves some categories, such as floor, sofa, board and clutter §4.3. Overall mAP is 36.3 against SGPN's 54.4 tab. 3. Door is weak and the paper ties that to low semantic accuracy §4.3.
• Reproducing the S3DIS numbers. The split in the repository (Area 5) differs from what the README says the paper used (Area 6). Check which split a reported number used before comparing.
• Metric definitions. The code's AP-like number is not AP (see Paper vs code). The code's precision and recall count duplicate proposals on the same instance as true positives, so they are not comparable to the paper's AP.
• Data preparation does not run as shipped. scripts/prepare_h5.py uses argparse without importing it prepare_h5.py and passes an undefined num_points prepare_h5.py. The paper's 4,096 points per window cannot be confirmed from the code. The preprocessed HDF5 files are downloaded separately and were not checked.
• CRF hard-codes 13 classes main.cpp. Changing the dataset means editing the C++ file.
• Compute. The CRF runs on the whole scene after window merging, not per window §4.1 pred.py. The paper gives no runtime or memory figures for inference.
• Unstated learning details. Warm-up, weight initialisation, batch size and epoch count are not stated in the paper §4.1. The code's values are in the Hyperparameters table.

Lineage

• Builds on: PointNet (Qi et al., CVPR 2017) for the backbone §3.1; the discriminative embedding loss of de Brabandere et al. (2017) §3.1; the DenseCRF work of Krähenbühl and Koltun (NIPS 2011) for the mean-field approach §3.3; mean shift (Comaniciu and Meer) for the initial clustering §4.1; SGPN's block merging §4.1. None of these has a card here, so none is linked.
• Compared against: PointNet, Pointwise CNN and SEGCloud (semantic); SGPN and Armeni et al. (instance) tab. 2 tab. 3 tab. 4 tab. 5.
• Superseded by: not stated in this wiki.