ConceptFusion: Open-set Multimodal 3D Mapping

In plain words

A robot that builds a 3D map of a room usually stores shapes and colours. A map of this kind can only answer questions about the object types it was trained to recognise. ConceptFusion stores, for each point of the map, a numeric description of what that point looks like in the language of a large pretrained image model. The map can then be searched with a word, a photo, a sound or a click on the map, and it finds the points that match.

The hard part is that the pretrained image model describes whole pictures, not small regions. The authors combine the description of the whole picture with descriptions of many small cut-out regions, giving more weight to the whole picture for regions that resemble it. These per-pixel descriptions are then added to the 3D map in the same way depth and colour are.

The authors report that this works without any further training and keeps rare object names searchable. Their evaluation is on indoor, tabletop and driving scenes, and on two robots: a tabletop arm and a car.

TL;DR

• Each 3D map point stores a language-aligned feature vector, fused over frames with a confidence-weighted average §3.1 eq. 1 eq. 2.
• Per-pixel features are built zero-shot from a CLIP-style image embedding of the whole frame and of mask-cropped regions, combined with similarity weights §3.2 eq. 3 eq. 4 eq. 5 eq. 6.
• Queries from text, click, image or audio are cosine scores against the map, with no training §3.3.
• Author-reported: on UnCoCo structured text queries, 0.446 3D mIoU against 0.289 for supervised OpenSeg-3D tab. 1.
• The official repository is MIT-licensed. Its extraction script differs from the paper: SAM replaces Mask2Former and the uniqueness term is removed extract_conceptfusion_features.py README.md.

Problem

Maps that tag objects with semantic labels mostly work over a fixed label set chosen at training time. Maps can usually be queried only by class name or by text prompt §1. Foundation models such as CLIP cover open vocabularies but output one vector per image, not per pixel §1. Pixel-aligned models built on CLIP, such as LSeg and OpenSeg, are finetuned on small labelled datasets, and the authors report that this finetuning makes them forget rare concepts §1 §3.2.

The target is a map that is open-set, meaning concepts are supplied at query time, and multimodal, meaning it can be queried by text, images, audio and 3D clicks, all in one vector space §1 §3.

Method

Overview. The system takes posed RGB-D frames and fuses a per-pixel foundation-model feature into a point map, using the same surface-fusion pipeline as for depth and colour §1 §3.1. Pixel features are built from one image-level embedding, one embedding per instance-mask bounding box, and similarity weights §3.2. The map is then searched with cosine similarity to a query vector §3.3. Spatial relations are answered by a small set of functions over query results, with an LLM optionally parsing the language §3.4 app. A8.

Steps.

1. Frame preprocessing. Each frame gives a colour image, a depth image, a vertex map, a normal map and a camera pose §3.1. Points with noisy depth are dropped by the depth-fusion procedure of PointFusion §3.1 app. A3.
2. Class-agnostic masks. A universal instance-segmentation model proposes R masks per image. Mask2Former (100 proposals per image) or SAM is used §3 app. A4.
3. Global and local embeddings. The global embedding is the foundation model's embedding of the whole image. Each mask's bounding box is embedded the same way, giving R local embeddings §3.2.
4. Similarity weights. For each region, compute the cosine similarity of its local embedding to the global one eq. 3. Compute the average similarity of that local embedding to the other local embeddings, which measures uniqueness eq. 4. Combine both in a softmax across regions with temperature τ eq. 5.
5. Region feature. The region feature is a weighted mix of the global and local embeddings, with weight w_i on the global one eq. 6. The result is normalised and assigned to every pixel of that region §3.2.
6. Pixel feature. A pixel can lie in several regions. Its feature is the sum of the region features assigned to it, normalised once after accumulation §3.2.
7. Map fusion. Each pixel's feature is fused into the map point it projects to, using a confidence weight that falls off with radial distance from the camera centre §3.1 eq. 1 eq. 2. The fused vector per map point is the feature the queries use.
8. Querying. Each map point gets a score in [-1, 1], the cosine similarity between its fused vector and the query vector. Thresholding, non-maxima suppression and optional clustering then give 3D regions of interest §3.3. The query vector depends on the modality: CLIP text encoder for text; the clicked point's fused vector for clicks; CLIP image embedding for images; AudioCLIP embedding for audio §3.3.
9. Spatial queries. Five comparator functions take two queries (distance, left-of or right-of, on-top-of, under, plus containment and size in the appendix). An LLM maps a natural-language question onto these functions. Other experiments feed the raw text query to the CLIP text encoder §3.4 app. A8.

# Pixel-aligned features for one frame  [§3.2] [eq. 3-6]
# Input:  image (H, W, 3); R mask proposals with bounding boxes
# Output: pixel_features (H, W, d), normalised per pixel

def pixel_aligned_features(image, masks):
    # Whole-image embedding from the foundation model  [§3.2]
    global_feature = normalise(encode_image(image))
    # One embedding per proposal's bounding box  [§3.2]
    local_features = [
        normalise(encode_image(crop(image, box)))
        for box in bounding_boxes(masks)
    ]
    # Cosine similarity of each local to the global  [eq. 3]
    similarity = [dot(local, global_feature) for local in local_features]
    # Mean similarity to the other locals, i.e. uniqueness  [eq. 4]
    # Written as divided by R, not R-1, in the paper
    uniqueness = [
        sum(dot(local_i, local_j) for j != i) / R
        for i, local_i in enumerate(local_features)
    ]
    # Softmax over regions, temperature tau = 1  [eq. 5] [§3.2]
    logits = [(s + u) / tau for s, u in zip(similarity, uniqueness)]
    weights = softmax(logits)
    # Weighted mix of global and local, normalised  [eq. 6]
    region_features = [
        normalise(w * global_feature + (1 - w) * local)
        for w, local in zip(weights, local_features)
    ]
    # Sum region features onto their pixels, normalise once  [§3.2]
    pixel_sums = zeros(H, W, d)
    for region, feature in zip(masks, region_features):
        pixel_sums[region.pixels] += feature
    return normalise(pixel_sums)

# Fuse one pixel's feature into a map point  [§3.1] [eq. 1-2]
# Input:  point k with feature f_k and confidence c_k;
#         pixel feature f_uv and radial distance gamma of that pixel
# Output: updated f_k and c_k

def fuse_pixel(f_k, c_k, f_uv, gamma):
    # Confidence falls off with radial distance; sigma = 0.6  [§3.1]
    alpha = exp(-gamma ** 2 / (2 * sigma ** 2))
    # Confidence-weighted running average of the feature  [eq. 1]
    f_k = (c_k * f_k + alpha * f_uv) / (c_k + alpha)
    # Accumulate the confidence  [eq. 2]
    c_k = c_k + alpha
    return f_k, c_k

Hyperparameters

Results

Author-reported throughout. Setup: UnCoCo tabletop sequences (20 RGB-D sequences, 78 household objects, about 500K queries), with the first two sequences used for tuning and excluded §4.1 app. A7. Baselines are LSeg-3D and OpenSeg-3D (LSeg and OpenSeg features fused with the same pipeline) and MaskCLIP-3D §4. Metrics are 3D mIoU and the fraction of queries whose detection reaches IoU 0.15, 0.25 or 0.5 tab. 1.

Text queries, UnCoCo structured tab. 1

Text queries, UnCoCo unstructured (long free-form) tab. 2, averaged over 20 trials

Image queries, UnCoCo structured tab. 3, averaged over 3 trials

Audio queries, UnCoCo tab. 4

Open-set 2D segmentation, mAcc / f-mIoU tab. 5

Ablation, ScanNet tab. 6

Segmentation model, Replica tab. 8

Spatial reasoning, 100 queries over 5 ScanRefer scenes, 25 per type tab. 7

Qualitative only. Zero-shot tabletop rearrangement with a UR5e arm on novel objects, and text-driven navigation on a drive-by-wire car over a 4000 m² urban area §4.3 fig. 8 fig. 9. The paper reports no success rates for either. The LLM-planner demo in AI2-THOR is also qualitative §5 fig. 10.

Claimed margin. The abstract says the method outperforms supervised baselines "by more than 40% margin on 3D IoU". The table above gives 0.446 against 0.289 mIoU for the best supervised baseline, which is a difference of 0.157 in mIoU. The paper does not define the 40% figure §1 tab. 1.

Paper vs code

Checked against the official repository at 4457c1f718a5612c2efc57b93a2dc152fed8a03d, which the project page links. The repository's README lists its own deviations README.md.

• Mask proposals. Paper uses Mask2Former (100 proposals) or SAM §3. Code uses only SAM, with points_per_side=8 in the call, although the argument class default is 32 extract_conceptfusion_features.py extract_conceptfusion_features.py. The mask-area filter is declared but unused extract_conceptfusion_features.py.
• Uniqueness term removed. Paper includes the mean-similarity term in the weights eq. 4 eq. 5. Code uses only the global-to-local similarity in the softmax: softmax(similarity_scores) extract_conceptfusion_features.py. The README gives the same deviation README.md.
• No temperature. Paper has τ = 1 explicitly eq. 5. Code has no temperature term, which amounts to τ = 1 extract_conceptfusion_features.py.
• Normalisation. Paper normalises once after accumulating over regions §3.2. Code normalises each region's feature and then again after each accumulation step extract_conceptfusion_features.py.
• Global feature. Paper computes it from the foundation model on the full image §3.2. Code uses the OpenCLIP ViT-H-14 laion2b_s32b_b79k checkpoint extract_conceptfusion_features.py extract_conceptfusion_features.py.
• Odometry and fusion. Paper uses LeGO-LOAM for outdoor sequences app. A3. Code fuses with gradSLAM's PointFusion, using ground-truth odometry (odom="gt") and nearest-neighbour resizing of the feature map to image size run_feature_fusion_and_save_map.py run_feature_fusion_and_save_map.py.
• Fusion confidence. Paper gives α = exp(−γ²/2σ²) with σ = 0.6 eq. 1 §3.1. The repository does not contain this formula. It relies on gradSLAM's conceptfusion branch, which is not pinned here and not included in this checkout.
• Default feature model. Paper's pipeline is CLIP-based §3.2. The fusion script's default model_type is "vit" (DINO), and LSeg and DINO extractors are in models_common.py. The CLIP path is only in extract_conceptfusion_features.py run_feature_fusion_and_save_map.py models_common.py.
• LLM parser and 3D comparators. Paper has an LLM parser and five comparator modules §3.4 app. A8. The repository has no code for either. A grep for the function names and for the LLM API finds nothing; the only OpenAI mention is the optional CLIP install in the README.
• Queries. Paper describes image and audio queries §3.3. The repository has text and click demos only demo_text_query.py demo_click_query.py. Both demos rescale scores to [0, 1] and threshold them for display, with thresholds of 0.6 (text) and 0.8 (click) demo_text_query.py demo_click_query.py. The paper gives no threshold value.
• Datasets and evaluation. UnCoCo is not in the repository. The README lists its download as a TODO. There is no evaluation code for the mIoU or the detection metrics README.md §4.1.

Pitfalls

• γ is undefined in units. The paper calls γ the radial distance and in one place says it is normalised to the camera centre §3.1. σ = 0.6 gives very different confidences in metres than in normalised units. The implementation is in gradSLAM, outside this repository.
• Initial confidence and new points. The paper does not state the initial confidence of a new map point or how it is set for points that receive their first observation. Read the gradSLAM implementation before you rely on eq. 2 (c_k update) for new points.
• Uniqueness sign. Eq. 4 is named "uniqueness" in the text, but it is a mean similarity to the other regions, and eq. 5 adds it to the similarity to the global embedding. A region that resembles the others therefore gets more weight on the global embedding, the opposite of what the name suggests. Implement the formula as written.
• Uniqueness normalisation. Eq. 4 divides the sum over j ≠ i by R, not R − 1 eq. 4. Eq. 5 reuses the index i in its denominator sum; read it as a sum over regions. Implement as written, or follow the paper's text for R − 1. The paper does not say which.
• Several regions per pixel. The paper lets a pixel belong to several regions and normalises after accumulation §3.2. Normalising after every addition, as the code does, changes the scale of contributions. The repository's choice is the one that runs.
• Compute cost. Each frame needs one CLIP pass for the whole image and one per mask (100 with Mask2Former), which the paper reports as 10 to 15 s per image on an RTX 3090 §3. The authors list the memory cost of dense maps with millions of points, each carrying a high-dimensional concept vector, as the first limitation §6.
• Dimension and storage. The paper does not state the feature dimension. The repository's default ViT-H-14 gives 1024 values per point extract_conceptfusion_features.py.
• Spatial distances. The paper does not define how point sets for "how far" are formed or how their distance is computed, beyond the figure caption, which describes distance between centroids of the matched point sets §3.4 fig. 5. The appendix gives the LLM prompt only app. A8. Relation definitions such as "to the right of" are described only in words.
• Thresholds for detection. Thresholding, non-maxima suppression and clustering are described but not parameterised §3.3. The demo thresholds are for display and should not be used as detection thresholds.
• Negation and composition. The authors say the pixel features are foreground-centred, lack compositionality and do not handle negation §6. Queries such as "not a cup" are not supported by the method.
• Table typo. The SemanticKITTI LSeg f-mIoU in tab. 5 is printed as "082". We read it as 0.82, but check the PDF before quoting it.
• Numbering. Equation numbers are taken from the LaTeX order. Table and figure numbers are counted in source order (the appendix uses A-numbers). Check them against the PDF before citing them externally.

Lineage

• Builds on: CLIP and its variants for open-set image embeddings, AudioCLIP for audio, and DINO for alternative features §1 app. A5. The mapping uses gradSLAM and PointFusion app. A3 §3.
• Compared against: LSeg-3D and OpenSeg-3D, which fuse finetuned pixel-aligned models with the same pipeline, and MaskCLIP-3D, a zero-shot per-pixel method §4.1. The related-work section also covers VL-Maps, NLMap-SayCan, CoWs, CLIP-Fields and OpenScene as concurrent work §2 §4.
• Superseded by: not stated in this paper, and no later card in the wiki covers it.