ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

In plain words

Most 3D recognition models learn from small datasets with a fixed list of categories. Images and text, by contrast, are backed by huge collections, and a model that has learned to match pictures with captions already knows a lot about what objects look like and how they are named.

ULIP tries to borrow that knowledge. It keeps a pre-trained image-text model frozen and trains a 3D point-cloud network so that each object's point cloud lands close to the embedding of its picture and of its name. Once the 3D network sits in the same space, it can be fine-tuned with labels, or asked to name an object it has never been trained to label, by comparing its embedding with the text embeddings of candidate class names.

The approach is useful because the 3D network can be any architecture; the authors report gains on several point-cloud backbones. The cost is that training needs a rendered image for every 3D model, and the pre-trained image and text encoders must be available.

TL;DR

• ULIP aligns a 3D point-cloud encoder with the frozen image and text encoders of SLIP, a CLIP-style model §3.2 §4.3.
• Training uses triplets of image, text and point cloud built from ShapeNet55 CAD models, without manual annotation §1 §3.1.
• The 3D encoder is backbone-agnostic and is trained with symmetric contrastive losses on the 3D-text and 3D-image pairs §3.2 eq. 4 eq. 5.
• Author-reported: standard classification gains of about 3% on ScanObjectNN for PointMLP and PointBERT, and zero-shot top-1 of 60.4% for PointBERT+ULIP on ModelNet40 (all classes) tab. 1 tab. 4.
• The code is BSD-3-Clause and runs pre-training and zero-shot evaluation; fine-tuning code is not in the repository.

Problem

3D recognition is limited by small labelled datasets with a fixed set of categories §1. ShapeNet55 is one of the largest public 3D sets, and the authors argue that the small scale of 3D data hinders generalisation and real-world use §1. Vision-language models such as CLIP show that aligning images and text at scale enables zero-shot recognition §1 §2, but the 3D modality had not been studied in that setting §1.

Learning directly from triplets (image, text, point cloud) would need data that is hard to collect §3. ULIP sidesteps this by reusing an image-text model that is already aligned and by building the triplets automatically from ShapeNet55 §3 §3.1.

Method

ULIP keeps the image encoder f_I and text encoder f_S of a pre-trained vision-language model frozen, and trains only the 3D encoder f_P so that its feature for each object is close to the object's image and text features §3.2. The frozen models are SLIP, which the authors chose over CLIP §4.3. The 3D backbone is any point encoder whose output is projected to a shared 512-dimensional space ULIP_models.py.

1. Build one triplet per CAD model §3.1

• Render each CAD model from virtual cameras placed every 12 degrees, giving 30 RGB images and 30 depth maps, 60 candidates in total §3.1 fig. 2.
• Each training iteration draws one of the 60 candidates as the image input §3.1.
• The text is the metadata of the model: a synset name list with the category word §3.1. The paper wraps the word in 64 prompts, the 63 image-text templates from earlier work plus "a point cloud model of [WORD]", and averages the 64 text embeddings §3.1 eq. 3.
• The point cloud is the model's own point cloud, uniformly sampled to N_p points, then augmented with random point dropout, scaling, shifting and a small rotation perturbation §3.1.

2. Encode the three modalities §3.1 eq. 1 eq. 2 eq. 3

• The 3D feature is h^P = f_P(P), the output of the trainable backbone eq. 1.
• The image feature is h^I = f_I(I) from the frozen image encoder eq. 2.
• The text feature is h^S, the average of f_S over the 64 prompts for the object's word eq. 3.

3. Contrastive loss per modality pair §3.2 eq. 4

• For two modalities M1 and M2, the loss is a symmetric InfoNCE over a batch. Each object's own pair is the positive; the other objects in the batch are negatives §3.2 eq. 4.
• Similarities are cosine similarities of normalised features, scaled by a learnable temperature initialised as in CLIP §3.2 eq. 4.

4. Combined objective §3.2 eq. 5

• L_final = alpha L(I,S) + beta L(I,P) + theta L(P,S) eq. 5.
• By default alpha = 0, beta = theta = 1 §3.2. The image-text term is therefore off, and the 3D encoder is pulled toward both the image and the text features §3.2.
• The authors freeze the image and text encoders during the whole pre-training. Updating them, they report, causes catastrophic forgetting on the small triplet set and a large drop on downstream tasks §3.2.

# ULIP pre-training loss for one batch  [§3.2]
# points (B, N, 3); tokens (B, 77); images (B, 3, 224, 224)

def ulip_loss(points, tokens, images):
    # 3D embedding from the trainable backbone  [eq. 1]
    pc_embed = point_encoder(points) @ pc_projection         # (B, 512)

    # Frozen image and text encoders (SLIP)  [§3.2] [§4.3]
    image_embed = vision_encoder(images)                     # (B, 512)
    text_embed = text_encoder(tokens)                        # (B, 512)

    # Unit-normalise all three embeddings  [eq. 4]
    pc_embed = normalize(pc_embed)
    image_embed = normalize(image_embed)
    text_embed = normalize(text_embed)

    # Learnable temperature, applied as a scale  [§3.2]
    scale = exp(logit_scale)

    # Object i's pair is the positive on the diagonal  [eq. 4]
    logits_pc_text = scale * pc_embed @ text_embed.T
    logits_pc_image = scale * pc_embed @ image_embed.T

    # Average of the row-wise and column-wise cross-entropy  [eq. 4]
    loss_pt = symmetric_ce(logits_pc_text)
    loss_pi = symmetric_ce(logits_pc_image)

    # alpha = 0 drops the image-text term; beta = theta = 1  [eq. 5]
    return loss_pt + loss_pi

5. Downstream use

• Standard classification: the pre-trained backbone is fine-tuned with labels and the head is trained in the usual way §4.4. The architecture is unchanged, so inference cost does not change §4.4.
• Zero-shot classification: no fine-tuning. Each class name is filled into the 64 prompts, the text embeddings are averaged, and the object is assigned to the class whose embedding is nearest to its 3D embedding §4.5.
• Retrieval: the pre-trained PointBERT+ULIP model is used directly to retrieve point clouds for real photographs; this is shown qualitatively only §4.7.

# Zero-shot 3D classification, no fine-tuning  [§4.5]
# points (N, 3); class_names: list of str; prompts: 64 templates

def zero_shot_scores(points, class_names, prompts):
    # One text embedding per class: normalised mean over its prompts  [§4.5]
    class_embeds = []
    for name in class_names:
        captions = [p.format(name) for p in prompts]
        embeds = normalize(text_encoder(tokenize(captions)))
        class_embeds.append(normalize(mean(embeds, axis=0)))

    # 3D embedding, compared with each class by cosine similarity  [§4.5]
    pc_embed = normalize(point_encoder(points) @ pc_projection)
    scores = stack(class_embeds) @ pc_embed
    # Top-1 and top-5 are read off the sorted scores  [§4.5]
    return scores

Hyperparameters

Pre-training values from the paper and from the code defaults and shipped scripts. "not stated" means neither the paper nor the code gives a value.

Downstream fine-tuning values, from the paper. The fine-tuning code is not in this repository, so these are not checked against code.

Results

Author-reported. All numbers come from the paper; the setup is noted per row. Table and figure numbers follow the order of the LaTeX source, so check them against the compiled PDF.

Standard 3D classification, ScanObjectNN (overall accuracy; 1K points unless marked) tab. 1

PointNeXt, reproduced by the authors, goes from 87.4 to 89.2 overall (1K points) and from 85.8 to 88.0 class-mean app. PointNeXt.

Standard 3D classification, ModelNet40 (overall accuracy) tab. 2

The class-mean accuracy rises for PointMLP (91.3 to 92.3) and for PointMLP with voting (91.4 to 92.4) tab. 2.

Zero-shot 3D classification, ModelNet40, top-1 / top-5 tab. 4

The abstract's 28.8-point gain over PointCLIP on ModelNet40 is the Hard set (37.1 against 8.3) tab. 4. The Medium and Hard sets remove categories that also appear in the ShapeNet55 pre-training list §4.5 app. eval sets.

Zero-shot 3D classification, ScanObjectNN, top-1 / top-5 (all classes) tab. 5

Aligning two versus three modalities, zero-shot tab. 3 tab. 6

For PointBERT, top-1 on ModelNet40 (all classes) is 44.7 when aligned to text only, 35.5 to image only, and 60.4 to both tab. 3. On ScanObjectNN the same three settings give 31.0, 36.3 and 48.5 tab. 6. The authors conclude that aligning all three modalities always beats either pair alone tab. 3 tab. 6.

Other results

• Data efficiency: ULIP improves PointMLP and PointBERT when fine-tuning on a fraction of the training data fig. 3. The text notes that PointBERT beats the PointMLP baseline when it has under 20% of the training data, because PointBERT itself was pre-trained on ShapeNet55 §4.6. The exact curve values are not given in the text.
• Indoor detection on ScanNet-v2 with Group-Free-3D: mAP@0.5 goes from 48.9 to 50.2 app. indoor 3D det..
• Real-image to point-cloud retrieval uses PointBERT+ULIP on ModelNet40 with Caltech101 queries. Results are qualitative only §4.7 fig. 4.

Paper vs code

Checked against the official repository salesforce/ULIP at commit 95d480fe2b16c06d0558c60b5cfea981b4cdc8eb.

• Text prompts in pre-training. The paper averages 64 templates per object at every iteration §3.1 eq. 3. The ShapeNet loader builds one caption from a random synonym and sets template expansion to off dataset_3d.py dataset_3d.py. The template averaging in the model then runs over that single caption ULIP_models.py. The 64 prompts are used only at zero-shot test time main.py. The paper says the zero-shot step uses the same prompt strategy as pre-training §4.5, which the code does not do.
• Learning rate. The paper gives 1e-3 §4.3. The code default is 3e-3 main.py. The shipped scripts use 3e-3 for PointNet++ and PointBERT pretrain_pointnet2_ssg.sh pretrain_pointbert.sh and 1e-3 for PointMLP and PointNeXt pretrain_pointmlp.sh pretrain_pointnext.sh.
• Points and sampling. The paper names 1024, 2048 or 8192 points by backbone §4.3. The default and all shipped pre-training scripts use 8192 main.py. The paper says uniform sampling §3.1; the code uses farthest-point sampling dataset_3d.py.
• Augmentation. The paper lists four operations §3.1. The code adds a random rotation about the up axis dataset_3d.py. Its dropout sets dropped points to the first point rather than removing them dataset_3d.py.
• Batch and negatives. The paper gives batch size 64 and eight A100 GPUs §4.3. The code treats 64 as a per-GPU size and gathers the embeddings across GPUs before computing the logits main.py losses.py. The scripts launch eight processes, so each step contrasts against 512 objects.
• Frozen encoders. The paper freezes the image and text encoders §3.2. The code freezes every parameter whose name also appears in the SLIP checkpoint ULIP_models.py. That set may include logit_scale; this cannot be checked without the checkpoint.
• Model selection. The paper does not say how the reported checkpoint is chosen §4.5. The code evaluates zero-shot top-1 on the ModelNet40 test split after every epoch and keeps the best one main.py dataset_catalog.json.
• Medium and Hard sets. The paper defines them §4.5 app. eval sets. The repository has no code that filters categories for them; only the full 40-class ModelNet40 test set is evaluated.
• README numbers. The README's ULIP-1 zero-shot table differs from the paper's ALL column: PointNet2 (ssg) 57.7 / 78.9, PointMLP 60.0 / 79.4, PointBERT 60.3 / 84.0 README.md. The paper gives 55.7, 61.5 and 60.4 for the same three models tab. 4. The README does not say which run each number comes from.
• Two-modality ablation. The paper's alignment ablation tab. 3 tab. 6 cannot be reproduced by flags, because the loss always uses the 3D-text and 3D-image pairs losses.py.
• Zero-shot distance. The paper speaks of the smallest distance §4.5. The code ranks by cosine similarity of normalised vectors main.py. For unit vectors these rank the same, so this is not a conflict.

No other discrepancies found in the pre-training objective: the code matches eq. 4 and eq. 5 with alpha = 0, beta = theta = 1 losses.py.

Pitfalls

• Pre-training needs a rendered image for every CAD model. The paper renders 60 candidates per model §3.1. The repository expects pre-rendered images and its README warns that the full rendered set is about 1 TB README.md. The renderer used is an external repository linked in a footnote §3.1.
• Frozen encoders still run in every step. Each training step runs the SLIP image encoder and the text transformer ULIP_models.py. The image encoder is a ViT-B/16 at 224 px. Budget for that forward pass.
• Checkpoint file. The model is initialised from slip_base_100ep.pt in data/initialize_models/, and the code loads it by name [code: models/ULIP_models.py#L258]. The paper does not name the checkpoint path.
• Name matching decides what is frozen. Freezing is by parameter name against the SLIP checkpoint ULIP_models.py. A renamed model would silently train encoders the paper keeps frozen. Check which parameters have requires_grad after loading.
• Temperature bounds. The log-scale starts at ln(1/0.07) and is clamped to [0, ln 100] after every optimiser step ULIP_models.py main.py. A run that exceeds this bound will not match the paper.
• Non-finite losses stop training. The loop exits if the loss is not finite main.py. Mixed precision is on by default main.py.
• Pre-training uses all of ShapeNet55. When the train split is built, the test list is merged into it dataset_3d.py dataset_3d.py. There is no held-out ShapeNet split for pre-training, and the paper does not say so §3.1.
• Zero-shot numbers are sensitive to the split and to checkpoint choice. The headline ModelNet40 numbers use the test split tab. 4, and the code picks the best checkpoint on that same split (see Paper vs code). Expect optimistic numbers unless a separate validation split is used.
• PointNeXt uses an extra height channel. The code appends a height feature when --use_height is set, and the PointNeXt script sets it dataset_3d.py pretrain_pointnext.sh. The input dimension therefore differs from the other backbones.
• Point-cloud backbone output size. The projection input dimension is set per backbone: 256 for PointNet++, PointMLP and PointNeXt, 768 for PointBERT ULIP_models.py ULIP_models.py. A new backbone must match this value.
• License split. The paper is CC BY 4.0. The code is BSD-3-Clause LICENSE.txt. The vendored PointNeXt folder carries its own LICENSE LICENSE and was not used for this card.
• Fine-tuning is not in this repository. The downstream tables above come from the paper only. The repository's test scripts run zero-shot evaluation only test_pointbert.sh.

Lineage

• Builds on: the CLIP image-text pre-training approach, and its SLIP variant used as the frozen encoder §1 §4.3. The 63 image-text prompts are taken from prior image-text work §3.1.
• Compared against: PointCLIP, the prior zero-shot baseline for 3D, which turns point clouds into depth maps for CLIP §2 §4.5. Standard classification is compared against PointNet++, PointMLP, PointBERT and PointNeXt backbones, and other 3D methods including PointMAE and RepSurf tab. 1 tab. 2.
• Superseded by: the README points to a follow-up, ULIP-2, whose models are pre-trained on 10K xyzrgb point clouds from Objaverse and ShapeNet triplets README.md. It is not covered by this card.