ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding

In plain words

Recognising 3D objects is easier when a model has seen each object described in words, but 3D shapes rarely come with good text descriptions, and writing them by hand does not scale. ULIP-2 gets around this. It takes a 3D shape, renders it from several fixed angles, and asks a large vision-language model to describe each picture. Of the descriptions it gets for a picture, it keeps the one that best matches the picture. The shape then becomes a set of matching items: its point cloud, its pictures and their descriptions.

The training step teaches a 3D network to place a shape's point cloud close to its own pictures and its own text in one shared space, and far from those of other shapes. The picture and text encoders come from a pretrained model that already aligns images with language, and they stay frozen. Only the 3D network learns. Because the pipeline needs only the 3D shapes, it can run on large collections with no labels.

The payoff is better recognition of 3D objects, both when naming a class the model was never trained on and when fine-tuned on a labelled benchmark. The main claims are the ones in the summary above.

TL;DR

ULIP-2 aligns three modalities (3D point cloud, rendered images, language) in the space of a frozen OpenCLIP ViT-G/14 model, used for most experiments §3.3. The language side is generated, not annotated: BLIP-2 writes about ten descriptions per rendered view, CLIP ranks them, and the top one is kept §3.2 §4.1. The 3D encoder is trained with two symmetric contrastive losses eq. 1 eq. 2 eq. 3. Author-reported: 50.6% top-1 zero-shot on Objaverse-LVIS and 84.7% on ModelNet40 tab. 1. The arXiv paper does not give training hyperparameters, and the released code is partial, so the recipe is only partly reproducible from this card.

Problem

Earlier multimodal 3D pre-training (ULIP) built its language side from dataset metadata turned into sentences §3.1. The paper argues that such language is neither scalable nor diverse [abstract] §1. The authors want language that is holistic (covering the whole object, view by view) and that needs no 3D annotation, so that any unlabelled 3D dataset can be used §1 §3.2.

Method

ULIP-2 keeps the pre-training framework of ULIP and changes how the triplets are built §3.

1. Scalable triplet creation §3.2

1. Extract the 3D point cloud from the object surface. This is the 3D input to the 3D encoder §3.2.
2. Render the object from a fixed set of viewpoints. The paper uses 12 views for Objaverse and 30 for ShapeNet §4.1.
3. For each rendered image, BLIP-2 (OPT-6.7B variant) generates ten descriptions independently §4.1.
4. Rank the ten descriptions by CLIP-ViT-Large image-text similarity and keep the top-1 as that view's language item §4.1 §5.4.

Figure 1 and Figure 2 show the overview and the per-view captioning fig. 1 fig. 2.

# Language item for each view of one 3D shape  [§3.2] [§4.1]
# Input:  shape O; a fixed list of viewpoints
# Output: point cloud P, rendered images, one caption per image
def make_triplets(shape, viewpoints):
    # Extract the 3D point cloud from the surface  [§3.2]
    point_cloud = sample_points(shape)
    # Render the shape from each fixed viewpoint  [§3.2] [§4.1]
    images = [render(shape, view) for view in viewpoints]
    captions = []
    for image in images:
        # Generate ten candidate descriptions with BLIP-2  [§4.1]
        candidates = blip2_describe(image, num_candidates=10)
        # Rank by CLIP image-text similarity, keep the top-1  [§4.1] [§5.4]
        ranked = sort_by_clip_similarity(image, candidates)
        # Keep the top-1 caption for this view  [§4.1] [§5.4]
        captions.append(ranked[0])
    # Return the triplet items for this shape  [§3.2]
    return point_cloud, images, captions

2. Tri-modal pre-training §3.3

The image encoder E_I and text encoder E_T are the frozen, pre-aligned OpenCLIP ViT-G/14 encoders, used for most experiments §3.3. The 3D encoder E_P is the only part trained. Its output is compared with image and text features under a CLIP-style contrastive loss, once for images and once for text §3.3.

• 3D-to-image loss eq. 1: a symmetric contrastive term in which each point cloud should match its own image against the other images in the batch, and each image should match its own point cloud against the other point clouds.
• 3D-to-text loss eq. 2: the same form with text features in place of image features.
• Objective eq. 3: minimise the sum of the two losses over the encoder E_P.
• The temperature τ is learnable §3.3.

# Tri-modal contrastive pre-training step  [§3.3] [eq. 1] [eq. 2] [eq. 3]
# Input:  point_feats (B, d), image_feats (B, d), text_feats (B, d)
# Output: scalar loss; only the 3D encoder receives gradients
def pretrain_loss(point_feats, image_feats, text_feats, tau):
    # Unit-normalise each modality  [code: models/losses.py#L33-L36]
    point_feats = normalize(point_feats)
    image_feats = normalize(image_feats)
    text_feats = normalize(text_feats)
    # Similarity matrices scaled by the learnable temperature  [§3.3]
    p2i = point_feats @ image_feats.T / tau
    p2t = point_feats @ text_feats.T / tau
    # Symmetric cross-entropy: row-wise and column-wise  [eq. 1]
    loss_p2i = 0.5 * (cross_entropy(p2i) + cross_entropy(p2i.T))
    # The same for text  [eq. 2]
    loss_p2t = 0.5 * (cross_entropy(p2t) + cross_entropy(p2t.T))
    # Minimise the sum of both losses  [eq. 3]
    return loss_p2i + loss_p2t

Note that the paper writes each term as a sum over samples, while the code averages over the batch. See Paper vs code.

3. Scaling the backbones §3.4 §5.5

The paper swaps the ViT-B vision-language encoders used in ULIP for OpenCLIP ViT-G/14 §3.4. It also varies the 3D encoder size and reports that performance peaks near 32.5M parameters, then gains diminish §5.5 tab. 9. The chosen setting is the 32.5M model §5.5.

4. Downstream tasks §4.2 §4.3

• Zero-shot 3D classification: the point cloud is scored against text embeddings of the class names, using prompt templates, as in ULIP and OpenShape §4.3 tab. 1.
• Standard 3D classification after fine-tuning, on ScanObjectNN §4.3 tab. 3.
• 3D-to-language generation: the pretrained encoder is plugged into the X-InstructBLIP framework with a frozen LLM, and scored with CIDEr §4.3 fig. 3 tab. 4.

# Zero-shot 3D classification  [§4.3] [code: main.py#L372-L383]
# Input:  point_cloud; class_names; prompt templates
# Output: predicted class name
def classify(point_cloud, class_names, templates):
    # Average the normalised embeddings of each class's prompts  [§4.3]
    class_embeds = []
    for name in class_names:
        prompts = [t.format(name) for t in templates]
        embeds = normalize(encode_text(prompts))
        class_embeds.append(normalize(embeds.mean(dim=0)))
    # Cosine similarity between the point cloud and each class  [§4.3]
    point_embed = normalize(encode_pc(point_cloud))
    scores = point_embed @ stack(class_embeds).T
    # Return the name of the highest-scoring class  [§4.3]
    return class_names[argmax(scores)]

Hyperparameters

Items marked "paper" are stated in the arXiv text. Items marked "code" are read from the official repository at code_commit. The paper gives no optimiser, learning rate, batch size or epoch count for pre-training.

Results

Author-reported. All numbers are top-1 and top-5 accuracy unless marked. The setup of each row is only partly stated in the paper; see the notes after each table.

Zero-shot 3D classification tab. 1

The caption of tab. 1 says the OpenShape rows are the current state of the art and that ULIP-2 is 3.8 points above OpenShape on Objaverse-LVIS top-1. The "Manual captions?" column is as printed; the caption defines a tick as captions that rely to some degree on manual effort tab. 1.

ScanObjectNN fine-tuning tab. 3

Voting follows the protocol cited in the paper's caption. The caption says ULIP-2 "significantly outperforms the baselines" tab. 3.

3D-to-language generation (CIDEr) tab. 4

Both encoders are pre-trained on the same Objaverse + ShapeNet data tab. 4.

Ablations (ModelNet40 and Objaverse-LVIS, unless stated)

Captions: manual versus top-1 holistic BLIP-2 captions, Point-BERT, ShapeNet, SLIP ViT-B encoders tab. 5:

Captioner: BLIP versus BLIP-2, same setup tab. 6:

Number of holistic views, ModelNet40, with top-1 BLIP-2 caption tab. 7:

Top-k CLIP-ranked BLIP-2 captions kept per view, ModelNet40 tab. 8:

Scaling, Objaverse (no LVIS) pre-training, Point-BERT tab. 9:

Appendix app. A.1 app. A.2

Input colour, Objaverse-LVIS, OpenCLIP ViT-G, Point-BERT, Objaverse + ShapeNet tab. 10:

Backbone, ModelNet40, ShapeNet pre-training, SLIP ViT-B tab. 11:

Paper vs code

Repository: salesforce/ULIP at code_commit (BSD-3-Clause). It is the official repository the abstract links, and it includes the ULIP-2 model class and evaluation scripts. The README says the released ULIP-2 weights are the CVPR 2024 version, which it says improved on the initial arXiv release. This card follows arXiv v4 (2024-04-26).

1. Feature normalisation. Paper: Eqs. 1 and 2 use raw dot products eq. 1 eq. 2. Code: each modality is L2-normalised before the similarity losses.py. Use the code's behaviour; the paper does not mention normalisation.
2. Reduction over the batch. Paper: each term is written as a sum over samples i, with a factor of -1/2 eq. 1 eq. 2. Code: cross_entropy averages over the batch, so the loss is the paper's sum divided by the batch size losses.py. The two directions (point-to-image and image-to-point) match the paper's two terms. AdamW updates are largely invariant to a constant loss scale, so the effect on training is small, but the logged loss values differ by a factor of B.
3. Temperature. Paper: τ is learnable §3.3. Code: a learnable log-scale initialised to log(1/0.07), exponentiated, and clamped to at most ln 100 after each step ULIP_models.py main.py. The initial value and clamp are not in the paper.
4. Frozen encoders. Paper: the OpenCLIP encoders are frozen during pre-training §3.3. Code: ULIP2_PointBERT_Colored calls eval() on OpenCLIP but never sets requires_grad = False on its parameters ULIP_models.py. main.py builds the optimiser from every parameter with requires_grad set main.py. A run that trained this model as written would therefore update the OpenCLIP weights. The released scripts do not train this model: all pretrain_*.sh scripts use ULIP-1 models pretrain_pointbert.sh. The ULIP-2 class is used only for zero-shot evaluation main.py test_ulip2_pointbert_modelnet40.sh. So the frozen-encoder recipe is not checked by the released code.
5. Captions. Paper: BLIP-2 captions, CLIP-ranked, top-1 per view §3.2 §4.1. Code: the model accepts K captions per sample and averages their embeddings ULIP_models.py. The ShapeNet training loader does not read BLIP-2 captions. It picks one random ShapeNet category name per shape dataset_3d.py. No BLIP-2 generation or CLIP ranking code is in the repository. The released ULIP-2 triplets are on the dataset host named in the README, and their format is not described in the code.
6. Projection to text width. Paper: no projection is mentioned §3.3. Code: a trainable 768-to-1280 linear map is applied to the 3D feature ULIP_models.py ULIP_models.py.
7. Training hyperparameters. Paper: none stated. Code defaults are listed in the Hyperparameters table main.py. ULIP_2_PointBERT_10k_colored_pointclouds.yaml lists an optimiser, schedule, epochs and batch size that differ from these defaults, and no code reads them ULIP_2_PointBERT_10k_colored_pointclouds.yaml.
8. Colour in ModelNet40. Paper: ModelNet40 point clouds are used as in ULIP §4.2. Code: colour channels are filled with a constant 0.4 for ModelNet40 dataset_3d.py. The ModelNet40 numbers therefore use no real colour. Objaverse-LVIS uses the stored rgb values dataset_3d.py.
9. Points per task. Paper: 10k, 8k and 2k points, with no mapping to tasks §4.1. Code: the ULIP-2 test scripts use 10000 points test_ulip2_pointbert_objaverse_lvis.sh. The ULIP-1 pre-training scripts use 8192 pretrain_pointbert.sh.
10. Encoder backbone name. Paper: "ViT-G/14" §3.3. Code: ViT-bigG-14 with pretrained tag laion2b_s39b_b160k ULIP_models.py. The tag is only in the code.
11. Sampling. The README says the released point clouds are subsampled with farthest point sampling README.md, which is the code path dataset_3d.py.

Pitfalls

• The pre-training recipe is incomplete. The paper gives no optimiser, learning rate, batch size, epoch count or schedule. The only pre-training scripts in the release use the ULIP-1 models with SLIP ViT-B encoders and 8192 points pretrain_pointbert.sh ULIP_models.py. Do not assume the ULIP-2 numbers come from those defaults.
• Whether the frozen encoder is really frozen is not verifiable from the release. See Paper vs code item 4.
• Caption quality is the main lever, and it is not released. The paper's ablations show a large gain from holistic BLIP-2 captions over manual ones (69.7 vs 60.4 top-1 on ModelNet40) tab. 5, and from keeping the top-1 CLIP-ranked caption rather than the top-3, top-5 or top-10 tab. 8. Reproducing the captions needs a BLIP-2 setup and a CLIP ranking that the release does not provide.
• Hidden cost of triplet creation. The Objaverse triplets contain about 800k point clouds, about 10 million images and about 100 million language items tab. 2. Each image needs ten BLIP-2 generations and CLIP scoring §4.1. The paper does not report the compute for this.
• Objaverse-LVIS contamination is not addressed. The 50.6% row uses "Objaverse + ShapeNet". The table has a separate "Objaverse (no LVIS)" row. The paper does not say whether the "Objaverse + ShapeNet" row contains the LVIS evaluation shapes tab. 1 §4.3.
• Encoder for the ShapeNet-only ULIP-2 row is ambiguous. The appendix caption for the 75.2 ModelNet40 number says SLIP ViT-B encoders tab. 11. The main table does not name the encoder for that row tab. 1. Treat the 75.2 number as ambiguous until the encoder is confirmed.
• Some improvements are stated loosely. "Improves ModelNet40 top-1 over ULIP by 14.8%" is a difference of 14.8 points (75.2 vs 60.4) §4.3 tab. 1. "Improves the captioning score by 28.3%" is a difference of 28.3 CIDEr points (160.5 vs 132.2), which is 21.4% relative §4.3 tab. 4.
• Gain over scratch training is not like-for-like. "ULIP-2 achieves a 4.0% gain over training from scratch" compares the voting result (91.5) with the non-voting baseline (87.5). Without voting the gain is 3.6 points (91.1 vs 87.5) §4.3 tab. 3.
• The "baseline without multimodal pre-training" for standard classification is not in the table. The 6.6% gain is measured against "Point-BERT (official)" at 83.1, which the table does not describe as trained from scratch §4.3 tab. 3.
• Input colour. The headline 3D input is xyzrgb with 10k points tab. 10. The ModelNet40 numbers in the code are computed with constant colour (Paper vs code item 8).
• Scale ablation numbers differ from the main table. The scaling ablation trains on Objaverse without LVIS and reports 35.7 on Objaverse-LVIS for the chosen setting tab. 9. The main table's 50.6 comes from a different training set. Do not compare the two rows directly.
• Table and figure numbers in this card follow LaTeX source order: Figures 1 to 3 and Tables 1 to 9 in the body, Tables 10 and 11 in the appendix. The appendix section label is inferred from \appendix and \section{Appendix} in the source app. A.1 app. A.2; the PDF may label it differently.
• Scope. Experiments cover object-level 3D shapes only. The authors name scene-level data as future work §6.

Lineage

• Builds on: ULIP, which uses a tri-modal (3D, image, language) contrastive pre-training with metadata-derived language, arXiv:2212.05171 §3.1. ULIP-2 keeps its training framework and replaces the language source §3.
• Uses: BLIP-2 for captions §3.2, CLIP for ranking §4.1, OpenCLIP ViT-G/14 as the frozen encoders §3.3, Point-BERT and PointNeXt as 3D backbones §4.1.
• Compared against: OpenShape (current state of the art on Objaverse-LVIS at 46.8% top-1 in tab. 1), PointCLIP, PointCLIPv2, ReCon, CLIP2Point, and ULIP tab. 1.
• Applied to 3D-to-language generation through X-InstructBLIP §4.3.
• Superseded by: not known from content in this wiki.