LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

In plain words

Language models answer text well, but a 3D scene is a cloud of points, and a model that only reads text has to guess which object you mean when you say "the chair by the window". LL3DA is a language model that can also read a 3D scene directly and can be pointed at things: you can type a question, and you can also click on an object or draw a box around it.

The scene is turned into a short list of learned summary vectors by a small attention module. Those vectors are placed in front of the question as if they were extra words, and a language model that stays frozen writes the answer. The clicks and boxes are turned into vectors in the same way, so the module can combine "what the user pointed at" with "what the user asked" and the scene.

The authors train this on several 3D tasks at once: describing objects, answering questions, describing a whole room, holding a short dialogue about a room, and listing steps for a task. They report that it beats earlier specialist models on two of these tasks.

TL;DR

• Frozen 3D scene encoder, frozen OPT-1.3B, and a trainable 6-layer Q-Former with 32 queries make up the model; about 111M parameters train app. E.
• Visual prompts (a click point or a box) enter the Q-Former with the text instruction, so the same model answers "describe this" and "where is this" §3.2 §4.1.
• Author-reported: 65.19 C@0.5 on ScanRefer and 51.18 C@0.5 on Nr3D for dense captioning, and 76.79 CiDEr on ScanQA validation §5.2 tab. 1 tab. 2.
• Most reported gains depend on a box source (Vote2Cap-DETR) and on details the released code handles differently from the text; see Paper vs code.

Problem

Multi-view approaches that lift 2D features into 3D are costly and lose geometry, and plain-text references are ambiguous in cluttered scenes §1. The paper wants one model that takes a raw point cloud and both text and visual interactions, and produces free text that can be read back as 3D coordinates §3.1 §1.

Method

Overview. The input is a point cloud PC with N points, each with xyz plus extra features (color, normal, height), a text instruction, and optional visual prompts (clicks or boxes) §3.1. The output is free-form text; numbers in it are read as 3D coordinates. The model has four parts, in order: a frozen 3D scene encoder, a visual prompt encoder, a Multi-Modal Transformer (a Q-Former), and a frozen LLM §3.2 fig. 2.

Step 1. Coordinates as text §3.1. A point is written as <loc>x, y, z</loc> and a box as its centre and size <obj>cx, cy, cz, w, h, l</obj>. Every number is an unsigned integer in [0, 255], scaled to the scene's boundary. No new tokens are added to the LLM vocabulary.

Step 2. Instruction format §3.1. The text follows the ### human: / ### assistant: convention from PointLLM, so multi-turn dialogue is possible.

Step 3. Scene encoder §3.2 eq. 1 app. E. A frozen masked transformer encoder, pre-trained for ScanNet detection with Vote2Cap-DETR, maps the sampled points to M feature vectors of width d (appendix: 256). Training samples 40k points per scene §5.1.

Step 4. Click prompt §3.2 eq. 2 eq. 3. The click is normalised to [0, 1] by the scene's size and encoded with a 3D Fourier embedding [sin(2π p·B); cos(2π p·B)]. A two-layer MLP maps it to 8 tokens of width 768.

Step 5. Box prompt §3.2 eq. 3. The box is represented by a detector ROI feature and a separate MLP maps it to 8 tokens. The click and box MLPs have the same shape but separate weights.

Step 6. Q-Former (Multi-Modal Transformer) §3.2 fig. 2. Six layers, width 768, up to 12 heads app. E. It holds 32 learned query vectors. In each layer: the queries, the prompt tokens and the instruction text share one self-attention; then queries and prompt tokens cross-attend to the scene features. The 32 outputs are projected by one linear layer into the LLM embedding space and used as a prefix §3.2.

Step 7. Frozen LLM §3.2. OPT-1.3B is frozen, including its embedding layer, and loaded in float16 §5.1. The prefix is concatenated with the embedded instruction, and the response is generated. At inference the response is the beam search result with beam size 4 §3.2 eq. 4.

Training objective §4.2 eq. 5 eq. 6. The trainable parameters θ are fitted by maximising the likelihood of the target response given the point cloud, the visual prompt and the instruction. The loss is token-wise cross-entropy over the response tokens eq. 6. Dense captioning and QA draw a visual prompt per sample at random; the other tasks have none §4.2 §4.1.

Task formats §4.1 app. D.

• 3D dense captioning: a click or a box identifies the object; the instruction either asks to describe it or to describe and localise it §4.1.
• 3D QA: the instruction asks for an answer, or for an answer plus the related object locations; visual prompts are added to objects named in the question §4.1.
• Scene description, embodied dialogue and embodied planning are multi-turn text tasks §4.1 app. B.

# LL3DA prefix and training loss  [§3.2] [§4.2]
# Input:  points (N, 3 + F); instruction and response token ids;
#         optional box and click prompts for the sample
# Output: scalar cross-entropy over response tokens

def ll3da_loss(points, instruction_ids, response_ids,
               box_prompts, click_prompts):
    # 1. Frozen scene encoder: N points to M tokens of width 256
    #    [§3.2] [eq. 1] [app. E]
    scene = frozen_scene_encoder(points)              # (M, 256)

    # 2. Click prompts: Fourier features of the normalised xyz, then an
    #    MLP that emits 8 tokens per click  [§3.2] [eq. 2] [eq. 3]
    clicks = fourier_features(normalise_to_unit(click_prompts))
    click_tokens = click_mlp(clicks).reshape(-1, 8, 768)

    # 3. Box prompts: ROI feature of the detector proposal that matches
    #    each box, then an MLP that emits 8 tokens per box  [§3.2] [eq. 3]
    box_feats = matched_proposal_feature(box_prompts, points)
    box_tokens = box_mlp(box_feats).reshape(-1, 8, 768)

    # 4. Q-Former: 32 learned queries share self-attention with the prompt
    #    tokens and the instruction text, then cross-attend to the scene
    #    [§3.2]
    prompt_tokens = concat(click_tokens, box_tokens)
    latents = qformer(learned_queries, prompt_tokens, instruction_ids,
                      scene)                          # (32, 768)

    # 5. Linear projection of the 32 latents into the LLM embedding space
    #    [§3.2]
    prefix = linear_projection(latents)               # (32, d_llm)

    # 6. Frozen LLM on prefix followed by the embedded instruction and
    #    response  [§3.2]
    logits = frozen_llm(concat(prefix, embed(instruction_ids,
                                             response_ids)))

    # 7. Cross-entropy over response tokens only; the paper writes a sum,
    #    the code averages  [§4.2] [eq. 6]
    return masked_mean_cross_entropy(logits, response_ids)

# Dense captioning at test time  [§3.2] [§5.2]
# Input:  scene, detector proposals (box, objectness), instruction
# Output: one caption per object proposal

def caption_proposals(scene, proposals, instruction_ids):
    # 1. Keep only proposals the detector labels as objects, not background
    #    [code: captioner.py#L395-L399]
    kept = [p for p in proposals if p.is_object]
    captions = []
    for box in [p.box for p in kept]:
        # 2. Build the prefix with this box as the visual prompt  [§3.2]
        prefix = build_prefix(scene, box, instruction_ids)
        # 3. Beam search with 4 beams, at most 48 new tokens
        #    [§3.2] [code: captioner.py#L289-L295]
        captions.append(beam_search(prefix, beams=4, max_new_tokens=48))
    # 4. Non-object proposals keep an end-of-sequence output
    #    [code: captioner.py#L413]
    return captions

Hyperparameters

Values marked "code" appear only in the released repository and are not in the paper text. Values marked "paper" come from the paper; the appendix is cited where it gives them.

Results

Author-reported. Setup: ScanNet scenes with Vote2Cap-DETR boxes as the box prompt for dense captioning §5.2 tab. 1; ScanQA validation and two test splits tab. 2.

3D dense captioning tab. 1. Metric: C@k, CiDEr at IoU threshold k. Full B-4, M and R are in tab. 1.

The paper reports +3.38 on ScanRefer C@0.5 and +7.34 on Nr3D C@0.5 over Vote2Cap-DETR §5.2.

3D question answering tab. 2. Metric: CiDEr.

The paper reports +7.39 CiDEr over 3D-LLM on validation §5.2.

Ablations (all trained as stated in each table; numbers are the paper's).

Test-time clicks on QA raise CiDEr by 6.12 but lower BLEU-4 (13.53 to 11.80) tab. 8. This result is not reproduced by the released QA code path; see Paper vs code.

Appendix results app. B app. C. Own split of the ScanNet part of 3D-LLM (scenes below id 600 for training). Decoding comparison, with OPT zero-shot and LLaMA-7B zero-shot as reference:

Open-vocabulary detection with the model's own boxes, ScanNet vocabulary: mAP 32.48 at IoU 0.5 against VoteNet's 31.50 at IoU 0.5 (VoteNet has AR 50.08 and the model 49.46) app. C. With the model's own boxes, dense captioning falls to 51.12 C@0.5 on ScanRefer, against 65.19 with the detector's boxes app. C.

Paper vs code

Official repository: https://github.com/Open3DA/LL3DA (linked from the paper's author block), commit ebbdd771b616ce3714932c3627d91ece6d8d6cd1, MIT licence (copyright Sijin Chen). Searched the paper's links only; the repository is the one the paper names.

• Fourier matrix. Paper: B is a learnable matrix §3.2 eq. 2. Code: B is a random Gaussian drawn once and stored as a non-trainable buffer position_embedding.py.
• Box feature. Paper: an ROI feature from the pre-trained 3D detector §3.2. Code: the feature of the detector proposal with the highest IoU to the given box, restricted to proposals labelled as objects; if no proposal overlaps, the argmax falls back to index 0 captioner.py.
• Extra projection. Code adds a two-layer ReLU MLP on scene features before the Q-Former. The paper does not describe it captioner.py captioner.py.
• Q-Former initialisation. Paper: only the BERT word and position embeddings are used, because loading full BERT weights gives repetitive output §3.2. Code: loads the bert-base-embedding checkpoint with a 6-layer config captioner.py. Which tensors are loaded from that checkpoint was not checked by running the code.
• Learning rate schedule. Paper: 1e-4 decaying to 1e-6 with cosine §5.1. Code: the generalist script uses 1e-4 to 1e-6 cosine with a one-epoch linear warm-up from 1e-6 [scripts/opt-1.3b/train.generalist.sh] main.py engine.py. The per-task ScanRefer script uses a constant 1e-6 (base and final both 1e-6) for 16 epochs [scripts/opt-1.3b/tuning.scanrefer.sh]. The code default is 5e-4 main.py. The paper does not mention warm-up.
• Batch size. Paper: total batch 16 for all training §5.1. Code: the generalist script uses 4 per GPU on 8 GPUs, total 32 [scripts/opt-1.3b/train.generalist.sh]; the ScanRefer tuning script uses 8 per GPU on 2 GPUs, total 16 [scripts/opt-1.3b/tuning.scanrefer.sh].
• Gradient clipping. Code clips the global gradient norm at 0.1 main.py engine.py. The paper does not mention clipping.
• Loss normalisation. Paper: eq. 6 is a sum over response tokens §4.2. Code: a mean over response tokens captioner.py. The difference changes the loss scale.
• QA visual prompts in training. Paper: visual prompts are "randomly adopted" for objects in the question §4.1. Code: a click is drawn for 25% of training samples only; boxes are never used for QA training unified_scanqa.py.
• Test-time QA prompts. Paper: Table 8 reports QA with a click at test time §5.3 tab. 8. Code: the QA prediction path calls the prefix builder without the box or click arguments captioner.py, and the ScanQA test set sets all prompts to zero test_set_scanqa.py. The shipped code does not produce a click-conditioned QA answer. Whether the table came from other code is not known.
• Dense captioning decoding. Paper: beam size 4 §3.2. Code: captions are generated only for object-labelled proposals, in batches of 8, and other proposals get end-of-sequence output captioner.py. A maximum of 48 new tokens is set in code captioner.py; the paper states neither.
• Scene token count. Appendix: 1,024 point tokens of width 256 app. E. The detector configuration has 256 decoder queries and encoder width 256 config.py config.py. The two counts are not reconciled in the text.
• Repetition penalty. The 4-gram repetition penalty is stated for the appendix evaluation app. B. In the code it is applied only in the chat path captioner.py; dense captioning and QA do not use it.
• Released artefacts. The appendix says code and pre-trained weights "will be fully released" app. intro. The clone contains the detector weights under pretrained/ but not the LL3DA checkpoints the scripts load from ./ckpts/ [scripts/opt-1.3b/eval.scanqa.sh].
• No discrepancy found for: coordinate discretisation to [0, 255] §3.1 main.py; 40k points §5.1 unified_densecap_scanrefer.py; 8 tokens per prompt §3.2 captioner.py; 32 queries §3.2 captioner.py; 6 Q-Former layers app. E captioner.py; frozen detector and LLM §3.2 main.py main.py; beam 4 when beam search is on captioner.py.

Pitfalls

• The box source matters. The dense captioning numbers use Vote2Cap-DETR boxes as the visual prompt §5.2. With the model's own boxes, C@0.5 drops from 65.19 to 51.12 app. C. Treat the 65.19 as a pipeline result, not a standalone detector.
• Coordinates as text are coarse. Each coordinate has 256 levels across the scene's extent §3.1. Boxes decoded from text are only as precise as that grid; the authors note a gap to specialist localisers app. C.
• Visual prompts at test time are not wired in the released QA path (see Paper vs code). Any test-time click result should be checked against code that passes the prompt through the prefix builder.
• Random visual prompts in training are a hidden design choice. The paper does not give the sampling rate; the code uses 50/50 box or click for dense captioning and a 25% click rate for QA unified_densecap_scanrefer.py unified_scanqa.py.
• Compute. Dense captioning runs one generation per object proposal, batched 8 at a time captioner.py. The paper does not report inference time; its only compute figure is the training hardware §5.1.
• Decoding choice changes the ranking. In the appendix, beam search has the highest dialogue and planning CiDEr, but greedy decoding has the highest scene-description CiDEr app. B.
• Cross-paper comparisons. 3D-VisTA is listed from a version not trained on extra scenes §5.2; 3D-LLM is the fine-tuned version tab. 2. Numbers use different protocols; the paper marks some as not comparable.
• Frozen LLM, small trainable part. Only the scene projection, the visual prompt MLPs, the Q-Former and the output projection train. The paper ties the frozen-LLM choice to memory cost §3.2. Without a text instruction, dense captioning scores 60.20 against 62.84 with one tab. 7, so the instruction helps but is not the main source of the gain.
• Training-length mismatch. The paper says about 100k iterations for the general model §5.1; the released generalist script trains for 32 epochs with evaluation from iteration 19,999 [scripts/opt-1.3b/train.generalist.sh]. Compare runs by epochs and data, not iteration counts.

Lineage

• Builds on: the Q-Former bridge from BLIP-2 and InstructBLIP, with the BERT-based query transformer §3.2. Scene encoder and box prompts come from Vote2Cap-DETR §3.2. The coordinate-as-text design follows the PointLLM instruction format §3.1.
• Compared against: Scan2Cap, MORE, SpaCap3D, REMAN, D3Net, Contextual, UniT3D, 3DJCG, 3D-VLP, 3D-VisTA, Vote2Cap-DETR (dense captioning) and ScanQA, Clip-Guided, Multi-CLIP, 3D-VLP, 3D-VisTA and 3D-LLM (QA) §5.2 tab. 1 tab. 2.
• Superseded by: no card in this wiki states this.