ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation

In plain words

A robot that has to open a door or pull a drawer needs to know where to grab the object and which way to move its hand. Learning this from raw robot trajectories needs a lot of robot data and generalises poorly across object types. This paper starts from a language model that already knows what objects are and how they are usually handled, and teaches it to answer questions about where to touch an object and how to orient the gripper.

The model gets one picture and one written request, and replies with a point in the picture and two directions for the gripper. The training data comes from simulated objects with moving parts. A simple probing loop then pushes the gripper in small test directions and keeps the one that moves the part most. After a real attempt, the model is adjusted a little using whether that attempt worked.

The idea matters because it reuses a language model's general knowledge about objects and adds robot-specific training on top.

TL;DR

ManipLLM fine-tunes LLaMA-Adapter, a multimodal language model, on simulated pulling tasks so that from an RGB image and a text prompt it outputs a contact pixel plus two gripper direction vectors. An affordance-prior task, masked pose infilling and a closed-loop impedance policy complete the method. The authors report 0.54 initial-movement success on 10 unseen simulated categories against 0.41 for the best baseline tab. 1. Their code is public, but the impedance policy and test-time adaptation are not in it (see Paper vs code).

Problem

Training a multimodal model directly on low-level robot trajectories needs many trajectory samples that pretraining data rarely contains, so generalisation is weak §1. Learned manipulation policies often treat prediction as a black box and struggle when many object categories are involved §1 §2. The paper instead asks the model to predict an object-centric target: the contact point and gripper orientation for the first contact §1.

The setup assumes articulated objects from PartNet-Mobility in the SAPIEN simulator, a Franka Panda with a suction gripper, and pulling primitives such as opening a drawer, a door or lifting a lid §4.1. Real-world tests use a Franka Emika arm with a suction gripper and a RealSense 415 camera §4.4.

Method

Backbone. The model is LLaMA-Adapter. A CLIP visual encoder produces image features, a LLaMA tokenizer encodes the text, and a multimodal projection module aligns the two §3.1.1. Only the injected adapters and the projection are meant to be trained; the rest stays frozen §3.1.1. The paper names LoRA as the injection method §3.1.1. What the public code actually unfreezes differs (see Paper vs code).

Fine-tuning tasks §3.1.2 fig. 2. Four tasks use the same image:

1. Object category identification (OCI). The prompt asks for the object category. The paper says the model is not updated in this stage and the task only provides a category prior §3.1.2.

2. Affordance prior reasoning (APR). The affordance map is computed in simulation: for each movable part, each pixel gets a score from how far it moves when the part is rotated or slid §3.1.2. Pixels score as

   \mathcal{A} = \frac{\mathcal{D}}{\left|\max(\mathcal{D}) - \min(\mathcal{D})\right|} eq. 1

   where \mathcal{D} is the per-pixel 3D displacement §3.1.2. Positive samples have score above 0.8 and negatives below 0.2 §3.1.2. The model gets n positive and n negative pixel coordinates in the prompt and answers "yes" or "no" for each, with n = 20 §3.1.2 §4.1. Cross-entropy is applied §3.1.2. For prismatic parts (drawers) every movable pixel is set to 1 §3.1.2.

3. Fine-tuning (FT). The prompt asks for the contact point and gripper direction. The answer is the pixel, the up vector and the forward vector, with each direction component put into one of 100 bins of width 0.02 over [-50, 50], so the task becomes classification §3.1.2.

4. Masked language modelling (MLM). Parts of the pose answer (coordinates or direction values) are masked in the prompt, and the model fills them in §3.1.2. The authors say plain pose regression was inaccurate, so MLM was added §3.1.2.

The total training objective is the sum of the three losses named in the paper: APR, MLM and FT §3.1.2.

Inference. The model reasons in three steps that follow the training tasks and then outputs the pixel (x, y), the up vector and the forward vector §3.1.2 fig. 4. The depth map gives the 3D contact point, and the two directions together define the end-effector rotation §3.1.2.

Active impedance adaptation §3.2. After first contact, a loop chooses the next direction. Starting from the predicted forward direction d_i, the method draws perturbations of norm below ε1, giving N+1 candidate directions, applies a force of norm ε2 along each, measures each end-effector displacement δ_j, and keeps the candidate with the largest motion:

d_{\mathrm{opt}}, \ \mathrm{opt} = \underset{j \in \{0,1,\ldots,N\}}{\arg\max} \ \| \delta_j \| §3.2

The authors assume that larger motion means a more efficient force direction in constrained manipulation §3.2. Values of N, ε1 and ε2 are not stated.

Test-time adaptation (TTA) §3.3 fig. 5. Real-world gaps (lighting, texture, hardware) are handled by updating part of the model during testing. For each test sample the model gets the APR prompt, with the wording "Determine if operating on the following point can effectively manipulate the object within the image: (x, y)", applied to its predicted point §3.3. Its prediction is treated as "yes", and the real success or failure of the trial gives the supervision signal, so the answer becomes "yes" or "no" §3.3. Only the visual adapter is updated §3.3. The paper says this adds an extra reasoning step per test sample §3.3.

# Turn the model's answer into a 3D contact point  [§3.1.2]
# Pixel order (x, y) and the depth read-out are as the paper describes.
def contact_point_3d(pixel, depth_map, intrinsics):
    # Depth at the predicted pixel  [§3.1.2]
    z = depth_map[pixel]
    # Back-projection formula and camera model are not stated  [§3.1.2]
    return back_project(pixel, z, intrinsics)

# Decode one direction component from its predicted bin  [§3.1.2]
# Bins have width 0.02 over [-50, 50]; bin edge choice is not stated.
def decode_component(bin_index):
    # Scale the integer bin back to a unit-vector component  [§3.1.2]
    return bin_index * 0.02

# One step of the active impedance loop  [§3.2]
# eps1, eps2 and N are not stated in the paper.
def next_forward_direction(forward, eps1, eps2, n_samples):
    # Start from the predicted forward direction  [§3.2]
    candidates = [forward]
    # Add perturbations of norm below eps1  [§3.2]
    for j in range(n_samples):
        noise = sample_perturbation(eps1)
        candidates.append(forward + noise)
    # Apply a force of norm eps2 along each candidate and measure motion  [§3.2]
    motions = [push_and_measure(d * eps2) for d in candidates]
    # Keep the candidate with the largest end-effector movement  [§3.2]
    best = argmax([norm(delta) for delta in motions])
    return candidates[best]

# Test-time adaptation after one real trial  [§3.3]
# Only the visual adapter is trained; the target is the real outcome.
def adapt_visual_adapter(model, rgb, pixel, succeeded):
    # Ask the APR yes/no question about the predicted point  [§3.3]
    question = affordance_prompt(rgb, [pixel])
    # The answer target is "yes" or "no" from the real outcome  [§3.3]
    target = "yes" if succeeded else "no"
    loss = model.cross_entropy(rgb, question, target)
    # Update the visual adapter only  [§3.3]
    model.step_visual_adapter(loss)

# Training objective: one loss per fine-tuning task  [§3.1.2]
# The paper sums the three terms; the public code samples one task per example.
def training_loss(model, sample):
    # Affordance-prior yes/no answers, loss L_A  [§3.1.2]
    loss_a = model.cross_entropy(sample.aff_prompt, sample.aff_answer)
    # Masked coordinate or direction infilling, loss L_M  [§3.1.2]
    loss_m = model.cross_entropy(sample.mlm_prompt, sample.mlm_answer)
    # Pose answer, loss L_F  [§3.1.2]
    loss_f = model.cross_entropy(sample.pose_prompt, sample.pose_answer)
    return loss_a + loss_m + loss_f

Hyperparameters

Results

Author-reported. Setup: simulated PartNet-Mobility objects, 16 training categories in the main table plus 4 more in the second block (20 training categories in total, averaged) and 10 unseen test categories, initial-movement success §4.1 tab. 1. Baselines are Where2Act, UMPNet, FlowBot3D and Implicit3D §4.2.

The per-category numbers behind these means are in tab. 1. The 28 of 35 real world total is summed from tab. 3, not reported by the authors. The paper reports a real-world TTA example with and without TTA only qualitatively §4.4 fig. 5.

Paper vs code

The public repository is at github.com/clorislili/ManipLLM. The authors' project page links it (the abstract gives the URL), and its owner handle resembles the first author's email address on the paper. The repository has no LICENSE file, so no code is quoted here. The commit is 184adfa7e245acf45a65f6a2a331ac36d85b4b78.

• Inference: the public code infers only the final prompt, without chain-of-thought README.md. The test script uses the FT prompt test_llama.py. Paper: CoT in three steps §3.1.2.
• Impedance policy and TTA: a search of train/ and test/ for impedance, TTA and force-probing code found nothing. The only force code found is passive robot dynamics in the simulator env_ori.py. Paper: §3.2 and §3.3.
• Trainable parameters: the code sets requires_grad = False on every parameter, then re-enables only LLaMA parameters whose names contain norm, bias or lora llama_adapter.py. The CLIP projection, the visual blocks and the adapter query tokens are not re-enabled. Paper: the injected adapters in visual CLIP and LLaMA, plus the projection, are fine-tuned §3.1.1. This is what the code does as written; the code was not run.
• Tasks per example: the code draws one of four task indices uniformly per example dataset.py. Indices 0 and 3 both use the FT prompt and answer dataset.py dataset.py, so FT gets about half the samples and the category question is not trained. Paper: OCI is not updated §3.1.2 yet appears as its own ablation row tab. 2.
• Loss: the code computes one cross-entropy over answer tokens and multiplies the MLM loss by zero engine_finetune.py. Paper: the sum of three losses §3.1.2.
• MLM masking: the code masks three location tokens, drawn with replacement, inside the answer dataset.py. A 15% random index list is computed and never used dataset.py. Paper: mask the coordinate or direction values §3.1.2.
• Contact point: the code samples a pixel whose affordance is above 0.6 collect_data.py. Paper: random point on the movable part §4.1. The 0.6 threshold is not in the paper.
• Pixel order: the code stores [row, col] as pixel_locs collect_data.py, and the affordance prompt also lists (row, col) pairs create_dataset_aff.py. The paper writes (x, y).
• Affordance normalisation: the code min-max normalises the displacements over non-zero pixels, (D - min) / (max - min) collect_data.py. Eq. 1 divides by |max - min| with no subtraction of the minimum eq. 1. The two give different values, and the paper does not say which was used for the reported results.
• Forward direction and noise: the forward vector is a random vector, orthogonalised against the up vector collect_data.py. The up vector gets noise drawn from a normal distribution with mean -0.01 and standard deviation 0.01 collect_data.py. Neither is described in the paper.
• Success rule: training labels need direction cosine > 0.5 and motion > 0.1 (or fraction > 0.5) collect_data.py. The test rule is motion > 0.01 (or fraction > 0.5) with cosine > 0.3 for pulling test_one_sample_longterm.py test_one_sample_longterm.py. The paper gives thresholds 0.01 and 0.1 §4.1 but not the cosine rules.
• Long-distance result: the text says that without AIA the long-distance score drops from 0.57 to 0.50 §4.3. The 0.57 matches tab. 1. No table contains 0.50, and the AIA ablation row in tab. 2 is 0.55 (initial movement).
• Bin decoding: the test script decodes a direction as int(v) * 0.02 test_one_sample_longterm.py, which is the lower edge of the bin. The paper does not state the decoding.
• Matches: the APR thresholds (0.8 and 0.2), n = 20 and 10 epochs agree between paper and code.

Pitfalls

• Coordinates are (row, col). Both the contact pixel and the affordance prompt use row-first order in the code (see Paper vs code). An implementation that follows the paper's (x, y) label will swap axes.
• Bins use floor, not round. Encoding is floor(v / 0.02) on the printed text dataset.py. The code cuts the last two characters off each token (word[:-2]), so the result depends on the number of decimal places printed. Check the number format before reusing the scheme.
• Crop augmentation shifts labels. RandomResizedCrop is applied to the image dataset.py dataset.py, but the pixel coordinates in the answer are not changed. The paper does not mention augmentation.
• Task mix in code is 50% FT, 25% MLM, 25% APR, 0% OCI. One of four task indices is drawn per example, and two of those indices use the same FT prompt dataset.py.
• OCI is not trained in the code. The category question is written out but unused create_dataset_aff.py, and the paper says OCI does not update the model §3.1.2 while tab. 2 credits it with +3%.
• Trainable set is narrower in code. Unless the checkpoint is changed, only LLaMA's norm, bias and LoRA weights are trained llama_adapter.py. Matching the paper's visual-side adapter training requires changing that rule.
• Layer count conflict. The paper says the projection module has 32 transformer layers §4.1. The code builds a 16-layer visual block llama_adapter.py and 32 adapter query layers llama_adapter.py. Which one the paper means is not stated.
• Batch size, learning rate and warmup are not in the paper. The shell script uses batch 1 and base learning rate 1e-3, which gives an actual rate of 3.9e-6 under the code's scaling rule finetune.sh main_finetune.py. Warmup is one epoch finetune.sh.
• Dataset size. The paper states about 10,000 samples §4.1. The code generates about 20,000 candidates and keeps at most 900 per category run_gen_offline_data.sh create_dataset_aff.py.
• Success thresholds differ between training labels and testing. Training labels are stricter collect_data.py than the test rule test_one_sample_longterm.py. Compare results only on the same rule.
• Impedance and TTA are unreproducible from the release. N, ε1 and ε2 are not stated and the code is missing. The method description includes both steps §3.2 §3.3, so the public model alone does not reproduce the full pipeline.
• Real-world evidence is small. There are 7 categories with 5 trials each tab. 3, one arm and one gripper §4.4. The text says the model works "with or without TTA" §1, but the paper gives no quantitative TTA comparison.
• Figures. Figure captions and layouts are not reproduced here. The paper's numbering is: fig. 1 teaser, fig. 2 training tasks, fig. 3 affordance maps, fig. 4 inference chain, fig. 5 TTA visualisation.

Lineage

• Builds on: LLaMA-Adapter (backbone and training strategy) §3.1.1; LoRA (injected adapters) §3.1.1; the affordance-map idea is inspired by Flowbot3D §3.1.2.
• Compared against: Where2Act §4.2, UMPNet §4.2, FlowBot3D §4.2, Implicit3D §4.2, VoxPoser (opening drawer only) §4.2.
• TTA borrows the idea from two cited test-time adaptation works §3.3.
• Superseded by: not stated in the paper or repository.