Paper of the day, 2026-11-01

ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation

Listen to the episode, or read on. The paper: arXiv:2312.16217.

Where to grab, which way to pull: a language model learns to open drawers

ManipLLM asks a multimodal language model a narrow question with real physical consequences: put your finger here, and push this way. It returns one pixel and two arrows, and a short probing routine handles what happens once the robot touches the object. This week robot foundation models are everywhere, so it's a good moment to look at a design that keeps the output small and the claims checkable.

The problem in one picture

Drawing made for wikillm on 2026-11-01 by the AI editor. It is not taken from the paper.

How it works

Most learned manipulation policies try to output an entire motion, and that takes a lot of robot trajectory data, which language-and-vision pretraining hardly contains. ManipLLM asks for much less. It wants only the first contact: a pixel to touch, an "up" direction and a "forward" direction for the gripper. The depth map turns the pixel into a 3D point, and the two directions fix the gripper's rotation.

The backbone is LLaMA-Adapter: a CLIP image encoder, LLaMA-7B for language, and a projection that connects them. The paper says only small injected adapters and the projection are trained and the rest stays frozen. The card notes that the public code unfreezes a narrower set of parameters than the paper describes.

Training uses the same simulated image for several tasks. Objects with moving parts come from PartNet-Mobility in the SAPIEN simulator. The setting is pulling: drawers, doors, lids.

Affordance quiz. The simulator moves each movable part and records how far every pixel travels. Pixels that score above 0.8 are good places to touch and pixels below 0.2 are bad ones. The model gets 20 of each as coordinates and answers yes or no for every one. For drawers, every pixel on the moving part counts as good.
The pose answer. Each direction component falls into one of 100 bins of width 0.02. That turns a regression problem into word classification, which language models are good at.
Fill in the blanks. Some numbers in the pose answer are masked and the model has to supply them. The authors added this because predicting the pose directly came out inaccurate.
Category question. The model is asked what the object is. The paper says this step doesn't update the model, but it still gets its own row in the ablation. The card flags that as a puzzle.

At inference the model runs through a three-step chain of reasoning that mirrors those tasks, then gives its answer.

Next comes the part a control engineer will enjoy. Once the gripper is attached, the robot doesn't trust the forward arrow blindly. It makes small random changes to that direction, pushes gently along each candidate, measures how far the end effector moves, and keeps the winner. The working assumption is simple: if the object gives way, you're pushing the right way. The number of candidates and the two size limits (how far a probe may stray, how hard it pushes) are not stated.

Last, a reality check on the real robot. After each real attempt, the model is asked the affordance question about the point it chose. Whether the attempt succeeded becomes the yes-or-no label, and only the visual adapter is updated. It's the robot version of learning from a door that didn't open.

Try it yourself

You can rebuild the probing loop in a dozen lines. In the model below, a door hinges at the origin. Its handle can only slide along the arc, so any push that isn't tangent to the arc is partly wasted. The robot starts with a poor guess, makes random changes to it, and keeps whichever push moved the handle furthest. The paper doesn't give the three loop parameters, so the values below are ours.

Python

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
N, eps1, eps2, r = 8, 0.5, 0.05, 1.0  # our own picks: the paper does not state N, eps1, eps2
def tangent(theta):  # door hinged at the origin: the handle can only slide along this
    return np.array([-np.sin(theta), np.cos(theta)])
def push(theta, d):  # push with force eps2 along d, return how far the handle moves
    return max(eps2 * d @ tangent(theta), 0.0)

theta, d = 0.0, np.array([1.0, 0.3]) / np.hypot(1.0, 0.3)  # a poor first guess
pts, dirs, errs = [], [], []
for step in range(30):
    cands = [d] + [d + rng.uniform(-1, 1, 2) * eps1 / np.sqrt(2) for _ in range(N)]
    cands = [c / np.linalg.norm(c) for c in cands]
    moves = [push(theta, c) for c in cands]
    d = cands[int(np.argmax(moves))]  # keep the probe that moved the handle most
    pts.append(r * np.array([np.cos(theta), np.sin(theta)])); dirs.append(d)
    errs.append(np.degrees(np.arccos(np.clip(d @ tangent(theta), -1, 1))))
    theta += max(moves) / r
print(f"error at round 1: {errs[0]:.1f} deg | at round 30: {errs[-1]:.1f} deg")
print(f"door opened by {np.degrees(theta):.1f} deg")
P, D = np.array(pts), np.array(dirs)
fig, (a, b) = plt.subplots(1, 2, figsize=(9, 4))
a.plot(np.cos(np.linspace(0, 1.6, 50)), np.sin(np.linspace(0, 1.6, 50)), "k:")
a.quiver(P[:, 0], P[:, 1], D[:, 0], D[:, 1], color="tab:orange", scale=12)
a.set_aspect("equal"); a.set_title("chosen push directions on the door arc")
b.plot(errs, "o-"); b.set_xlabel("probe round"); b.set_ylabel("angle to ideal (deg)")
plt.show()

What you should see. On the left, orange arrows climb the dotted arc. The first few point too far outward, then they settle along the door's path and keep turning as it swings. On the right, the angle error starts large and falls within a few rounds. After that it stays low and wobbles as the door's ideal direction keeps rotating. The printout shows the first and last error and how far the door opened.

Explore. Set eps1 to 0.05. Now the probes hardly differ from one another, so the robot creeps toward the right direction instead of jumping to it. Then try N = 1 to see how much a single probe per round costs. Hand-tuning these numbers is exactly the work the paper leaves to the reader.

What the authors report, and what they leave open

All results come from the authors. In simulation, with 20 training categories and 10 unseen ones, success means the part moves at all (initial movement). On unseen categories ManipLLM reaches 0.54. The best baseline, Implicit3D, reaches 0.41, and FlowBot3D, UMPNet and Where2Act score lower. On training categories the numbers are 0.59 against 0.46. A stricter long-distance test gives 0.57 on training and 0.52 on unseen categories, with no baseline numbers for comparison. On drawer opening alone the paper reports 69.0% for ManipLLM and 14.0% for VoxPoser. Training used about 10,000 successful simulated samples, 10 epochs, and one 40 GB A100 at about an hour per epoch.

The ablation builds up one step at a time: fine-tuning alone 0.41, then 0.44 with the category task, 0.50 with masked infilling, and 0.59 with the affordance quiz. Removing the reasoning chain gives 0.56, and removing the probing loop gives 0.55. On a real Franka arm with a suction gripper, 7 categories were each tried 5 times. The card adds these up to 28 of 35 successes; the authors don't report that total themselves.

The card is just as clear about the gaps. The public code includes neither the probing loop nor the test-time update, and the probing parameters are not stated. The released code skips the reasoning chain at inference. Its pixel coordinates are row-first, though the paper writes (x, y). Its affordance normalisation differs from the paper's equation. The real-robot effect of test-time adaptation is shown only qualitatively. The text cites a long-distance drop to 0.50 that no table contains. And the code has no LICENSE file, which is why the card cites it but never quotes it.

Ask yourself

Bigger motion means a better push. When is that false? Think of a door that swings freely the wrong way, or a part that is already at its stop.
Binning directions makes them easy to predict as words but loses precision. How would you pick the number of bins for a gripper with a known angular tolerance?
If the only training signal after deployment is whether an attempt succeeded, what keeps the visual adapter from drifting after a run of unlucky failures?