DexDiff: Towards Extrinsic Dexterity Manipulation of Ungraspable Objects in Unrestricted Environments

In plain words

Picking up a large flat thing such as a book or a pan from a table is hard, because its edge is too thin and too low to grip from above. People solve this by sliding the object to the table's edge or against a wall so one side hangs over, then picking it up from that side.

DexDiff teaches a robot to do the same. A vision-language model looks at the scene and writes a short plan, for example "push it to the edge and grasp it from the side". A second model, a diffusion-based controller, then turns that plan into a short sequence of arm movements. The plan is passed in as a goal signal, so the controller knows which way the object should end up.

The authors train and test this in four simulated tabletop layouts and on a real robot arm, and report that it succeeds more often than the baselines they tried. The approach needs a lot of demonstration data, and the paper says it is computationally expensive.

TL;DR

• Ungraspable large flat objects are grasped after an "extrinsic dexterity" push against a wall or table edge §1.
• A fine-tuned VLM produces the high-level plan, and a goal-conditioned action diffusion model (GCAD) produces the low-level actions §3.
• Author-reported: 70% average success over four simulated scenes, against 23% for the best baseline (Finetuned-VLM with Diffusion Policy-T) tab. 1.
• Author-reported: 65% success on a real UR5e across four daily-life setups tab. 2.
• No official code was found, so the card relies on the paper alone.

Problem

Large flat objects lying on a surface cannot be grasped from the top, because the only grasp points are on the side, which is unreachable §1. Prior extrinsic-dexterity methods use a wall or a table edge to make the side reachable, but each is built for one structure type and one task, and none does task planning to find the pre-grasp condition §1, §2. The paper targets unrestricted environments, where the robot must choose among several structures and handle unseen objects §1.

Method

DexDiff has two modules §3, fig. 2.

High-level plan (VLM). A VisualGLM-6B model is fine-tuned to read a scene image and a text prompt and to output a plan naming a skill, a structure, a direction and the target object §3.1. Training samples are tuples of an object expression, a scene image and a labelled plan §3.1. The paper reports about 1.6k simulated and 1.4k real labelled images §3.1, §4.1. A pre-trained semantic parser turns each plan into relational tuples, such as "right of toolbox, edge" §3.1. The paper says these tuples are used to filter action sequences and trajectories, but the filter itself is not described §3.1.

Return-to-go values. The plan is turned into a series of return-to-go values by computing goal-conditioned grasp poses §3.1. The paper gives no formula for these values §3.1, §3.2.

Low-level actions (GCAD). GCAD is a diffusion policy that predicts a chunk of actions rather than one action §3.2. It starts from the DDPM noise-prediction loss eq. 2, modified as follows §3.2:

1. The model outputs a chunk of robot actions at each denoising step, not an image or a state §3.2.
2. The observation O_t is added as a conditioning input at each denoising step eq. 3, eq. 4.
3. The return-to-go embedding is added to the embedding layer, so the policy is conditioned on the plan-derived values §3.2.
4. The paper models p(A_t | O_t, R̂_t) rather than the joint distribution with future states §3.2.

Observations pass through a per-timestep visual encoder. The paper describes a ResNet-18 with spatial softmax pooling and GroupNorm, applied to each timestep independently app. A.4. A configuration encoder and the visual encoder feed a transformer, together with the return-to-go embedding and a diffusion-step embedding §3. The output is a predicted action sequence after K denoising steps, and an operational-space controller executes it §3.

Training. Sample a clean action chunk A^0 from the offline dataset, pick a random step k, add Gaussian noise, and regress the noise with MSE §3.2 eq. 4.

\mathcal{L}_{O, A, \hat{R}} = \mathrm{MSE}\big(\varepsilon^k,\; \varepsilon_\theta(O_t, \hat{R}_t, A_t^0 + \varepsilon^k, k)\big) eq. 4

Inference. Start from noise and apply the update of eq. 3 for K steps §3.2. Then execute the predicted T_a actions without re-planning, using a T_o-step observation history app. A.5.

A_t^{k-1} = \alpha\big(A_t^k - \gamma\,\varepsilon_\theta(O_t, \hat{R}_t, A_t^k, k) + \mathcal{N}(0, \sigma^2 I)\big) eq. 3

The paper calls α, γ and σ functions of the step k, the "noise schedule" §3.2. It names the "Square Cosine Schedule" and cites "Improved denoising diffusion probabilistic models" for it app. A.4.

Simulated demonstrations come from Soft Actor-Critic policies with task-specific reward functions, not from a human app. A.1 (see Pitfalls for a conflict with §4.1). Real demonstrations come from human teaching, sampled at 2 Hz, with a sparse reward of +1 on success and -1 otherwise app. A.3.

# GCAD training step  [§3.2] [eq. 4] [app. A.5]
# Input:  obs (B, T_o, ...), actions (B, T_a, 6), return_to_go (B, T_o)
# Output: scalar loss

def training_step(obs, actions, return_to_go):
    # Encode each observation step independently  [app. A.4]
    obs_emb = visual_encoder(obs)                 # (B, T_o, d)
    # Embed the return-to-go values  [§3.2]
    rtg_emb = rtg_embedding(return_to_go)         # (B, T_o, d)
    # Draw one denoising step k per sample; the law is not stated  [§3.2]
    k = random_step(batch_size=B)
    # Draw Gaussian noise with the shape of the action chunk  [§3.2] [eq. 4]
    noise = sample_normal(shape=actions.shape)
    # Add the noise to the clean actions A^0 as written in the paper  [eq. 4]
    noisy_actions = actions + noise
    # Predict the noise from observations, return-to-go, actions and k  [eq. 4]
    predicted = noise_net(obs_emb, rtg_emb, noisy_actions, k)
    # Mean squared error between predicted and true noise  [eq. 4]
    return mse(predicted, noise)

# GCAD inference: K denoising updates, then execute T_a actions  [§3.2] [eq. 3]
# Input:  obs_history (T_o steps), rtg_history (T_o values)
# Output: action chunk (T_a, 6)

def predict_actions(obs_history, rtg_history):
    # Start the action chunk from Gaussian noise  [§3.2]
    actions = sample_normal(shape=(T_a, 6))
    # Apply the update from step K down to step 1  [§3.2] [eq. 3]
    for k in range(K, 0, -1):
        # Predict the noise from observations, return-to-go and actions  [eq. 3]
        eps = noise_net(obs_history, rtg_history, actions, k)
        # Schedule coefficients alpha, gamma, sigma: formulas not given  [app. A.4]
        actions = alpha(k) * (actions - gamma(k) * eps
                              + sample_normal(sigma(k)))
    # Execute all returned actions without re-planning  [app. A.5]
    return actions

Hyperparameters

Results

Author-reported. Setup: Franka Emika Panda in Robosuite/MuJoCo, four simulated layouts (Basic, Empty, Broad, Surround) §4.1, fig. 3. Every method uses the same VLM planner, so the table compares only the low-level policy §4.2. Distance error is the weighted metric of eq. 5; the paper's table reports negative values and does not explain the sign or the units tab. 1.

Simulation tab. 1. Cells are Dist. E / SR.

Text claims: GCAD has the highest average SR (70%) and lowest distance error §4.2. The paper says transformer diffusion beats CNN diffusion, and BC and DT fail to learn successful trajectories §4.2.

Real robot, four daily-life setups tab. 2. Ten trials each.

Real robot, object generalisation tab. 3. Edge = push to table edge; Wall = push to wall.

Irregular and curved table edges, with the pushing model reused unchanged, reached 20% (2/10) and 30% (3/10) app. A.9.

Task planner comparison, heuristic versus VLM planners across motion planners, in simulation fig. 4. Average SR:

• Heuristic planner + Behaviour Cloning: 0.00; + Diffusion Policy-T: 0.13; + GCAD: 0.35.
• Finetuned-VLM + Behaviour Cloning: 0.00; + Diffusion Policy-T: 0.23.
• DexDiff: 0.70.

Generalisation in simulation (object position, pose, size, friction and gripper start randomised) §4.4, fig. 5. The paper reports some degradation from tab. 1 but no separate numbers in the text §4.4.

Headline: the abstract says "47% higher success rate" in simulation. That matches 0.70 minus 0.23 from tab. 1, an absolute gap of 47 percentage points against DP-T (about three times the baseline rate).

Paper vs code

No official code.

Searched: the arXiv abstract and body, which give no code link. The LaTeX source has a commented-out line, "Code and training details will be publicly available" (not in the compiled paper). The project page (exdiff.github.io) has no code link. GitHub repository search for "DexDiff", "extrinsic dexterity", "ungraspable diffusion" and "GCAD diffusion robot extrinsic" found no repository by the authors. Hits were unrelated Android dex-diff tools, DexDiffuser repositories by other owners, and extrinsic-dexterity code from other groups (Wenxuan-Zhou/ungraspable, PKU-EPIC/DAPL, tangty11/ExDex). GitHub user search for "Xuguang Lan" returned no users.

Pitfalls

• Symbol clashes. γ is the learning rate in the text §3.2 and the table lists lr = 1e-4 tab. 4, yet γ also appears as a step coefficient in eq. 3. α is the iteration coefficient in eq. 3, yet α1 and α2 are distance weights in eq. 5 and are also listed under training settings tab. 4. Decide which is the optimiser learning rate and which is the distance weight before implementing.
• No noise schedule formula. The forward process and the schedule are not written out. Eq. 4 adds the noise to A^0 with no scaling coefficient, which a standard DDPM does not do eq. 4 eq. 2. The schedule family is named only as "square cosine" app. A.4. An implementer must choose the forward process and the coefficients, and should record that choice.
• K and the sampler are not stated. The number of denoising steps at training and inference is missing, as is the sampling law for k.
• Return-to-go is undefined. The paper says return-to-go values come from goal-conditioned grasp poses and relational tuples §3.1, but gives no formula, reward scale or reference frame. The simulation reward is described only in words app. A.1. The sparse ±1 reward is given only for the real robot app. A.3.
• Relational-tuple filter is undefined. The parser is unnamed and the rule that uses tuples to filter trajectories is not given §3.1.
• Demonstration source conflicts. §4.1 says simulated trajectories combine reinforcement learning with human demonstration §4.1. The appendix says the simulated data comes only from SAC policies app. A.1.
• Planning frequency is unclear. The text says the VLM perceives "for each scenario" §3, but does not say whether it runs once per episode or repeatedly.
• Receding-horizon wording. The appendix calls the scheme receding-horizon control but describes executing T_a actions "without re-planning" app. A.5. Implement the explicit description, which is open-loop chunk execution, and do not assume standard replanning.
• Very short chunks. The best horizons are T_o = 2 and T_a = 2 app. A.6. The parameter-study curves in fig. 11 are not tabulated in the text, so the trade-off cannot be checked numerically.
• Gripper action missing. Actions are 6-D gripper motion app. A.1 tab. 4. How grasp closing is commanded is not stated.
• Percent labels. The headers of tab. 2 and fig. 4 say "SR(%)" and "Avg.SR(%)", but the values are fractions (0.70 means 70%).
• Distance metric. α1, α2, ΔT and ΔR are not defined, and the sign convention of the negative values is not explained tab. 1 eq. 5.
• Small trial counts. Real results use 10 trials per scenario and simulation uses 35 §4.1. In the real tests, one trial is 10 percentage points of a scenario's rate.
• Hidden cost. The paper says the transformer model "requires much demonstration", is computationally expensive, and has decision latency that affects smoothness on the real robot §5.
• Real vs sim setup. Simulation uses a Franka Panda §4.1; the real experiments use a UR5e with a Robotiq gripper §4.5 app. A.3. Real-world policies are trained on real demonstrations app. A.3, and the paper does not report a sim-to-real transfer result.
• Figure numbers. Figure numbers here follow LaTeX source order (commented-out figures excluded). Check them against the compiled PDF before citing a figure number.
• Human priors in the real world. The paper says the VLM was guided with human prior knowledge about which real surfaces act as walls or table edges, such as a bookshelf or drawer interior §4.5. Generalisation to new structure types is therefore partly human-specified.

Lineage

• Builds on: Diffusion Policy (cited as [diffusion]), which supplies the action-diffusion backbone and the DP-C and DP-T baselines; DDPM (cited as [ho2020denoising]), which supplies the loss in eq. 2; and the "Square Cosine Schedule" from "Improved denoising diffusion probabilistic models" (cited as [nichol2021improved]) app. A.4.
• Return-to-go conditioning is compared against Decision Transformer (cited as [decisiontransformer]), which is a baseline. The paper does not say that its return-to-go formulation follows DT.
• Compared against: Behaviour Cloning, Decision Transformer and Diffusion Policy as low-level baselines §4.1; a YOLOX-plus-threshold heuristic as the high-level baseline §4.1.
• Related but not compared: extrinsic-dexterity work for non-prehensile and ungraspable manipulation §2.
• Superseded by: not known from content in this wiki.