---
kind: concept
slug: motion-retargeting
title: Motion retargeting
summary: >-
  Mapping motion from one body onto a robot whose joints differ. Variants: per-step keypoint or fingertip-vector optimization for dexterous hands, frame-wise or clip-level IK with feasibility filtering or smoothing for humanoids, task-space QP with a fixed tracker-to-joint assignment for VR teleoperation, and a learned human-to-cobot mapping refined by IK.
tags: [retargeting, teleoperation, dexterous-manipulation, humanoid, robot-learning, inverse-kinematics]
papers: ["1910.03135", "2307.04577", "2408.07295", "2411.04005", "2411.07534", "2412.11360", "2412.14172", "2506.04982"]
related: [imitation-learning, learning-from-demonstration, robot-simulation, teleoperation, sim-to-real-transfer, inverse-kinematics, quadratic-programming, neural-symbolic-integration]
status: draft
updated_at: 2026-11-01
---

## Overview

Motion retargeting takes the motion of a source body, such as a human hand
tracked by a camera, a VR controller, or a human motion-capture clip, and
computes joint commands for a target body with different proportions and
joints. The problem is to keep the motion's shape while respecting the
target's limits. This page draws on eight cards. Three of them, AnyTeleop,
GEX and DexPilot, solve the problem per step for a dexterous hand by optimizing
joint angles against finger keypoints or fingertip vectors
[arxiv:2307.04577 §4.3] [arxiv:2506.04982 §3.2] [arxiv:1910.03135 §7.1].
The humanoid cards use inverse kinematics (IK) on reference motions, with
feasibility filtering, smoothing or a learned controller
[arxiv:2408.07295 §4.1] [arxiv:2412.14172 §3.2]. The VR teleoperation card
uses a task-space quadratic program [arxiv:2411.07534 §II]. The human-to-cobot
card learns a mapping and then refines it with IK [arxiv:2412.11360 §IV.B].
The object-centric dexterity card skips per-finger retargeting and transfers
only the wrist [arxiv:2411.04005 §1].

In all of these papers retargeting is one stage of a larger pipeline, not the
whole of it. In AnyTeleop it is the hand stage of a teleoperation system. The
arm is not retargeted; the wrist pose from detection goes to a separate motion
generator that plans collision-free arm trajectories [arxiv:2307.04577 §4.4].
In GEX it links a glove worn by the operator to the robot hand
[arxiv:2506.04982 §3.2]. In the humanoid paper retargeting prepares training
references for a learned controller [arxiv:2408.07295 §4.1]. In UH-1 it
produces the training data for a text-to-motion model [arxiv:2412.14172 §3.2].
In DexPilot it is the kinematic step between hand tracking and the controllers:
a separate motion generator computes the palm target from the tracked hand
[arxiv:1910.03135 §7.2].

## Variants

### Keypoint-fitting optimization for dexterous hands

Each time step, find robot joint angles that place the robot's finger keypoints
close to scaled human keypoint vectors. A penalty keeps each solution near the
previous one, and joint bounds constrain the result [arxiv:2307.04577 eq. 1]
[arxiv:2307.04577 §4.3]. The paper's stated form uses squared errors. Its
scaling factor α and smoothing weight β are not given values in the text
[arxiv:2307.04577 §4.3]. Card: [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](/papers/2307.04577).

Its reported effect is at the system level. Retargeting, detection, fusion and
arm motion are evaluated together, so the paper gives no retargeting-only
error or success rate [arxiv:2307.04577 tab. 4] [arxiv:2307.04577 tab. 3].

The mapping from human finger keypoints to robot links is set by hand for each
new hand [arxiv:2307.04577 §4.3]. The paper's system needs this step for every
robot it supports, and gives no default mapping for other hands
[arxiv:2307.04577 §4.3]. This is a cost of the variant, not a separate method.

GEX uses the same per-step form for a three-finger hand worn with an
exoskeleton glove. Its keypoints are the glove's fingertips and palm base
[arxiv:2506.04982 §3.2]. Its objective matches fingertip poses to scaled glove
fingertip poses, adds a smoothing term with weight k, and treats joint limits
as hard bounds. The scaling factor α and the weight k are not stated
[arxiv:2506.04982 eq. 1]. The paper reports no retargeting-only number. Its
results are one real grasp and three simulated tasks, shown qualitatively, with
no success rates and no baseline [arxiv:2506.04982 §4] [arxiv:2506.04982 §4.1].
Card: [GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove](/papers/2506.04982).

### Fingertip-vector matching with switching weights

DexPilot matches vectors rather than keypoint positions. Each frame, vectors
between fingertips, the palm and the thumb on the human hand are matched to the
same vectors on the robot hand. Each target is a unit direction scaled by β
times the human vector's length [arxiv:1910.03135 §7.1]. When a vector is
shorter than a threshold ε, the target becomes a fixed length instead, and its
weight rises to 200 or 400, depending on which vector group it belongs to. A
small regulariser pulls the robot hand toward an open pose
[arxiv:1910.03135 §7.1]. The solve is SLSQP, warm-started from the previous
frame, followed by a first-order low-pass filter [arxiv:1910.03135 §7.1]. The
paper gives β = 1.6 and the regulariser weight, but not ε or the filter cut-off
[arxiv:1910.03135 §7.1]. No official code is
released, so the cost is taken from the paper alone. Card: [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](/papers/1910.03135).

The paper gives no retargeting-only error. It is evaluated inside a full
teleoperation loop with two pilots on 15 tasks, five trials each, where the
appendix reports 9 failed trials out of 150 and completion times from 16 s to
358 s [arxiv:1910.03135 app. completion times]. Its one keypoint figure, 9.7 mm,
is measured on the authors' own validation set and is not a retargeting metric
[arxiv:1910.03135 §6]. The authors say earlier retargeting was not good enough
for precision grasps such as pinches [arxiv:1910.03135 §7.1]. The results in
the card include no baseline numbers.

### Frame-wise inverse kinematics with feasibility filtering for humanoids

Each frame of a reference motion is retargeted to the robot with inverse
kinematics. The paper solves each frame as a nonlinear program in Drake's IK
module, using SNOPT. Stance-foot contact and the closed leg chain are hard
task-space constraints, and upper-body targets enter as soft costs. Frames with
no feasible solution are dropped, and the remaining states are linearly
interpolated in time [arxiv:2408.07295 §4.1]. The reference set mixes motion
capture (AMASS and Reallusion), video demonstrations and optimized spring-mass
trajectories for the robot [arxiv:2408.07295 §4.1]. Card: [Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots](/papers/2408.07295).

Digit's one-degree-of-freedom rotational joints cannot express the spherical
joints of motion capture, so some retargeted motions are only approximate
[arxiv:2408.07295 §4.1]. The card's summary of §4.1 describes no step-to-step
smoothness penalty. Continuity comes from the interpolation over dropped frames.

The paper reports no error for retargeting alone. Its numbers are for the
tracking controller trained on these references, such as a 0.450% simulated
failure rate on mimicry [arxiv:2408.07295 tab. 3].

### Shape fit and smoothed IK over a large video and mocap corpus

UH-1 retargets a corpus of 163,800 clips to a humanoid. First the SMPL body
shape is fitted so the human T-pose matches the robot's T-pose, with each shape
coefficient bounded [arxiv:2412.14172 eq. 4] [arxiv:2412.14172 eq. 5]
[arxiv:2412.14172 app. B.5]. The human joints are then mapped to 12 robot
joints through forward kinematics, and robot joint angles come from IK with a
smoothness term weighted by λ = 0.05 [arxiv:2412.14172 §3.2]
[arxiv:2412.14172 eq. 7] [arxiv:2412.14172 app. B.5]. A PPO tracking policy
then makes the result physically feasible [arxiv:2412.14172 app. B.6]. Card: [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172).

The paper's assumption is that human and humanoid skeletons are close enough
for a shape fit to work [arxiv:2412.14172 §3.2]. Its results use the same
retargeting for the baselines, so the comparison measures generation, not the
retargeting step [arxiv:2412.14172 §5.1]. No retargeting-only error is reported.

### Task-space QP with a fixed tracker-to-joint assignment for VR teleoperation

Each tracked body part drives a fixed group of robot joints. The robot joint
vector is split into torso pitch, torso yaw, neck, and left and right arm
groups, and each tracker is assigned to some of these groups
[arxiv:2411.07534 §II] [arxiv:2411.07534 fig. 1]. For each task, the Jacobian
is restricted to the assigned columns so that unassigned joints do not move
[arxiv:2411.07534 §II] [arxiv:2411.07534 eq. 2]. One QP solve per control
step then gives joint velocities from weighted task errors, regularizers, and
a relaxed-barrier term for each self-collision pair [arxiv:2411.07534 eq. 3]
[arxiv:2411.07534 eq. 4]. The barrier is soft, so the solver returns a best
effort instead of rejecting a command [arxiv:2411.07534 §I]. Card: [Effective Virtual Reality Teleoperation of an Upper-body Humanoid with Modified Task Jacobians and Relaxed Barrier Functions for Self-Collision Avoidance](/papers/2411.07534).

The paper's results are qualitative. The setup is one humanoid in a table-top
scene, with box packing and a two-handed handover, and the authors say
self-collision handling helped safety and responsiveness. No collision counts,
clearances or timings are reported, and no baselines are run
[arxiv:2411.07534 §III] [arxiv:2411.07534 §IV]. The paper's IK cost is the only
place its controller is written down; the barrier's parameters and the gains
are not given [arxiv:2411.07534 eq. 3].

### Learned human-to-cobot mapping with IK refinement

A learned model predicts the human hip, shoulder and elbow positions from the
wrist position at each time step. A fixed one-to-one joint correspondence then
maps the human angles to the cobot's four arm joints. A learned forward model
of those joints gives the end-effector position, and an iterative IK step
moves the joints toward the target [arxiv:2412.11360 §IV.B] [arxiv:2412.11360 eq. 6].
The loop stops when the position error falls below 0.01 m or after 10,000
iterations [arxiv:2412.11360 §IV.B]. Card: [Visual IRL for Human-Like Robotic Manipulation](/papers/2412.11360).

The paper does not give the correspondence as an equation, and it does not say how the human joint
angles are computed from the predicted joint points [arxiv:2412.11360 §IV.B].

The baselines are RRT and RRT-connect, which find collision-free paths without
imitating the human's motion [arxiv:2412.11360 §V.D]. Against them the author-
reported jerkiness is lower (914 vs 265 degrees for onion sorting, and 1,334 vs
359 for bottle pouring) and the time is shorter [arxiv:2412.11360 tab. 5]
[arxiv:2412.11360 tab. 6]. These are comparisons with planners, not with an IK
retargeter, so they do not show that the learned mapping beats one.

### Wrist-only transfer with no finger retargeting

The object-centric dexterity paper states that finger trajectories transfer
poorly between human and robot hands, and that wrist motion transfers better
[arxiv:2411.04005 §1]. Its planner outputs 6-DoF wrist poses for each hand,
trained on human wrist motion, and a PPO controller learns the finger joints
[arxiv:2411.04005 §4.1] [arxiv:2411.04005 §4.2]. The paper states that per-frame
IK retargeting does not guarantee task success across embodiments
[arxiv:2411.04005 §2]. Card: [Object-Centric Dexterous Manipulation from Human Motion Data](/papers/2411.04005).

No retargeting step is measured. The planner's accuracy is reported as a
translation error in cm and a second error, OE, whose unit the paper does not
state [arxiv:2411.04005 tab. 1]. Simulation results are means over 10 seeds and
real-world results over 20 trials [arxiv:2411.04005 §5]. The card's Paper vs
code section finds that the released repository does not contain the planner.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Dexterous hand driven by a hand keypoint detector, live | Keypoint-fitting optimization with a hand-set mapping | The paper's real-robot runs use an Allegro hand on an XArm6 arm [arxiv:2307.04577 §6.2]; its simulated settings use SAPIEN, with the hand model not named in the text [arxiv:2307.04577 §7.1] |
| Humanoid reference motions from mocap or video, used for offline training | Frame-wise IK with feasibility filtering | Contact and closed-chain constraints are hard, and frames with no feasible solution are dropped [arxiv:2408.07295 §4.1] |
| Large, mixed video and mocap corpus for a humanoid, with a body-shape difference | Shape fit followed by smoothed IK, as in UH-1 | The shape-fit and IK settings are given in the appendix [arxiv:2412.14172 app. B.5]; the paper gives no retargeting-only accuracy [arxiv:2412.14172 §5.1] |
| Glove-free tracking from several RGB-D cameras, driving an Allegro hand on an arm with pinch-style grasps | Fingertip-vector matching with switching weights, as in DexPilot | The authors say earlier retargeting was not good enough for precision grasps such as pinches, and the weight rises for short vectors [arxiv:1910.03135 §7.1]. Their study reports completion times and failures, with no retargeting-only error [arxiv:1910.03135 app. completion times] |
| Low-cost, fully motor-driven three-finger hand with a matching glove | Per-step keypoint optimization with a smoothing term, as in GEX | The authors state a cost of USD 600 per device and under four hours to build [arxiv:2506.04982 §1]. The paper gives no retargeting accuracy, latency or success rate, so the choice cannot rest on measured performance [arxiv:2506.04982 §4] |
| VR trackers on a humanoid with more joints than trackers | Task-space QP with a fixed tracker-to-joint assignment | Each tracker drives its own joint group, and self-collision is handled inside the solve [arxiv:2411.07534 §I] [arxiv:2411.07534 §II]. No quantitative evidence is given |
| Human arm to a cobot with different joints and limb lengths | Learned mapping then IK refinement, as in Visual IRL | Lower jerkiness than RRT in the authors' two tasks [arxiv:2412.11360 tab. 5] [arxiv:2412.11360 tab. 6]. Results are on one Sawyer and one LBR iisy, with 10 trials each [arxiv:2412.11360 §V.D] |
| Dexterous bimanual task where the wrist matters more than finger shape | Transfer wrist poses only; let RL learn the fingers | The authors argue wrist motion transfers better [arxiv:2411.04005 §1]. Their task results are simulated and the real-world trials are few, and no alternative retargeting is measured [arxiv:2411.04005 §5] |
| Arm motion for a whole-body or arm target | Not covered by these cards | AnyTeleop plans the arm with a separate collision-free motion generator [arxiv:2307.04577 §4.4]; the humanoid paper does not describe a separate arm planner in the cited sections |
| A new hand or robot with no mapping yet | Budget time for manual keypoint correspondence | No default mapping is given for other hands [arxiv:2307.04577 §4.3] |
| Joint types in the source motion differ from the robot's | Expect approximate references, and check the dropped-frame rate | One-DoF joints cannot express spherical joints, so some motions are approximate [arxiv:2408.07295 §4.1] |
| Depth camera not available | Keep the same retargeting; expect a noisier wrist input | Wrist position from RGB only is less accurate than with depth, but the paper gives no figure for it [arxiv:2307.04577 §4.1] |
| Demonstrations for imitation learning from teleoperated hands | Keypoint-fitting retargeting feeding the demonstrations | The paper reports higher imitation success than a simulator-specific baseline on 5 of 6 settings, with a system-level comparison [arxiv:2307.04577 tab. 3] |

## Implementation sketch

The common core is one optimization per time step. The paper's eq. 1 gives the
objective for the keypoint variant. Scale the human keypoints, minimize keypoint
error plus a step-size penalty, and enforce joint bounds [arxiv:2307.04577 eq. 1].
The frame-wise IK variant is the same loop with the step-size term dropped and
contact constraints added, and with infeasible frames removed afterwards
[arxiv:2408.07295 §4.1]. The shape-fit variant adds a body-shape step before
the loop and keeps a smoothness term over neighbouring frames
[arxiv:2412.14172 app. B.5].

```pseudocode
# Retarget one frame of human finger keypoints  [arxiv:2307.04577 §4.3]
# Input:  human keypoint vectors v[i], previous robot joints prev
# Output: robot hand joint vector q for this frame

def retarget_step(v, prev, alpha, beta):
    # Minimize over q: sum over keypoints i of
    #   || alpha * v[i] - f_i(q) ||^2 + beta * || q - prev ||^2
    # where f_i is the forward kinematics of robot keypoint i  [eq. 1]
    # subject to lower <= q <= upper  [eq. 1]
    q = bounded_minimize(
        objective(v, alpha, beta, prev),
        lower=lower,
        upper=upper,
        start=prev,
    )
    return q
```

Where variants differ: the norm and loss on the keypoint term, the solver, and
how α and β are set. The paper gives no values for α or β [arxiv:2307.04577 §4.3].
GEX's eq. 1 has the same three parts, a pose-matching term, a smoothing term and
hard joint bounds, with α and k unstated [arxiv:2506.04982 eq. 1]. Its footnote
names the dex-retargeting library [arxiv:2506.04982 §3.2]. A check of the
released config, recorded on the GEX card, found that it selects that library's
DexPilot optimizer rather than eq. 1 as written.
DexPilot's own cost [arxiv:1910.03135 §7.1] has no alpha-scaled keypoint term.
It matches vectors, scales the target by β, and adds a switching weight and an
open-hand pull, as the variant section above describes.
For humanoid references, the per-frame problem also carries the contact and
closed-chain constraints of [arxiv:2408.07295 §4.1]. The card does not give its
objective in closed form, so the sketch does not reproduce it.

The learned-mapping variant replaces the per-frame objective with a loss that
trades position error against the change from the initial angles
[arxiv:2412.11360 eq. 6]. Its initial angles come from the fixed joint
correspondence, not from an optimizer [arxiv:2412.11360 §IV.B].

## Contradictions

Papers disagree on whether per-frame IK retargeting is enough. The object-centric
dexterity paper says it does not guarantee task success across embodiments
[arxiv:2411.04005 §2]. AnyTeleop, DexPilot and UH-1 rely on per-frame or
per-clip retargeting without measuring it on its own, and the cards do not test
one approach against the other [arxiv:2307.04577 tab. 4]
[arxiv:1910.03135 app. completion times] [arxiv:2412.14172 §5.1]. This is a difference
in stance, not in numbers, and this page does not resolve it.

Within the AnyTeleop card, the paper's stated objective and the optimizer in its
released retargeting library differ in the loss and in the smoothing term. The
card records these differences as "Paper vs code" [arxiv:2307.04577 §4.3]. The
paper also describes output commands as smooth, but never mentions a filter,
while the released configs apply one [arxiv:2307.04577 §4].

The UH-1 card records paper-against-code differences in its retargeting. The
paper's appendix describes a signed sum of second differences for smoothing, but
the released code uses the absolute difference from the mean of neighbouring
frames [arxiv:2412.14172 app. B.5] [code: retarget/retarget_motion_smplx.py#L233-L238].
Its §3.2 gives 12 retargeted robot joints, while Appendix C.1 describes 27 DoF
plus a root state. The card records the released output as 34 values
[arxiv:2412.14172 §3.2] [arxiv:2412.14172 app. C.1].

The GEX paper's hand and glove counts disagree with a check of its released
robot models, recorded on the GEX card. The paper gives 11 hand DoF, with a
three-DoF thumb, 12 glove DoF, and 23 in total [arxiv:2506.04982 §2.1]
[arxiv:2506.04982 §2.2] [abstract]. The released models have 10 and 12 revolute
joints, 22 in total, and the thumb chain has four. The paper's numbers are the
ones cited above; this disagreement is paper against code, not paper against paper.

## Papers

- [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](/papers/1910.03135): per-step fingertip-vector retargeting with switching weights and SLSQP, inside a four-camera bare-hand teleoperation system; end-to-end results only.
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](/papers/2307.04577): per-step keypoint-fitting retargeting for a dexterous hand, inside a vision-based teleoperation stack.
- [Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots](/papers/2408.07295): frame-wise inverse-kinematics retargeting of mocap, video and generated references for a humanoid, feeding a learned whole-body controller.
- [Object-Centric Dexterous Manipulation from Human Motion Data](/papers/2411.04005): wrist-only transfer from human motion, with a planner and a PPO finger controller; argues against per-frame IK retargeting for task success.
- [Effective Virtual Reality Teleoperation of an Upper-body Humanoid with Modified Task Jacobians and Relaxed Barrier Functions for Self-Collision Avoidance](/papers/2411.07534): task-space QP retargeting with a fixed tracker-to-joint assignment and soft self-collision avoidance; qualitative results only.
- [Visual IRL for Human-Like Robotic Manipulation](/papers/2412.11360): learned human-to-cobot joint mapping with a fixed correspondence and IK refinement; compared with RRT planners.
- [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172): shape fit and smoothed IK over a large video and mocap corpus, feeding a text-to-motion model and a tracking policy.
- [GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove](/papers/2506.04982): low-cost three-finger hand and glove kit, with per-step glove-to-hand retargeting; qualitative results only.