Motion retargeting

Overview

Motion retargeting takes the motion of a source body, such as a human hand tracked by a camera, a VR controller, or a human motion-capture clip, and computes joint commands for a target body with different proportions and joints. The problem is to keep the motion's shape while respecting the target's limits. This page draws on eight cards. Three of them, AnyTeleop, GEX and DexPilot, solve the problem per step for a dexterous hand by optimizing joint angles against finger keypoints or fingertip vectors 2307.04577 §4.3 2506.04982 §3.2 1910.03135 §7.1. The humanoid cards use inverse kinematics (IK) on reference motions, with feasibility filtering, smoothing or a learned controller 2408.07295 §4.1 2412.14172 §3.2. The VR teleoperation card uses a task-space quadratic program 2411.07534 §II. The human-to-cobot card learns a mapping and then refines it with IK 2412.11360 §IV.B. The object-centric dexterity card skips per-finger retargeting and transfers only the wrist 2411.04005 §1.

In all of these papers retargeting is one stage of a larger pipeline, not the whole of it. In AnyTeleop it is the hand stage of a teleoperation system. The arm is not retargeted; the wrist pose from detection goes to a separate motion generator that plans collision-free arm trajectories 2307.04577 §4.4. In GEX it links a glove worn by the operator to the robot hand 2506.04982 §3.2. In the humanoid paper retargeting prepares training references for a learned controller 2408.07295 §4.1. In UH-1 it produces the training data for a text-to-motion model 2412.14172 §3.2. In DexPilot it is the kinematic step between hand tracking and the controllers: a separate motion generator computes the palm target from the tracked hand 1910.03135 §7.2.

Variants

Keypoint-fitting optimization for dexterous hands

Each time step, find robot joint angles that place the robot's finger keypoints close to scaled human keypoint vectors. A penalty keeps each solution near the previous one, and joint bounds constrain the result 2307.04577 eq. 1 2307.04577 §4.3. The paper's stated form uses squared errors. Its scaling factor α and smoothing weight β are not given values in the text 2307.04577 §4.3. Card: AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System.

Its reported effect is at the system level. Retargeting, detection, fusion and arm motion are evaluated together, so the paper gives no retargeting-only error or success rate 2307.04577 tab. 4 2307.04577 tab. 3.

The mapping from human finger keypoints to robot links is set by hand for each new hand 2307.04577 §4.3. The paper's system needs this step for every robot it supports, and gives no default mapping for other hands 2307.04577 §4.3. This is a cost of the variant, not a separate method.

GEX uses the same per-step form for a three-finger hand worn with an exoskeleton glove. Its keypoints are the glove's fingertips and palm base 2506.04982 §3.2. Its objective matches fingertip poses to scaled glove fingertip poses, adds a smoothing term with weight k, and treats joint limits as hard bounds. The scaling factor α and the weight k are not stated 2506.04982 eq. 1. The paper reports no retargeting-only number. Its results are one real grasp and three simulated tasks, shown qualitatively, with no success rates and no baseline 2506.04982 §4 2506.04982 §4.1. Card: GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove.

Fingertip-vector matching with switching weights

DexPilot matches vectors rather than keypoint positions. Each frame, vectors between fingertips, the palm and the thumb on the human hand are matched to the same vectors on the robot hand. Each target is a unit direction scaled by β times the human vector's length 1910.03135 §7.1. When a vector is shorter than a threshold ε, the target becomes a fixed length instead, and its weight rises to 200 or 400, depending on which vector group it belongs to. A small regulariser pulls the robot hand toward an open pose 1910.03135 §7.1. The solve is SLSQP, warm-started from the previous frame, followed by a first-order low-pass filter 1910.03135 §7.1. The paper gives β = 1.6 and the regulariser weight, but not ε or the filter cut-off 1910.03135 §7.1. No official code is released, so the cost is taken from the paper alone. Card: DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System.

The paper gives no retargeting-only error. It is evaluated inside a full teleoperation loop with two pilots on 15 tasks, five trials each, where the appendix reports 9 failed trials out of 150 and completion times from 16 s to 358 s 1910.03135 app. completion times. Its one keypoint figure, 9.7 mm, is measured on the authors' own validation set and is not a retargeting metric 1910.03135 §6. The authors say earlier retargeting was not good enough for precision grasps such as pinches 1910.03135 §7.1. The results in the card include no baseline numbers.

Frame-wise inverse kinematics with feasibility filtering for humanoids

Each frame of a reference motion is retargeted to the robot with inverse kinematics. The paper solves each frame as a nonlinear program in Drake's IK module, using SNOPT. Stance-foot contact and the closed leg chain are hard task-space constraints, and upper-body targets enter as soft costs. Frames with no feasible solution are dropped, and the remaining states are linearly interpolated in time 2408.07295 §4.1. The reference set mixes motion capture (AMASS and Reallusion), video demonstrations and optimized spring-mass trajectories for the robot 2408.07295 §4.1. Card: Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots.

Digit's one-degree-of-freedom rotational joints cannot express the spherical joints of motion capture, so some retargeted motions are only approximate 2408.07295 §4.1. The card's summary of §4.1 describes no step-to-step smoothness penalty. Continuity comes from the interpolation over dropped frames.

The paper reports no error for retargeting alone. Its numbers are for the tracking controller trained on these references, such as a 0.450% simulated failure rate on mimicry 2408.07295 tab. 3.

Shape fit and smoothed IK over a large video and mocap corpus

UH-1 retargets a corpus of 163,800 clips to a humanoid. First the SMPL body shape is fitted so the human T-pose matches the robot's T-pose, with each shape coefficient bounded 2412.14172 eq. 4 2412.14172 eq. 5 2412.14172 app. B.5. The human joints are then mapped to 12 robot joints through forward kinematics, and robot joint angles come from IK with a smoothness term weighted by λ = 0.05 2412.14172 §3.2 2412.14172 eq. 7 2412.14172 app. B.5. A PPO tracking policy then makes the result physically feasible 2412.14172 app. B.6. Card: Learning from Massive Human Videos for Universal Humanoid Pose Control.

The paper's assumption is that human and humanoid skeletons are close enough for a shape fit to work 2412.14172 §3.2. Its results use the same retargeting for the baselines, so the comparison measures generation, not the retargeting step 2412.14172 §5.1. No retargeting-only error is reported.

Task-space QP with a fixed tracker-to-joint assignment for VR teleoperation

Each tracked body part drives a fixed group of robot joints. The robot joint vector is split into torso pitch, torso yaw, neck, and left and right arm groups, and each tracker is assigned to some of these groups 2411.07534 §II 2411.07534 fig. 1. For each task, the Jacobian is restricted to the assigned columns so that unassigned joints do not move 2411.07534 §II 2411.07534 eq. 2. One QP solve per control step then gives joint velocities from weighted task errors, regularizers, and a relaxed-barrier term for each self-collision pair 2411.07534 eq. 3 2411.07534 eq. 4. The barrier is soft, so the solver returns a best effort instead of rejecting a command 2411.07534 §I. Card: Effective Virtual Reality Teleoperation of an Upper-body Humanoid with Modified Task Jacobians and Relaxed Barrier Functions for Self-Collision Avoidance.

The paper's results are qualitative. The setup is one humanoid in a table-top scene, with box packing and a two-handed handover, and the authors say self-collision handling helped safety and responsiveness. No collision counts, clearances or timings are reported, and no baselines are run 2411.07534 §III 2411.07534 §IV. The paper's IK cost is the only place its controller is written down; the barrier's parameters and the gains are not given 2411.07534 eq. 3.

Learned human-to-cobot mapping with IK refinement

A learned model predicts the human hip, shoulder and elbow positions from the wrist position at each time step. A fixed one-to-one joint correspondence then maps the human angles to the cobot's four arm joints. A learned forward model of those joints gives the end-effector position, and an iterative IK step moves the joints toward the target 2412.11360 §IV.B 2412.11360 eq. 6. The loop stops when the position error falls below 0.01 m or after 10,000 iterations 2412.11360 §IV.B. Card: Visual IRL for Human-Like Robotic Manipulation.

The paper does not give the correspondence as an equation, and it does not say how the human joint angles are computed from the predicted joint points 2412.11360 §IV.B.

The baselines are RRT and RRT-connect, which find collision-free paths without imitating the human's motion 2412.11360 §V.D. Against them the author- reported jerkiness is lower (914 vs 265 degrees for onion sorting, and 1,334 vs 359 for bottle pouring) and the time is shorter 2412.11360 tab. 5 2412.11360 tab. 6. These are comparisons with planners, not with an IK retargeter, so they do not show that the learned mapping beats one.

Wrist-only transfer with no finger retargeting

The object-centric dexterity paper states that finger trajectories transfer poorly between human and robot hands, and that wrist motion transfers better 2411.04005 §1. Its planner outputs 6-DoF wrist poses for each hand, trained on human wrist motion, and a PPO controller learns the finger joints 2411.04005 §4.1 2411.04005 §4.2. The paper states that per-frame IK retargeting does not guarantee task success across embodiments 2411.04005 §2. Card: Object-Centric Dexterous Manipulation from Human Motion Data.

No retargeting step is measured. The planner's accuracy is reported as a translation error in cm and a second error, OE, whose unit the paper does not state 2411.04005 tab. 1. Simulation results are means over 10 seeds and real-world results over 20 trials 2411.04005 §5. The card's Paper vs code section finds that the released repository does not contain the planner.

When to use which

Implementation sketch

The common core is one optimization per time step. The paper's eq. 1 gives the objective for the keypoint variant. Scale the human keypoints, minimize keypoint error plus a step-size penalty, and enforce joint bounds 2307.04577 eq. 1. The frame-wise IK variant is the same loop with the step-size term dropped and contact constraints added, and with infeasible frames removed afterwards 2408.07295 §4.1. The shape-fit variant adds a body-shape step before the loop and keeps a smoothness term over neighbouring frames 2412.14172 app. B.5.

# Retarget one frame of human finger keypoints  [arxiv:2307.04577 §4.3]
# Input:  human keypoint vectors v[i], previous robot joints prev
# Output: robot hand joint vector q for this frame

def retarget_step(v, prev, alpha, beta):
    # Minimize over q: sum over keypoints i of
    #   || alpha * v[i] - f_i(q) ||^2 + beta * || q - prev ||^2
    # where f_i is the forward kinematics of robot keypoint i  [eq. 1]
    # subject to lower <= q <= upper  [eq. 1]
    q = bounded_minimize(
        objective(v, alpha, beta, prev),
        lower=lower,
        upper=upper,
        start=prev,
    )
    return q

Where variants differ: the norm and loss on the keypoint term, the solver, and how α and β are set. The paper gives no values for α or β 2307.04577 §4.3. GEX's eq. 1 has the same three parts, a pose-matching term, a smoothing term and hard joint bounds, with α and k unstated 2506.04982 eq. 1. Its footnote names the dex-retargeting library 2506.04982 §3.2. A check of the released config, recorded on the GEX card, found that it selects that library's DexPilot optimizer rather than eq. 1 as written. DexPilot's own cost 1910.03135 §7.1 has no alpha-scaled keypoint term. It matches vectors, scales the target by β, and adds a switching weight and an open-hand pull, as the variant section above describes. For humanoid references, the per-frame problem also carries the contact and closed-chain constraints of 2408.07295 §4.1. The card does not give its objective in closed form, so the sketch does not reproduce it.

The learned-mapping variant replaces the per-frame objective with a loss that trades position error against the change from the initial angles 2412.11360 eq. 6. Its initial angles come from the fixed joint correspondence, not from an optimizer 2412.11360 §IV.B.

Contradictions

Papers disagree on whether per-frame IK retargeting is enough. The object-centric dexterity paper says it does not guarantee task success across embodiments 2411.04005 §2. AnyTeleop, DexPilot and UH-1 rely on per-frame or per-clip retargeting without measuring it on its own, and the cards do not test one approach against the other 2307.04577 tab. 4 1910.03135 app. completion times 2412.14172 §5.1. This is a difference in stance, not in numbers, and this page does not resolve it.

Within the AnyTeleop card, the paper's stated objective and the optimizer in its released retargeting library differ in the loss and in the smoothing term. The card records these differences as "Paper vs code" 2307.04577 §4.3. The paper also describes output commands as smooth, but never mentions a filter, while the released configs apply one 2307.04577 §4.

The UH-1 card records paper-against-code differences in its retargeting. The paper's appendix describes a signed sum of second differences for smoothing, but the released code uses the absolute difference from the mean of neighbouring frames 2412.14172 app. B.5 retarget_motion_smplx.py. Its §3.2 gives 12 retargeted robot joints, while Appendix C.1 describes 27 DoF plus a root state. The card records the released output as 34 values 2412.14172 §3.2 2412.14172 app. C.1.

The GEX paper's hand and glove counts disagree with a check of its released robot models, recorded on the GEX card. The paper gives 11 hand DoF, with a three-DoF thumb, 12 glove DoF, and 23 in total 2506.04982 §2.1 2506.04982 §2.2 [abstract]. The released models have 10 and 12 revolute joints, 22 in total, and the thumb chain has four. The paper's numbers are the ones cited above; this disagreement is paper against code, not paper against paper.

Papers

• DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System: per-step fingertip-vector retargeting with switching weights and SLSQP, inside a four-camera bare-hand teleoperation system; end-to-end results only.
• AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System: per-step keypoint-fitting retargeting for a dexterous hand, inside a vision-based teleoperation stack.
• Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots: frame-wise inverse-kinematics retargeting of mocap, video and generated references for a humanoid, feeding a learned whole-body controller.
• Object-Centric Dexterous Manipulation from Human Motion Data: wrist-only transfer from human motion, with a planner and a PPO finger controller; argues against per-frame IK retargeting for task success.
• Effective Virtual Reality Teleoperation of an Upper-body Humanoid with Modified Task Jacobians and Relaxed Barrier Functions for Self-Collision Avoidance: task-space QP retargeting with a fixed tracker-to-joint assignment and soft self-collision avoidance; qualitative results only.
• Visual IRL for Human-Like Robotic Manipulation: learned human-to-cobot joint mapping with a fixed correspondence and IK refinement; compared with RRT planners.
• Learning from Massive Human Videos for Universal Humanoid Pose Control: shape fit and smoothed IK over a large video and mocap corpus, feeding a text-to-motion model and a tracking policy.
• GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove: low-cost three-finger hand and glove kit, with per-step glove-to-hand retargeting; qualitative results only.