Paper · 2022-04-26 · cs.RO, cs.CV, cs.LG
From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation
Operators teleoperate a simulated robot hand that is built to match their own hand, using only an iPad camera. The demos are retargeted offline to Schunk, Adroit and Allegro hands and used in DAPG (RL plus behaviour cloning). Author-reported: DAPG beats TRPO-RL on most simulated tasks and transfers more robustly to a real Allegro hand on an XArm-6.
Imitation learningTeleoperationMotion retargetingSim-to-real transferteleoperationdexterous-manipulationimitation-learningsim-to-real
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Teaching a robot hand with many fingers to grab and move things is hard. Learning it from scratch by trial and error takes a huge number of attempts, and the hand often finds odd, unsafe ways to succeed in simulation that do not work on real hardware. A common shortcut is to record a person doing the task, but the usual recording setups need gloves or VR headsets.
This work records people with a single camera on a tablet. The software builds a simulated robot hand that has the same shape as the operator's own hand, so the operator can move it intuitively, without translating motions into a different hand. Once the demonstrations are recorded, they are converted offline into movements for other, commercially available robot hands. A learning method then mixes these demonstrations with ordinary reinforcement learning. The policies behave in a more human-like way, and the authors report that they carry over to a real robot arm with a real hand more reliably than policies trained with reinforcement learning alone.
TL;DR
- Single iPad camera teleoperation of a simulated hand that is customised to the operator's hand shape §IV.
- Demonstrations are retargeted offline to three specified hands (Schunk SVH, Adroit, Allegro) §V.
- Policies are trained with DAPG, which adds a behaviour-cloning-like term to a policy-gradient objective §VI.
- Author-reported: DAPG beats the TRPO RL baseline on most simulated task and hand pairs, and gives higher real-robot success on Relocate and Flip tab. IV tab. V.
- The training code is not in the public repository, so the DAPG and TRPO settings must be taken from the paper, where some values are missing.
Problem
Dexterous hands have many degrees of freedom and discontinuous contacts, so reinforcement learning needs many samples and can find unsafe behaviour §I. Virtual-reality rigs and data gloves give accurate demonstrations but limit how easily data can be collected §I. Vision-based teleoperation is cheaper, but the operator must mentally map their hand onto a robot hand that may have a different shape, for example one with fewer fingers, and demonstrations recorded for one robot can only train that same robot §I. The paper assumes a simulator for both data collection and policy learning §IV-A §VI.
Method
The pipeline has three stages §III fig. 5:
-
Customised hand teleoperation §IV. An RGB-D stream from an iPad front camera at 25 fps is processed by a hand detector §IV-B. The detector gives wrist pose, hand pose and hand shape. The shape is estimated during an initialisation phase and treated as the operator's true hand shape. A simulated hand is then built with the same kinematic skeleton as the operator's hand §IV-C fig. 4. The operator drives this hand in simulation.
- Hand detection uses MediaPipe to find and crop the hand, FrankMocap to regress SMPL-X pose and shape, and a Perspective-n-Point fit to place the wrist in camera coordinates §IV-B.
- The customised hand has 45 joint degrees of freedom, three per finger joint, matching the 15 SMPL-X hand joints §IV-C tab. I.
- Joint targets are low-pass filtered, and the proportional gain is scaled by a confidence value that falls when the current shape estimate moves away from the calibrated shape §IV-C eq. 1. The confidence is modelled as a density under a Gaussian with a diagonal covariance.
-
Multi-robot demonstration translation §V.
- Hand pose retargeting solves, for each frame, a least-squares problem over keypoint positions with joint limits, plus a penalty on the change from the previous frame §V-A eq. 2. The retargeting is done offline, not during teleoperation.
- Actions are then computed from the retargeted joint trajectory. The paper follows DexMV: the trajectory is low-pass filtered and joint torques come from robot inverse dynamics, or equivalently a motor control command §V-B.
-
Demonstration-augmented policy learning §VI. The policy gradient is the sum of a standard on-policy term and a second term over the demonstration pairs. The second term is weighted by a decaying factor and by the largest advantage found in the current on-policy batch. Its coefficients are named λ₀ and λ₁ and are not given numeric values §VI.
# Shape confidence p(t): density of the current shape estimate under a
# Gaussian centred on the calibrated shape, diagonal covariance [§IV-C]
def shape_confidence(shape_now, shape_calib, sigma_diag):
density = gaussian_density(shape_now, shape_calib, sigma_diag)
# The normalisation of p(t) is not stated in the paper [§IV-C]
return normalise(density)# Confidence-weighted PD command for one control step [§IV-C] [eq. 1]
# Inputs: target_pose (45,), current_pose (45,), confidence (scalar)
def pd_command(target_pose, current_pose, confidence):
# Low-pass filter on the target; cutoff is not stated [§IV-C]
target_pose = low_pass(target_pose)
error = target_pose - current_pose
# Proportional term scaled by confidence, plus a derivative term [eq. 1]
torque = confidence * kp * error + kd * (error - prev_error) / dt
return torque# Retargeting one frame from the customised hand to a specified hand [§V-A]
# [eq. 2]. Inputs: keypoints (K, 3) from the customised hand; prev_q (n,)
def retarget_frame(keypoints, prev_q, q_lower, q_upper):
# Keypoints are the tips and joint positions chosen per robot [§V-A]
loss = 0
for each keypoint i:
# Squared keypoint error in the specified hand [eq. 2]
loss = loss + squared_error(keypoints[i], fk_specified(q)[i])
# Penalty on the joint change from the previous frame [eq. 2]
loss = loss + alpha * squared_norm(q - prev_q)
# Solve under joint limits; alpha is not stated in the paper [eq. 2]
q = minimise(loss, lower=q_lower, upper=q_upper)
return q# Demonstration-augmented policy gradient, one update [§VI]
# The display equation is unnumbered in the paper.
# Inputs: rollouts (on-policy pairs), demos (pairs), k (iteration index)
def dapg_gradient(policy, rollouts, demos, advantages, k):
# Vanilla policy-gradient term on rollouts [§VI]
grad = 0
for each (s, a) in rollouts:
grad = grad + grad_log_pi(policy, a, s) * advantages(s, a)
# Demonstration term; weight decays as lambda0 * lambda1^k [§VI]
weight = lambda0 * lambda1 ** k * max_advantage(rollouts)
for each (s, a) in demos:
grad = grad + grad_log_pi(policy, a, s) * weight
return gradKey equations, as printed in the paper:
PD command with confidence eq. 1:
u(t) = p(t) K_p e(t) + k_d \frac{de(t)}{dt}
Retargeting objective with joint limits eq. 2:
\min_{q^R_t} \sum_{i=0}^{N} \left\lVert f_i^C(q^C_t) - f_i^R(q^R_t) \right\rVert^2 + \alpha \left\lVert q^R_t - q^R_{t-1} \right\rVert^2 \quad \text{s.t.} \quad q^R_{\text{lower}} \le q^R_t \le q^R_{\text{upper}}
DAPG gradient §VI, unnumbered display:
g_{aug} = \sum_{(s,a)\in\rho_{\pi_\theta}} \nabla \ln \pi(a \mid s)\, A^{\pi}(s,a) + \sum_{(s,a)\in\rho_{\pi_{\mathrm{demo}}}} \nabla \ln \pi_\theta(a \mid s)\, \lambda_0 \lambda_1^{k} \max_{(s',a')\in\rho_{\pi}} A^{\pi}(s',a')
Hyperparameters
Values stated in the paper or in its figures and tables. Entries marked "not stated" are not given in the paper, and the public repository does not contain the DAPG or TRPO training loop, so they cannot be taken from code either.
| Name | Value | Where | Notes |
|---|---|---|---|
| Camera stream | iPad front RGB-D camera, 25 fps | §IV | Per-frame processing under 30 ms on an RTX 2070 laptop §IV |
| Customised hand DoF | 45 (15 joints × 3) | §IV-C tab. I | Matches SMPL-X hand parameters |
| Confidence covariance Σ | diagonal, values not stated | §IV-C | Code uses σ = 0.2 per dimension; see Paper vs code |
| PD gains Kp, kd | not stated | §IV-C eq. 1 | Only the ablation varies stiffness and damping fig. 7 |
| Low-pass filter cutoff | not stated | §IV-C | Code: 5 Hz at 50 Hz |
| Retargeting penalty α | not stated | §V-A eq. 2 | Code default: 4e-3 |
| Action computation | low-pass filter, then inverse dynamics | §V-B | Filter parameters not stated |
| Demonstrations per task | 50 trajectories | §VII-B | Used to train all policies in the comparison |
| Demonstration rate | about 60 per hour (Relocate) | §III §VII-A | Allegro hand gives about 10 successful demos per hour §I |
| User study | 17 operators | §VII-A | Each task and robot pair tested five times, three-minute warm-up |
| RL baseline | TRPO | §VII-B | On-policy |
| Policy and value network | 2-layer MLP, 32 × 32 | §VII-B | Same for policy and value |
| Trajectories per TRPO step | 200 | §VII-B | |
| DAPG hyperparameters | "same as TRPO" | §VII-B | λ₀ and λ₁ not stated |
| Random seeds | 3 | §VII-B tab. IV | Learning curves show standard deviation over seeds fig. 6 |
| Relocate success (sim) | object within 0.1 "unit length" of target | §VII-B | Unit not defined |
| Flip success (sim) | angle between object and gravity under 5° | §VII-B | |
| Open Door success (sim) | hinge angle above 60° | §VII-B | |
| Relocate object set (sim) | tomato soup can, potted meat can, mustard bottle | §IV-A | From the YCB dataset |
| Relocate randomisation (sim) | initial and target pose both randomised | §IV-A | |
| Flip randomisation (sim) | position and rotation about gravity randomised | §IV-A | |
| Open Door randomisation (sim) | door position randomised | §IV-A | |
| Real robot | Allegro hand on XArm-6, RealSense D435 | §VII-D | ICP estimates object pose from the point cloud |
| Real initial object area | 20 cm square | §VII-D | |
| Real target area (Relocate) | 40 cm square, fixed height | §VII-D | |
| Real Relocate success | xy error within 5 cm | §VII-D | |
| Real Flip success | distance from table top to the highest point of the mug's bottom under 1 cm | §VII-D | Means the mug is nearly upright |
| Real trials | 9 initial/target pairs, three seeds | §VII-D | Same pairs for each policy |
| Real object sets | 3 known objects, 5 novel objects | §VII-D fig. 10 | |
| Sim-to-real augmentation | Gaussian noise on object pose; randomised friction, density and PD parameters | §VII-D | Values not stated |
Results
Author-reported, in simulation with SAPIEN and on a real robot. Simulation results are means and standard deviations over three seeds tab. IV. Real-robot results are over three seeds with nine pairs each tab. V. Baselines are the TRPO RL policy without demonstrations and, in the user study, the same teleoperation task with online retargeting onto the three specified hands.
Teleoperation user study, Relocate. Success is per stage; time is seconds tab. II:
| Robot | S.1 success | S.1 time | S.2 success | S.2 time |
|---|---|---|---|---|
| Schunk | 61.2% | 14.2 s | 30.6% | 30.3 s |
| Adroit | 58.8% | 11.5 s | 28.2% | 37.6 s |
| Allegro | 44.7% | 18.7 s | 16.9% | 42.5 s |
| Customised | 78.9% | 9.1 s | 55.3% | 23.0 s |
Teleoperation user study, Open Door tab. III:
| Robot | S.1 success | S.1 time | S.2 success | S.2 time |
|---|---|---|---|---|
| Schunk | 83.5% | 9.2 s | 60.0% | 20.4 s |
| Adroit | 81.2% | 8.5 s | 61.2% | 18.9 s |
| Allegro | 71.8% | 12.7 s | 41.1% | 23.6 s |
| Customised | 95.3% | 6.2 s | 82.4% | 15.3 s |
Simulated policy success rates, RL (TRPO) versus DAPG, mean ± standard deviation over three seeds tab. IV:
| Robot | Task | RL | DAPG |
|---|---|---|---|
| Schunk | Relocate, tomato soup can | 45.3 ± 4.0 | 85.0 ± 12.3 |
| Schunk | Relocate, potted meat can | 6.7 ± 6.3 | 41.0 ± 20.3 |
| Schunk | Relocate, mustard bottle | 44.0 ± 20.0 | 75.3 ± 24.7 |
| Schunk | Flip mug | 48.0 ± 4.3 | 77.3 ± 5.0 |
| Schunk | Open door | 55.0 ± 14.3 | 69.0 ± 7.7 |
| Adroit | Relocate, tomato soup can | 41.7 ± 30.3 | 95.0 ± 3.0 |
| Adroit | Relocate, potted meat can | 0 ± 0 | 53.3 ± 37.7 |
| Adroit | Relocate, mustard bottle | 0 ± 0 | 0 ± 0 |
| Adroit | Flip mug | 28.7 ± 17.7 | 54.7 ± 15.3 |
| Adroit | Open door | 58.3 ± 21.7 | 78.0 ± 19.7 |
| Allegro | Relocate, tomato soup can | 0 ± 0 | 59.7 ± 21.3 |
| Allegro | Relocate, potted meat can | 4.3 ± 3.7 | 38.3 ± 21.7 |
| Allegro | Relocate, mustard bottle | 36.3 ± 15.3 | 49.7 ± 18.3 |
| Allegro | Flip mug | 33.7 ± 15.0 | 51.3 ± 34.7 |
| Allegro | Open door | 69.3 ± 38.0 | 64.7 ± 14.7 |
The authors note that DAPG is below RL only for Open Door with the Allegro hand §VII-B fig. 6. Relocate with the mustard bottle is very hard for Adroit and Allegro in both methods §VII-B.
Real robot, Allegro hand on XArm-6, nine pairs per setting, three seeds tab. V:
| Task | RL | DAPG |
|---|---|---|
| Relocate, known objects | 22.2 ± 22.2 | 77.7 ± 11.1 |
| Relocate, novel objects | 18.5 ± 23.1 | 66.6 ± 11.1 |
| Flip mug | 3.6 ± 6.4 | 44.4 ± 19.2 |
Ablations on Relocate with the tomato soup can and the Schunk hand fig. 7: learning curves are reported as robust to friction and object density changes, stiffer PD gains reach the target sooner, and more demonstrations give better performance, with larger variance at 20 or 30 demonstrations §VII-C.
Other reported numbers: retargeting takes 76 ± 65 ms per frame on the laptop §VII-A. The customised hand avoids this online cost, so its teleoperation interface uses the operator's joint angles directly as PD targets §VII-A.
Paper vs code
Repository: https://github.com/yzqin/dex-hand-teleop, linked from the project
page; owner is Yuzhe Qin, an author. Commit 3f7b56de… (full hash in the
frontmatter). License: MIT (LICENSE file, copyright Yuzhe Qin, 2022). The
repository contains the teleoperation environments, retargeting, hand detector
and the DAPG wrapper, but no DAPG or TRPO training loop, no training
configuration, and no hyperparameter file. Each difference below is cited from
the code.
- Shape confidence. The paper defines one normalised density over the shape parameters with a diagonal covariance §IV-C. The code computes a product of 10 per-dimension Gaussian densities, each clipped to [0, 1], with the same standard deviation 0.2 in every dimension hand_monitor.py hand_monitor.py.
- Calibrated shape. The paper says the shape comes from the best view during initialisation §IV-C. The code averages all collected initialisation frames with linearly increasing weights hand_monitor.py.
- PD law. The paper's law has an explicit derivative term with gain k_d eq. 1. The code scales the error by confidence, then runs a PID with proportional gain 1, integral gain 0 and derivative gain 0, and clips the output to ±0.2 mano_robot_hand.py mano_robot_hand.py common_robot_utils.py. Drive parameters are passed to the simulator as the following triplets: root translation [1000, 40, 50], root rotation [200, 10, 10] and fingers [200, 10, 2]. Their meaning as stiffness, damping and force limit follows the simulator's call signature, which I did not verify mano_robot_hand.py.
- Low-pass filter. The paper gives no cutoff. The code uses a first-order filter with a 50 Hz control rate and a 5 Hz cutoff, both hard-coded mano_robot_hand.py common_robot_utils.py.
- Retargeting loss. The paper's objective uses squared keypoint errors eq. 2. The code uses a Huber (smooth L1) loss with beta 0.02 on 3D link positions retargeting_optimizer.py optimizer.py optimizer.py.
- Temporal penalty. The code's gradient includes the penalty on change from the previous frame, with weight 4e-3 (default), but the returned objective value does not include it optimizer.py optimizer.py.
- Keypoints. The paper says tip positions and related keypoints §V-A. The
conversion script lists 9 link names for Allegro, 11 for Adroit and 11 for
Schunk, and the bake method is
tip_middlereplay_generate_bc_data.py replay_generate_bc_data.py. - Action. The paper describes torques from inverse dynamics, or equivalently motor commands §V-B. The code's action for the BC data is different: palm velocities for the 6 base degrees of freedom, computed from pose differences and normalised to [-1, 1], plus finger position targets normalised with a gain of 2. It applies a low-pass filter to the velocities. I found no inverse dynamics for these actions player.py player.py.
- Joint format. The customised hand is driven by converting each SMPL-X
axis-angle joint to Euler angles in the
rxyzorder, after a fixed reorder of the 15 joints mano_robot_hand.py. - Episode length. The paper states no horizon. The Relocate environment
reports horizon 250, but its
is_doneonly checks whether the object falls below −0.03, so the time limit must be enforced by the caller relocate_env.py relocate_env.py. - Success and reward. The paper gives success criteria §VII-B. The RL environment files contain shaping thresholds, for example Flip with a lift-cosine above 0.9 and object distance below 0.08, and I did not find a success flag in them mug_flip_env.py. The Relocate reward is hand-designed and not described in the paper relocate_env.py.
- Frame rate. The paper states 25 fps §IV. The public README tells users to set the Record3D app to 15 fps first, then increase it if hardware allows (README.md, section "Use the teleoperation system to collect some demo").
- Demo collection. The collection script uses frame skip 5 and a control interval of frame skip times simulation timestep teleop_collect_data.py teleop_collect_data.py.
- Not in code. DAPG and TRPO training, λ₀, λ₁, network training settings,
and the real-robot sim-to-real randomisation ranges were not found in the
repository. A grep for
trpo,mjrl,dapgandlam0finds only the wrapper dapg_wrapper.py.
No discrepancies were checked for the real-robot environment; I read the code for simulation only.
Pitfalls
- Shape confidence. The paper's normalisation, covariance and scale are not given. The code uses a product of ten clipped densities with σ = 0.2. Pick one deliberately and document it.
- Calibration. The paper says "best view", but the code averages weighted frames. The two disagree on how the reference shape is chosen.
- Control gains. The paper gives no Kp, kd or filter values. The code's numbers are the only concrete ones, and they are tied to SAPIEN's drive semantics. Do not port them to another simulator without checking what the three numbers mean.
- Retargeting penalty. The code's reported objective value omits the temporal penalty while its gradient includes it. Logged losses from this code therefore do not equal the function the gradient descends.
- Actions. Do not assume the torques in the paper; the code path builds velocity and position commands. If you need torques, you must add inverse dynamics yourself, which the paper attributes to DexMV §V-B.
- DAPG coefficients. λ₀ and λ₁ are not given. The demonstration weight decays by λ₁^k, so the chosen values determine how long demonstrations matter. This is a major source of irreproducibility until the authors publish them.
- Missing training loop. The public repository ends at the environment and the DAPG wrapper. Expect to write the TRPO and DAPG training yourself.
- Sim success thresholds. Relocate's "0.1 unit length" has no stated unit. Flip's 5° and Open Door's 60° are the only numeric success rules in the paper for simulation.
- Reported variance. Several cells show very large standard deviations over only three seeds, for example Allegro Open Door with DAPG at ±14.7 and Adroit Relocate mustard bottle at 0 ± 0 for both methods. Treat small differences as noise.
- Real-world setup. Objects are placed in 20 cm and 40 cm squares, and success is judged in a 5 cm tolerance using a reference poster §VII-D. Sim-to-real transfer depends on noise and randomisation values that the paper does not state.
- Figure and table numbers. These follow source order in the LaTeX, because I could not compare them against the compiled PDF. Check fig. N and tab. N references against the PDF before trusting them.
Lineage
- Builds on: DAPG, the demonstration-augmented policy gradient of Rajeswaran et al. (cited as [Rajeswaran2018] in the paper's bibliography) §VI.
- Builds on: the action computation from DexMV, cited as [qin2021dexmv] (the paper's bibliography lists arXiv:2108.05877) §V-B.
- Compared against: vision teleoperation with online retargeting, for example DexPilot §II, and VR-based demonstration collection §I.
- Compared against: TRPO-based RL without demonstrations §VII-B.
- Concurrent work mentioned by the authors: Sivakumar et al. (vision teleoperation of an Allegro hand) and Arunachalam et al. (teleoperation with nearest-neighbour visual imitation) §II.
Source and license
Adapted from “From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation” by Yuzhe Qin, Hao Su, Xiaolong Wang (arXiv:2204.12490), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/yzqin/dex-hand-teleop at commit 3f7b56deed87 (MIT). Any quoted code is shown under that license.