AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

In plain words

Teaching a robot a skill often means showing it the skill many times. A common way to show it is to move a robot yourself while it copies you. Most camera-based versions of this only work with one robot model, in one place, with one camera setup, so each new robot means a new system.

AnyTeleop is one general system for this. A camera watches your hand. Software reads the finger and wrist positions, works out where the robot's fingers and wrist should go, and sends those commands to a robot arm and hand. The same pipeline works for different arms and hands, in a simulator or on a real robot, and with one or several cameras. A browser view lets an operator follow the scene from anywhere, and two operators can share one scene to hand an object between two robots.

The point is to make it cheap and easy to collect demonstrations across many robots and settings, and the authors report that policies trained on these demonstrations do better than ones trained on data from an earlier, specialised system in simulation.

TL;DR

• A vision-based teleoperation stack for dexterous arm-hand robots: hand pose detection, optional multi-camera fusion, optimization-based hand retargeting, and GPU motion generation for the arm §3.2 §4.
• Modular, with fixed input and output interfaces per module, so it can run in simulation or on hardware and with RGB or RGB-D cameras §3.1 §3.2.
• Author-reported: higher real-robot success than Telekinesis on 8 of 10 replicated tasks, and higher imitation-learning success than the baseline on 5 of 6 simulated settings tab. 4 tab. 3.
• The released code covers only the retargeting library. The server, fusion, motion generation and experiments have no public code in the linked repositories.

Problem

Vision-based teleoperation is a cheap way to collect demonstrations for dexterous hands §1. Earlier systems are tied to one robot model or one deployment environment. Some need hand-tracking models trained in the studio where they are deployed, or retargeting and collision models trained for one robot. Those systems scale poorly as robots and environments multiply §1.

Each earlier system also targets one "reality": the real world, or one chosen simulator. Most are built for one operator and one robot §1. The paper counts these gaps as the reasons existing systems do not scale data collection §1.

Method

Overview. AnyTeleop has four server modules with defined inputs and outputs: hand pose detection, detection fusion, hand pose retargeting and motion generation §4. The architecture also has a camera driver, the robot in simulation or on hardware, and a web visualizer §3.2 fig. 3. The authors state that the system needs no extrinsic calibration between cameras as input §3.1. A calibration step does run inside the fusion module (step 2 below) §4.2.

1. Hand pose detection §4.1.

• A lightweight RGB hand detector (MediaPipe) gives 21 hand keypoints in the wrist frame in 3D and 2D image keypoints. It runs in real time on a CPU §4.1.
• With an RGB-D camera, keypoint depths are read from the depth image and back-projected with the camera intrinsics. The wrist 6D pose then comes from the Perspective-n-Point (PnP) algorithm §4.1.
• With RGB only, hand orientation is computed from the local keypoints. The wrist position comes from an extra network that predicts the hand's weak-perspective scale, adopted from FrankMocap §4.1. The authors say this position is less accurate than depth but still usable. They report no number §4.1.
• The detector also outputs SMPL-X hand shape parameters, which the fusion step uses as a confidence signal §4.2.

2. Detection fusion §4.2.

• Each camera estimates the hand pose in its own frame. Single-camera detection is unreliable when the hand is perpendicular to the image plane (self-occlusion) §4.2.
• Calibration uses the hand as the marker. For the first N frames the operator spreads the fingers. The relative rotation between cameras is estimated in SO(3) from relative motion between consecutive frames, which the authors find more robust than absolute positions in RGB-only mode §4.2.
• Confidence per camera: take the mean of the SMPL-X shape parameters over the first N frames as a reference. The error between each frame's prediction and this reference is the confidence signal. A larger error means less confidence. The norm used for the error is not stated §4.2.
• At run time the fusion passes on the relative motion from the most confident camera §4.2.

3. Hand pose retargeting §4.3 eq. 1.

• The retargeting maps the detected human finger keypoints to the robot hand joints by an optimization at each time step t §4.3.
• Eq. 1 minimizes, over the joint vector q_t, the sum over keypoints i of the squared difference between the scaled human keypoint vector and the robot's keypoint vector, plus a penalty on the change from the previous step. It is subject to joint bounds q_l ≤ q_t ≤ q_u eq. 1.
• Symbols: v_t^i is the i-th human keypoint vector, f_i the i-th forward kinematics of the robot hand, α a hand-size scaling factor, and β the smoothing weight eq. 1. The paper gives no values for α or β §4.3.
• For a different morphology the keypoint correspondence between human and robot fingers is set by hand §4.3.
• This module handles the hand only, not the arm §4.3.

# Hand pose retargeting at time step t  [§4.3] [eq. 1]
# Input:  human keypoint vectors (v), previous joints (prev), scale alpha,
#         smoothing weight beta
# Output: robot hand joint positions

def retarget_step(v, prev, alpha, beta):
    # Minimize over q the sum, over keypoints i, of
    #   || alpha * v[i] - f_i(q) ||^2 + beta * || q - prev ||^2  [eq. 1]
    # subject to q_lower <= q <= q_upper  [eq. 1]
    q = minimize_bounded_least_squares(
        residuals(v, alpha, beta, prev),
        lower=q_lower,
        upper=q_upper,
        start=prev,
    )
    return q

4. Motion generation §4.4.

• The arm moves with CuRobo, a GPU-parallelized collision-free motion generator. The authors chose it over the acceleration-based RMP approach of DexPilot, which does not guarantee natural trajectories §4.4.
• Input: the end-effector Cartesian pose at 25 Hz from detection and retargeting. Output: collision-free joint trajectories within joint limits at 120 Hz §4.4.
• These trajectories go to impedance controllers on the simulated or real robot §4.4. CuRobo's internal settings are not stated in the paper.

5. Software interface and deployment §3.1 §3.2. Each module has defined input and output interfaces, so it can be swapped. The control modules can run on a separate machine from detection §6.1. The whole stack is packaged as a Docker image §3.1.

6. Web-based viewer §5 app., Web-based Teleoperation Viewer.

• Built on meshcat, with Three.js for rendering. After each simulation iteration the server sends the state to browsers. Operators see the scene and move their hand to control the robot §5.
• Multi-view: open more browser windows app., Web-based Teleoperation Viewer fig. 9.
• Rendering is lightweight. Simulation states can be saved for later rendering with a stronger engine app., Web-based Teleoperation Viewer fig. 8.

7. Collaborative teleoperation §7.2 fig. 6.

• Each teleoperation unit is a computer with at least one camera and one operator. The operator watches a browser view and moves the hand §7.2.
• A central server runs the physics simulation and the web visualizer. It takes detections from all units, converts them to robot commands, and keeps the visualization synchronized §7.2.
• The paper's example is a human-to-robot handover fig. 5 §7.2.

Setups used in the experiments.

• Real robot: XArm6 arm with an Allegro hand, one Intel RealSense camera, one trained operator §6.2.
• Simulation imitation learning: SAPIEN environments with floating-hand and arm-hand variants of three tasks §7.1 tab. 3.
• Camera configuration study: Play Piano task in IsaacGym [app., System Evaluation on Camera Configurations].

Hyperparameters

Results

Author-reported. The setups are listed under Method.

Real robot vs Telekinesis tab. 4. XArm6 with Allegro, one RealSense camera, ten runs per task. The Telekinesis numbers are copied from its paper, not re-run tab. 4 caption.

Higher on 8 tasks, equal on 2 §6.2 tab. 4.

Imitation learning in SAPIEN tab. 3. Success rate over 100 trials, mean ± standard deviation over three seeds. The table gives no unit, but the rate is per 100 trials. Baseline is the demonstrations of the prior system [qin:ral2022] and RL is pure reinforcement learning without demonstrations §7.1.

AnyTeleop is higher than the baseline on 5 of 6 rows, the exception being floating-hand Flip Mug tab. 3 §7.1. It is higher than RL on all 6 tab. 3.

Profiling tab. 2. Time per module in milliseconds, all modules running on one computer. The table does not say whether ± is a standard deviation.

The authors report that 120 Hz motion generation was hard to reach with all modules on one computer, so they move the control modules to a separate machine §6.1.

Comparison with related systems tab. 1. Qualitative capabilities, as stated by the authors. "Free" columns mean the system does not need the listed input or requirement.

Camera configurations tab. 5 (Play Piano in IsaacGym, ten runs each, one trained operator). Completion time and error rate on incorrect key presses:

The paper reads these as a trade-off: more information (depth) and more cameras mean faster and more accurate teleoperation [app., System Evaluation on Camera Configurations].

Paper vs code

Repository used. The project page linked in the abstract points to three repositories. Only one is cited here: https://github.com/dexsuite/dex-retargeting (MIT, cloned at commit 3f56141bc8bd2760d5e452e382937269554ebb21). Its CITATION.cff lists the eight AnyTeleop authors. It holds the retargeting library used for §4.3: the vector optimizer, a DexPilot optimizer, a low-pass filter and robot configs. It has no hand detector, no fusion, no CuRobo motion module, no teleoperation server and no experiment code. The project page also links https://github.com/NVlabs/sim-web-visualizer for the web viewer §5. It was not cloned or license-checked, so nothing from it is used. I searched GitHub for an official repository of the full system (AnyTeleop name search and the authors' organisations) and found none. The dexsuite/dex-retargeting repo is the only one linked from the project page.

Objective differs from eq. 1.

• Eq. 1 uses squared differences per keypoint eq. 1. The vector optimizer takes the Euclidean norm of each scaled vector error, then applies a Huber (smooth L1) loss with transition δ = 0.02 to the norms [code: src/dex_retargeting/optimizer.py#L272-L273]. Huber is not in eq. 1.
• The smoothing term is normal_delta times the squared step size, with normal_delta = 4e-3 by default optimizer.py retargeting_config.py. The paper's β is unstated.
• The solver is NLopt SLSQP with joint bounds widened by 1e-3 [code: src/dex_retargeting/optimizer.py#L41] [code: src/dex_retargeting/optimizer.py#L54-L60].

Low-pass filtering is not in the paper. SeqRetargeting applies an exponential filter to the output joints after each optimization [code: src/dex_retargeting/seq_retarget.py#L132-L133]. The update is y ← y + α(x − y) optimizer_utils.py. The teleop configs set α = 0.2 allegro_hand_right.yml. The paper describes the hand commands as smooth §4 but never mentions a filter.

Retargeting type. The paper describes a vector-style optimization eq. 1. The code also ships a DexPilot optimizer optimizer.py. The paper does not say which optimizer the evaluated system used. The configs exist for both.

Failure fallback not in the paper. If the optimizer raises a runtime error, the code prints it and returns the previous joints [code: src/dex_retargeting/optimizer.py#L100-L102].

Not checked against code. Detection, fusion, CuRobo settings, the web viewer, the imitation-learning pipeline and the real-robot results have no code in the linked repositories. Those claims rest on the paper alone.

Pitfalls

• Eq. 1 is not the implemented loss. Implement the released form (Huber on vector norms plus a squared step penalty) only if you want to match the code. Use eq. 1 only if you want to match the text. The paper's α and β are both unstated.
• The low-pass filter changes latency. The code's teleop configs filter every output with α = 0.2. A smaller α is smoother but adds lag [code: src/dex_retargeting/configs/teleop/allegro_hand_right.yml#L14-L15]. The paper does not mention this cost.
• Configuration key names. The YAML key normal_delta sets the code's norm_delta retargeting_config.py. The name normal_delta appears in the config, and norm_delta in the optimizer.
• Keypoint correspondence is manual. Each robot needs its own mapping from human keypoint indices to robot links §4.3. The Allegro mapping is in the config allegro_hand_right.yml. The paper gives no default for other hands.
• Fusion confidence norm is unstated. Only the reference (mean over the first N frames) and the direction (larger error means less confidence) are given §4.2.
• N = 50 requires a calibration gesture. The operator must spread the fingers during the first N frames §4.2. The authors ask for it so that the reference shape is accurate §4.2.
• RGB-only wrist position error is not quantified. The paper says it is larger than with depth but "sufficient" §4.1. Budget for this if you run without depth.
• Collision freedom is claimed, not measured. The paper says the arm-hand setting "guarantees no self-collision" §7.1 and CuRobo is collision-free §4.4. It gives no collision-check numbers.
• Demonstration conversion is not described. For the arm-hand setting, the AnyTeleop demonstrations are converted to floating-hand format so both variants can use them §7.1. The conversion method is not stated. The baseline's own floating-to-arm translation pipeline is only referenced §7.1.
• Roles in the collaborative example are inconsistent. The text says operator #2 controls a human hand §7.2. The caption of fig. 5 says operator #2 acts as the Kuka-Shadow robot.
• Small trial counts. Real-robot results use ten runs per task §6.2. Differences such as 0.6 vs 0.7 are one or two runs. The baseline numbers come from another paper, not a re-run tab. 4 caption.
• Imitation-learning variance is large. Several gaps in tab. 3 are smaller than the standard deviations, for example floating Relocate 53.7 ± 12.2 vs the baseline's 49.7 ± 18.3. Three seeds give little statistical power.
• Motion generation needs its own machine. Running all modules on one computer missed 120 Hz §6.1. Plan for a separate control host.
• Appendix numbering. The appendix subsections have no letter in the source, so they are cited by title. The camera-configuration section is the first section after \appendix in the source, and it is also cited by title.
• Table and figure numbers are counted from the source. They follow the order of \input and float environments in the LaTeX. The paper's compiled numbering, including IEEE Roman numerals for tables, may print differently.

Lineage

• Builds on: DexPilot (Handa et al., ICRA 2020) for the retargeting formulation and the morphology-specific mapping §4.3. The code's docstring links arXiv:1910.03135 optimizer.py. No card exists yet.
• Builds on: CuRobo (Sundaralingam et al., ICRA 2023) for motion generation §4.4. No arXiv id is given in the paper. No card exists yet.
• Builds on: the vision-based hand pose detector MediaPipe §4.1 and the weak perspective approach of FrankMocap §4.1. Neither has a card.
• Compared against: Telekinesis (Sivakumar et al.), arXiv:2202.10448 per the paper's bibliography tab. 4 tab. 1. No card exists yet.
• Compared against: Qin et al., RA-L 2022, "From One Hand to Multiple Hands", for simulated teleoperation and imitation learning §7.1 tab. 3. No card exists yet.
• Imitation learner: DAPG, Rajeswaran et al., RSS 2018 §7.1. No card exists yet.
• Superseded by: not known from the content of this wiki.