--- kind: concept slug: teleoperation title: Teleoperation summary: >- A human drives a robot in real time, most often to record demonstrations or to take over a running policy. Covers how operator motion is sensed (cameras, gloves, leader arms, VR trackers), mapped to robot joints, blended with autonomy and fed back as force, with the cost and evidence of each. All evidence is author-reported on different systems, so the numbers do not rank against each other. tags: [teleoperation, dexterous-manipulation, demonstrations, robot-learning, shared-autonomy, haptics] papers: ["1910.03135", "2307.04577", "2506.04982", "2503.24070", "2402.02758", "2402.10220", "2410.18727", "2411.01014", "2411.07534", "2412.00268", "2412.02613", "2412.03508", "2412.03619", "2412.06917"] related: [motion-retargeting, imitation-learning, learning-from-demonstration, human-in-the-loop, action-chunking, robot-simulation, shared-autonomy, haptic-feedback-rendering, impedance-control, disturbance-observer, inverse-kinematics, motion-primitives, magnetic-actuation, human-evaluation] status: draft updated_at: 2026-11-01 --- ## Overview Teleoperation means a person controls a robot as it moves, and the robot follows in real time. Its main use in this wiki is to collect demonstrations: the operator performs a task, the robot records the motion, and a learner is trained on the recordings [arxiv:2307.04577 §1]. A second use is correction. A human takes over a policy that is already running, and the takeover is recorded as training data [arxiv:2503.24070 §3.3] [arxiv:2503.24070 §4.2]. A third use is assistance. The robot blends its own commands into the operator's, or infers the task so that the operator has to give less input [arxiv:2402.02758 §3.2] [arxiv:2411.01014 §2]. The invariant all variants share is a loop run on every control step: sense the operator, turn the sensed motion into robot targets, and send those targets to the robot. Variants differ in what does the sensing (cameras, a glove with joint sensors, a leader arm, or VR trackers), in how the sensed motion is mapped to joints, in whether the robot adds its own commands, in whether force is sent back to the operator, and in where the compute runs [arxiv:2307.04577 §4] [arxiv:2307.04577 §7.2] [arxiv:2506.04982 §2] [arxiv:2503.24070 §3.2] [arxiv:2411.07534 §II] [arxiv:2412.02613 §3.1]. Fourteen papers with cards cover the topic. Most report on one system with their own tasks, operators and metrics, so their numbers cannot be ranked against each other. Several are small: one to ten people, one or a few trials, or a single robot. - [DexPilot](/papers/1910.03135): bare-hand teleoperation of a robot arm and dexterous hand, seen by four depth cameras. Two pilots on 15 tasks. - [AnyTeleop](/papers/2307.04577): camera-based teleoperation of dexterous arm-hand robots. Timed camera study and real-robot comparison. - [GEX](/papers/2506.04982): a low-cost glove-and-hand kit. Qualitative results only. - [HACTS](/papers/2503.24070): a leader arm with two-way joint sync, used to correct policies on real UR5 tasks. Success rates over 10 rollouts per cell. - [Effects of Shared Control on Cognitive Load and Trust](/papers/2402.02758): a 24-person study of blended robot and human commands on an arm. - [Operator intention detection from force data](/papers/2402.10220): infers a glovebox task from robot joint and force signals, with no extra sensors. - [Mixed Reality Teleoperation Assistance for Humanoids](/papers/2411.01014): learned movement primitives and affordance templates assist a humanoid's grasp. An eight-person door-opening study. - [VR teleoperation of an upper-body humanoid](/papers/2411.07534): four VR trackers mapped to joint groups, with soft self-collision avoidance. Qualitative. - [Haptic stiffness perception through a hand exoskeleton](/papers/2412.02613): ten participants discriminate soft samples through a glove that renders force. - [Bilateral teleoperation for rehabilitation](/papers/2412.03619): two impedance-controlled planar robots with a disturbance observer. Qualitative. - [Haptics in micro- and nano-manipulation](/papers/2412.06917): bilateral magnetic micro-robot tele-manipulation. Single-trial experiments. - [Non-expert teleoperation in the UK and Japan](/papers/2410.18727): 74 people stack cubes with three arms through a digital twin. Confounded by design. - [GRIP-tape gripper](/papers/2412.00268) and [a tendon-driven continuum manipulator](/papers/2412.03508): both are teleoperated by joystick, and their papers focus on the mechanism. AnyTeleop frames vision-based teleoperation as a cheap way to gather demonstrations for dexterous hands. Its problem is that earlier systems each fit one robot model or one environment [arxiv:2307.04577 §1]. ## Variants ### Sensing the operator #### Single RGB camera The hand is detected from a plain RGB image. Hand orientation comes from the keypoints, and the wrist position comes from a network that predicts a weak-perspective scale. The authors say this position is less accurate than with depth but still usable, and they give no number [arxiv:2307.04577 §4.1]. This is the cheapest sensing option in the paper. It is also the slowest and least accurate of the three camera setups in the one timed comparison [arxiv:2307.04577 tab. 5]. #### Single RGB-D camera Keypoint depths are read from the depth image and back-projected with the camera intrinsics. The wrist 6-D pose then comes from Perspective-n-Point [arxiv:2307.04577 §4.1]. In the camera study, adding depth cut completion time from 109 s to 87 s and the error rate on wrong key presses from 28.1% to 21.8% [arxiv:2307.04577 tab. 5]. #### Two RGB-D cameras (fused) Each camera estimates the hand pose in its own frame. A single camera fails when the hand is edge-on to the image plane, so the fusion module combines the views [arxiv:2307.04577 §4.2]. Fusion needs the cameras' relative rotation. It is estimated from the hand as a marker: the operator spreads the fingers for the first N = 50 frames [arxiv:2307.04577 §4.2]. In the camera study, two RGB-D cameras gave 74 s and 12.5% error [arxiv:2307.04577 tab. 5]. That study used one task, one operator and ten runs per configuration. #### Bare hand, four depth cameras, model-based tracking Four calibrated, time-synchronised RGB-D cameras look down at a table. The authors say depth quality degrades beyond about 1 m, and the work volume is 80 × 55 × 38 cm [arxiv:1910.03135 §3]. The operator's bare hand is tracked with DART, a model-based tracker that fits an articulated hand model to segmented points. DART needs a starting pose, so two neural networks supply it. The first takes 8192 uniformly sampled points from the fused cloud and outputs 23 keypoints and a hand segmentation. The second resamples points around the hand, adds random perturbation to the pose and predicts the keypoints again. A small MLP then maps the keypoints to 20 finger joint angles [arxiv:1910.03135 §6] [arxiv:1910.03135 tab. VI]. The networks need training labels, and the paper makes them in a separate phase. The operator wears a glove with coloured blobs, and only the palm blobs are used for the pose. DART then fits the model to points in a bounding volume, and the networks train on 100K point clouds labelled this way. At run time the operator uses the bare hand, and the glove is not worn [arxiv:1910.03135 §1] [arxiv:1910.03135 §6]. Without the network priors, the authors say DART lost the hand every few minutes, in spurious local minima [arxiv:1910.03135 §6]. The 9.7 mm keypoint error is measured on the authors' own 7000-image set, so it is not a benchmark result [arxiv:1910.03135 §6]. The robot has tactile sensors, but the teleoperation loop uses no tactile feedback [arxiv:1910.03135 §3] [arxiv:1910.03135 §10]. The system's end-to-end latency is about one second, with no breakdown [arxiv:1910.03135 §4]. #### Glove with joint sensors The operator wears an exoskeleton glove, and the robot hand is driven by the glove's joint readings instead of by a camera. Every joint of the hand and of the glove has its own motor, so joint positions are measured rather than estimated from images [arxiv:2506.04982 §2.1] [arxiv:2506.04982 §2.2]. The glove has 12 DoF and the hand 11 [arxiv:2506.04982 §2.1] [arxiv:2506.04982 §2.2]. The glove's fingertip and palm keypoints go into an optimization that sets the hand joints, with joint limits as hard bounds [arxiv:2506.04982 §3.2] [arxiv:2506.04982 eq. 1]. The paper also describes force feedback, where the glove switches to impedance control on contact. The released repository does not contain that logic [arxiv:2506.04982 §3.3]. The authors state a cost of USD 600 per device and under four hours of assembly [arxiv:2506.04982 §1]. Their results are qualitative: one real cup grasp and three simulated tasks, with no success rates and no baseline [arxiv:2506.04982 §4] [arxiv:2506.04982 §4.1]. #### Leader arm with two-way sync The operator moves a low-cost leader arm, built from 3D-printed parts and servo motors. The leader and the follower robot are synced in both directions. When a person teleoperates, the leader's calibrated readings become robot joint targets. When a policy drives the robot, the robot's joint readings are sent back to the leader's motors, so the leader moves with the robot [arxiv:2503.24070 §3.3]. A foot pedal switches between the two modes [arxiv:2503.24070 §3.3]. The authors say only position information needs to be synchronised, so the follower does not need force or tactile sensing [arxiv:2503.24070 §3.2] [arxiv:2503.24070 §1]. The stated cost is below USD 300, and the bill of materials for the UR5 version totals USD 247 [arxiv:2503.24070 §4.1] [arxiv:2503.24070 tab. 1]. The paper reports three uses. In each, human corrections are recorded and mixed with the original demonstrations for retraining: - Imitation learning with ACT on three real UR5 tasks. The HACTS-trained policy was higher than the 100-demonstration baseline in all nine task-and-setting cells [arxiv:2503.24070 tab. 2]. Its base data was 50 demonstrations plus 50 correction trajectories, so the data sources are not equal [arxiv:2503.24070 §4.2]. - Diffusion Policy on one out-of-distribution task. The HACTS version reached 40%, and both baselines reached 0% [arxiv:2503.24070 tab. 3]. - Online reinforcement learning (RLPD) on the CloseBin task. Success rose from 50% to 80% after 45 minutes of online training, with short human corrections [arxiv:2503.24070 tab. 4]. Each cell is 10 rollouts, so every success rate moves in steps of 10% and no variance is reported [arxiv:2503.24070 §4.2]. #### Operator intent from robot signals (no extra sensor) Here the robot does not need the operator's hand at all. A convolutional network reads a window of joint positions, velocities, efforts and wrist force-torque at 100 Hz, and classifies which of six glovebox tasks is under way [arxiv:2402.10220 §III-D] [arxiv:2402.10220 §IV-A]. A second model, a convolutional encoder followed by an LSTM, forecasts the next 500 frames, and an MLP classifies that forecast so that the robot can anticipate the next step [arxiv:2402.10220 §IV-E] [arxiv:2402.10220 §IV-F]. The authors argue that IMU, motion-capture and haptic-device data would not help here, so they use only the robot's own signals [arxiv:2402.10220 §V-B]. On the six-task glovebox set, the paper reports a best classification of 89% for the four-stream model [arxiv:2402.10220 §IV-C]. For forecasting, the best model reaches 1.91e-05 in the table, against 0.065 for a spatial-mean LSTM baseline [arxiv:2402.10220 tab. III]. The datasets are small, with 20 samples per task in the glovebox set [arxiv:2402.10220 §III-D]. Test splits are random, and the paper does not say whether one operator's samples appear in both train and test [arxiv:2402.10220 §IV-B]. #### VR trackers and joint groups Four 6-DoF trackers (headset, two hands, waist) drive a 17-DoF humanoid. Each tracker is assigned to one joint group, so the robot's joints answer to a clear input [arxiv:2411.07534 §II] [arxiv:2411.07534 fig. 1]. The paper gives no success rates, timings or tracker hardware [arxiv:2411.07534 §IV]. Its evidence is informal demonstrations, which is covered under Assistance and retargeting. ### Mapping to the robot #### Keypoint retargeting (camera and glove) For the camera and glove variants, human finger keypoints are optimised into robot joint targets, with joint limits as bounds [arxiv:2307.04577 §4.3] [arxiv:2506.04982 §3.2]. The camera paper's objective is a squared-error form in the text, and its released code uses a different loss (see Contradictions). The glove paper uses the DexPilot form from an external library in its released code, not the equation printed in the paper [arxiv:2506.04982 §3.2]. #### Task-Jacobian masking for joint groups For a humanoid with more joints than tracker degrees of freedom, the paper removes the columns of unassigned joints from each tracker's task Jacobian, so each tracker moves only its own joints [arxiv:2411.07534 §II]. This avoids hand-tuned task weights for the redundancy, which the authors say can cause oscillations [arxiv:2411.07534 §I]. The paper gives no weights, gains or control rate, so the method cannot be reproduced from the text alone [arxiv:2411.07534 §II] [arxiv:2411.07534 §I]. #### Fingertip-vector retargeting (bare hand) DexPilot does not match keypoints directly. Each retargeting vector runs between two points of the human hand, such as a fingertip, the palm or the thumb, and the robot's vector between the same points is pulled toward the human vector, scaled by 1.6 for distant pairs [arxiv:1910.03135 §7.1]. Pairs whose human distance falls below a threshold get a much larger weight, and the threshold itself is not stated [arxiv:1910.03135 §7.1] [arxiv:1910.03135 tab. I]. A small regulariser keeps the robot joints near open. The problem is solved once per frame with SLSQP, starting from zero on the first frame and from the previous solution afterwards [arxiv:1910.03135 §7.1]. The authors do not state a tolerance or an iteration cap, and they report no per-frame solve time [arxiv:1910.03135 §7.1]. The retargeted fingers go to a joint controller at 30 Hz, and the arm to an impedance controller at 200 Hz [arxiv:1910.03135 §7.2]. #### Joint-space copy (leader arm) With a leader arm, the mapping is a calibrated joint-to-joint copy, not a keypoint optimization. A mode switch picks the direction of the copy [arxiv:2503.24070 §3.3]. ### Shared control and assistance #### Blended commands (shared autonomy) The commanded position is a convex blend of a robot reference and the human position. A weight on the robot sets the level of autonomy: one is full autonomy and zero is none [arxiv:2402.02758 §3.2] [arxiv:2402.02758 eq. 3]. In a 24-person study on a robot arm with a haptic handle, the paper reports lower self-rated workload and higher trust when the robot did more of the work. The NASA-TLX effect was F(1, 22) = 46.6, p < .001, and the MDMT trust effect was F(1, 22) = 41.4, p < .001 [arxiv:2402.02758 tab. 1]. Objective tracking error fell from 2.16 cm to 0.83 cm between the low and high autonomy levels [arxiv:2402.02758 tab. 2]. Tapping error, used as a secondary-task measure, showed no clear autonomy effect [arxiv:2402.02758 tab. 1]. The analysis uses two of the four autonomy levels in the main phase, 0.2 and 0.8 [arxiv:2402.02758 §4]. The authors list the effect of this binary split as a limitation [arxiv:2402.02758 §6]. #### Learned primitives plus affordance templates For a humanoid grasping task, the operator's first motion and the object pose select and condition a learned motion primitive. Near the grasp, the trajectory switches to a stored, object-specific action sequence through a logistic blend [arxiv:2411.01014 §2.3] [arxiv:2411.01014 §2.4] [arxiv:2411.01014 §2.7]. The operator previews the plan as a ghost, accepts or rejects it, and then moves it with a joystick [arxiv:2411.01014 §3]. Primitives come from fewer than 30 demonstrations in simulation [arxiv:2411.01014 §2]. In an eight-person door-opening study with IHMC members, direct control completed 75.0% of trials for experts and 58.3% for novices. With assistance, all trials completed, and time on task fell from 81.1 s to 40.2 s for experts [arxiv:2411.01014 tab. 2]. No significance tests are reported, and the study has no per-trial data [arxiv:2411.01014 §4.2]. A punching task has no affordance template, so the paper uses the primitive alone there and reports an RMS agreement with the user, not a success rate [arxiv:2411.01014 §4.1]. #### Virtual barriers for self-collision Self-collision is a soft term in a QP-based inverse-kinematics solve. A relaxed barrier returns a best-effort command instead of rejecting the command, which the authors say avoids abrupt pauses during tasks [arxiv:2411.07534 §I] [arxiv:2411.07534 §II] [arxiv:2411.07534 eq. 4]. The paper reports no minimum clearance or violation rate, so the safety benefit is not measured [arxiv:2411.07534 §IV]. ### Haptic feedback to the operator #### Glove force rendering from contact The robot fingertip sensors give a contact force, which is scaled into the glove's force limit. Method I scales the contact force alone. The fingertip force is the largest of four sensor z-forces, and readings below a minimum give zero feedback [arxiv:2412.02613 §3.1] [arxiv:2412.02613 §3.2.1]. Method II multiplies that force by a ratio of leader and follower finger displacement and by a range factor [arxiv:2412.02613 §3.2.2] [arxiv:2412.02613 eq. 9]. Ten naive participants discriminated five soft silicone samples through the glove, with no visual feedback. The ABX success was 74.16% for Method I and 74.58% for Method II. The shape-only S task gave 67.91% and 64.58% [arxiv:2412.02613 tab. 2]. The authors report no consistent overall gain from Method II, and they say force is the more important component [arxiv:2412.02613 §5.2] [arxiv:2412.02613 §6]. The displacement factor appears to help for pairs of similar stiffness, but the pair-level tests are mostly not significant [arxiv:2412.02613 §5.5]. #### Bilateral position with virtual spring force Two planar robots run the same impedance controller, and each has a disturbance observer. The master renders a virtual spring from the position difference between the two end-effectors, with no force sensor [arxiv:2412.03619 §II-A] [arxiv:2412.03619 §II-B] [arxiv:2412.03619 §II-C]. The master's observer is switched off in free human interaction, so the operator can move it freely [arxiv:2412.03619 §V]. The paper shows trajectory plots and force ranges, and gives no tracking-error number or baseline [arxiv:2412.03619 §IV]. The experiments are passive movement only [arxiv:2412.03619 §V]. #### Magnetic micro-robots A haptic device drives a magnetically guided micro-robot. The operator's motion is scaled down and the interaction force is scaled up before it reaches the hand [arxiv:2412.06917 p. 2]. A PI loop on the desired force, with a damping term, keeps the contact force below a safe limit [arxiv:2412.06917 p. 9] [arxiv:2412.06917 eq. 3]. The force scale and the gains are not given [arxiv:2412.06917 p. 9]. The three experiments are single trials. They report an 8 µm steady-state error for a bead [arxiv:2412.06917 p. 11] and a 0.4 µN force for a cell-penetration case [arxiv:2412.06917 p. 12]. The abstract promises a stability analysis, but the chapter does not contain one [arxiv:2412.06917 p. 1] [arxiv:2412.06917 p. 13]. ### Teleoperation as a tool, and field studies #### Non-expert operators across robots A digital twin of each arm is driven by the operator, and the real arm follows through ROS 2 [arxiv:2410.18727 §III] [arxiv:2410.18727 §III-B]. In a 10-minute cube-stacking task, the UR5e in Japan built a mean of 10.50 towers per user, the Baxter in the UK 3.25 and the UR3 in the UK 1.03 [arxiv:2410.18727 §V-A] [arxiv:2410.18727 fig. 6]. The UR5e also had the lowest mental NASA-TLX score [arxiv:2410.18727 §V-B]. The UR5e was tested only in Japan, while the other two robots were tested only in the UK, and recruitment differed between the countries [arxiv:2410.18727 §IV]. The paper therefore describes setups, not arms alone. #### Joystick-driven arms with a fixed mechanism Two papers describe a mechanism and treat joystick teleoperation as the default control. The GRIP-tape gripper is driven by a joystick app, and its gamepad moves the target by at most 10 mm per loop cycle [arxiv:2412.00268 §Demonstrations] [arxiv:2412.00268 p. S6]. It reports a mean tip error of 3.72 ± 0.35 mm over 16 targets [arxiv:2412.00268 §Kinematics]. The tendon-driven continuum manipulator uses a joystick for the tip, and its decoupling is shown on steady postures [arxiv:2412.03508 §III-A] [arxiv:2412.03508 §IV]. Neither paper reports a tracking error for teleoperation under load. ## When to use which | Situation | Use | Why | |---|---|---| | A learned policy runs on the robot, and a person must step in mid-task | Leader arm with two-way sync | Takeover and sync are part of the design, and corrections are recorded as training data. The follower must accept joint position commands [arxiv:2503.24070 §3.2] [arxiv:2503.24070 §3.3] | | No camera rig, joint-level hand measurement wanted, budget matters | Glove with joint sensors | The authors put the kit at USD 600 and measure joints directly. No success rates exist, so test the hardware on your own task [arxiv:2506.04982 §1] [arxiv:2506.04982 §4] | | One camera, no depth sensor, and wrist accuracy is not critical | Single RGB | Works without depth, but the authors give no wrist error figure and the timed comparison is slower and less accurate [arxiv:2307.04577 §4.1] [arxiv:2307.04577 tab. 5] | | One depth camera available | Single RGB-D | Depth gives the wrist pose directly from PnP and lowered completion time and error in the one timed study [arxiv:2307.04577 §4.1] [arxiv:2307.04577 tab. 5] | | Fingers often hidden from one view, or two cameras available | Two RGB-D, fused | Fusion covers single-view failures. The fastest and most accurate setup in the one study, though it needs a calibration gesture [arxiv:2307.04577 §4.2] [arxiv:2307.04577 tab. 5] | | Bare-hand control of a many-joint hand, four depth cameras affordable, and a labelling phase with a glove is acceptable | Four-camera bare-hand tracking with a model-based tracker | Keeps the glove out of the run-time loop. Seeding is essential, the labelling needs a glove phase, and latency is about one second [arxiv:1910.03135 §1] [arxiv:1910.03135 §4] [arxiv:1910.03135 §6] | | Operator should keep control, and a robot assist must lower workload | Shared control with a blend weight | Lower self-rated workload and higher trust in one 24-person study, with a tracking gain. Test the weight on your own task [arxiv:2402.02758 tab. 1] [arxiv:2402.02758 tab. 2] | | Operator input should be reduced, and the task is a fixed set | Intent detection from robot signals | No extra sensor, and the robot can anticipate the next step. Small data and random splits, so expect optimistic numbers [arxiv:2402.10220 §III-D] [arxiv:2402.10220 §IV-B] | | Humanoid grasp, demonstrations can be made in simulation, object poses known | Learned primitives plus affordance templates | Lower time on task and higher completion in an eight-person study. Needs an affordance per object and a pose source [arxiv:2411.01014 tab. 2] [arxiv:2411.01014 §2.6] | | Upper-body humanoid, many joints, few trackers | Task-Jacobian masking for joint groups | Each tracker moves a fixed joint group, so the mapping is predictable. No performance numbers exist [arxiv:2411.07534 §II] [arxiv:2411.07534 §IV] | | Force cues wanted, and the operator must tell stiffness apart | Glove force rendering, Method I | The force-only baseline matched the displacement-scaled variant on ABX. Start here and add the extra factor only with data [arxiv:2412.02613 tab. 2] [arxiv:2412.02613 §6] | | Two robots, passive rehabilitation movement, no force sensor | Bilateral impedance with a disturbance observer | Avoids force sensing and keeps the robots soft. No tracking-error number is reported, so measure your own [arxiv:2412.03619 §IV] [arxiv:2412.03619 §V] | | Micro-scale tool, force cannot be felt directly | Bilateral magnetic micro-manipulation | Scales the force up to the hand. Single trials only, and no stability proof [arxiv:2412.06917 p. 11] [arxiv:2412.06917 p. 13] | | Need 120 Hz arm commands | Control modules on a second computer | One machine missed 120 Hz in the authors' setup [arxiv:2307.04577 §6.1] [arxiv:2307.04577 tab. 2] | | New robot arm or hand | Keep the module interfaces, redo the keypoint mapping | Modules have fixed inputs and outputs [arxiv:2307.04577 §3.1]. The keypoint correspondence is set by hand for each hand [arxiv:2307.04577 §4.3] | | Two operators, shared task | Collaborative setup with a central server | The paper demonstrates a handover, with no reported measure [arxiv:2307.04577 §7.2] | | Real robot, need a higher success rate than a prior system | Choose by per-task evidence | Higher on 8 of 10 tasks with 10 runs each. Some gaps are one run, so treat them as noisy [arxiv:2307.04577 tab. 4] | | Demonstrations for imitation learning, simulator | Vision-based demos of this kind | Higher than the baseline's demos on 5 of 6 settings, but several gaps are smaller than the spread across three seeds [arxiv:2307.04577 tab. 3] | | Choosing an arm for non-expert operators | Do not rank arms from one study | The UR5e result is confounded with country, gripper and recruitment [arxiv:2410.18727 §IV] [arxiv:2410.18727 §VI] | ## Implementation sketch A single control step of the camera-based loop, with the paper's module order. The rates are the paper's: keypoints per hand each frame, arm commands at 120 Hz [arxiv:2307.04577 §4.4] [arxiv:2307.04577 §6.1]. ```pseudocode # One control step of a camera-based teleoperation loop [§4] [§6.1] # Keypoints: 21 per hand [§4.1]. Arm output: 120 Hz [§4.4]. def teleop_step(cameras, state): # Detect the hand in each camera's frame [§4.1] detections = [detect_hand(cam.frame()) for cam in cameras] # Take relative motion from the most confident camera [§4.2] wrist_pose, finger_kp = fuse(detections, state.calibration) # Map human finger keypoints to robot hand joints [§4.3] [eq. 1] hand_q = retarget(finger_kp, state.prev_hand_q, state.scale) # Plan a collision-free arm path to the wrist pose [§4.4] arm_traj = plan_arm(wrist_pose, state.arm_limits) # Send joint targets to the robot's impedance controllers [§4.4] robot.send(arm_traj, hand_q) state.prev_hand_q = hand_q return hand_q ``` Shared control adds one step before the send. The blend is taken between the robot's reference and the human's target [arxiv:2402.02758 eq. 3]: ```pseudocode # Shared-control blend [arxiv:2402.02758 eq. 3] # gamma: autonomy weight in [0, 1]; robot_target, human_target: (3,) def blend(gamma, robot_target, human_target): # Weight gamma on the robot, 1 - gamma on the human [§3.2] return gamma * robot_target + (1 - gamma) * human_target ``` Notes on where variants differ: - The calibration step runs once, from the first N frames, and is needed only with more than one camera [arxiv:2307.04577 §4.2]. - Swapping in a different arm or hand changes the keypoint mapping and the scale factor [arxiv:2307.04577 §4.3]. The module interfaces stay the same [arxiv:2307.04577 §3.1]. - The retargeting code in the released repository is not the form written in the paper's equation. The card's Paper vs code section lists the differences: [AnyTeleop](/papers/2307.04577). - With a glove, the sensing step is replaced by the glove's joint readings, and the same retargeting step maps them to hand joints. The released code uses DexPilot from the `dex-retargeting` library rather than the paper's equation [arxiv:2506.04982 §3.2] [code: retargeting/gex_retarget.py#L19-L26]. - With a leader arm, the mapping is a calibrated joint-to-joint copy, not a keypoint optimization. A mode switch picks the direction of the copy [arxiv:2503.24070 §3.3]. - Where the robot can refuse or soften a command, such as the self-collision term, the solver returns a best effort instead of an error [arxiv:2411.07534 §II]. - Where force is returned to the operator, gate the feedback term on the mode. The bilateral master should add the virtual spring only in free interaction [arxiv:2412.03619 §II-B] [arxiv:2412.03619 §V]. ## Contradictions No paper with a card reports a conflicting number for the same setup, because each paper measures its own system on its own tasks. Within each source, the paper's text, tables or figures disagree on a few points. Each side is cited below, and none of these is resolved here. - DexPilot: the appendix says the pose classifier head has two outputs [arxiv:1910.03135 tab. V], while the keypoint read-out is described with a 23-channel mask [arxiv:1910.03135 app. PointNet++ keypoint head]. The paper does not say how the two combine. - AnyTeleop: the paper's retargeting objective is a squared-error form, and the released code uses a Huber loss on vector norms [arxiv:2307.04577 §4.3] [arxiv:2307.04577 eq. 1]. The card's Paper vs code section lists the details. - AnyTeleop: the collaborative example says operator #2 controls a human hand, while the figure caption says operator #2 acts as a robot [arxiv:2307.04577 §7.2] [fig. 5]. - GEX: the paper gives 23 DoF in total, with an 11-DoF hand [arxiv:2506.04982 §1] [arxiv:2506.04982 §2.1]. The released URDF files contain 10 and 12 revolute joints, which is 22 in total. The card's Paper vs code section records this. - HACTS: the text places XL430 motors on the first three links [arxiv:2503.24070 §3.2], but the bill of materials lists two XL430 and five XL330 motors [arxiv:2503.24070 tab. 1]. - Shared control: the text gives eta² = .248 for MDMT [arxiv:2402.02758 §4.3], while the table gives .234 [arxiv:2402.02758 tab. 1]. - Intent detection: the classification result is called accuracy in one section and F1 in another [arxiv:2402.10220 §IV-C] [arxiv:2402.10220 §V-A]. The forecasting loss is RMSE in one section and MSE in another [arxiv:2402.10220 §III-G] [arxiv:2402.10220 §IV-E]. - Humanoid assistance: the paper states that the blend starts at α(0) = 0 and reaches α(1) = 1 [arxiv:2411.01014 §2.7]. The logistic form in eq. 10 cannot reach either value with finite parameters [arxiv:2411.01014 eq. 10]. The paper gives no parameter values, so an implementation has to choose them. - Glove force rendering: Method II is described as based on a stiffness estimate [arxiv:2412.02613 §3.2.2], but the final rendered force does not use that estimate [arxiv:2412.02613 eq. 6] [arxiv:2412.02613 eq. 9]. - Micro-manipulation: the abstract refers to a stability analysis that the body does not contain [arxiv:2412.06917 p. 1] [arxiv:2412.06917 p. 13]. The force scale-up is given as about six orders of magnitude [arxiv:2412.06917 p. 12], while the force plots are in µN [arxiv:2412.06917 p. 9]. - Non-expert study: 25 towers at about 40 s each would take longer than the 10-minute run [arxiv:2410.18727 §V-A] [arxiv:2410.18727 §IV]. The paper does not reconcile this. Its trust result is also described as the opposite of an earlier cross-country study the authors cite [arxiv:2410.18727 §V-C]. The caption says a low NARS score means more trust, while the text reads the Japanese group as more trusting [arxiv:2410.18727 fig. 12] [arxiv:2410.18727 §V-C]. ## Papers - [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](/papers/1910.03135): bare-hand tracking from four RGB-D cameras, with neural priors for a model-based tracker and fingertip-vector retargeting. Author-reported on 15 tasks with two pilots, with no benchmark comparison. - [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](/papers/2307.04577): camera-based teleoperation of arm-hand robots with fused RGB or RGB-D cameras, a browser viewer, and a collaborative mode. Author-reported comparisons with Telekinesis and with a simulator-specific demonstration baseline. - [GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove](/papers/2506.04982): a glove and a three-finger hand, each with joint motors, plus a retargeting setup, sold as a low-cost kit. Qualitative results only. - [HACTS: a Human-As-Copilot Teleoperation System for Robot Learning](/papers/2503.24070): a leader arm with two-way joint sync, used to take over policies and record corrections. Author-reported success rates on three real tasks with ACT, one with Diffusion Policy, and one online RL task. - [Effects of Shared Control on Cognitive Load and Trust in Teleoperated Trajectory Tracking](/papers/2402.02758): a 24-person study of blended robot and human commands. Self-reported load and trust, with tracking error as the one objective measure. - [Towards Semi-Autonomous Robotic Arm Manipulation Operator Intention Detection from Force Data](/papers/2402.10220): classifies and forecasts glovebox tasks from a robot arm's own joint and force data. Author-reported on small datasets. - [Mixed Reality Teleoperation Assistance for Direct Control of Humanoids](/papers/2411.01014): learned movement primitives and affordance templates assist a humanoid's grasp and door opening. Feasibility study with eight people. - [Effective Virtual Reality Teleoperation of an Upper-body Humanoid with Modified Task Jacobians and Relaxed Barrier Functions for Self-Collision Avoidance](/papers/2411.07534): VR trackers mapped to joint groups, with a soft self-collision term in a QP-based solve. Qualitative demonstrations. - [Grasping and Rolling In-plane Manipulation Using Deployable Tape spring Appendages](/papers/2412.00268): a two-arm tape-spring gripper, teleoperated by joystick in its demonstrations. Author-reported tip error on 16 targets. - [Haptic Stiffness Perception Using Hand Exoskeletons in Tactile Robotic Telemanipulation](/papers/2412.02613): ten participants discriminate soft samples through an exoskeleton glove that renders force. Two force-rendering methods compared. - [Design and Control of an Ultra-Slender Push-Pull Multisection Continuum Manipulator for In-Situ Inspection of Aeroengine](/papers/2412.03508): a tendon-driven manipulator teleoperated by joystick, with a decoupling control law. Qualitative results. - [A Teleoperation System with Impedance Control and Disturbance Observer for Robot-Assisted Rehabilitation](/papers/2412.03619): two planar robots with impedance control and a disturbance observer, with virtual-spring force feedback. Qualitative trajectory tests. - [Haptics in Micro- and Nano-Manipulation](/papers/2412.06917): bilateral haptic control of magnetically guided micro-robots, with a PI force controller. Single-trial experiments. - [Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan](/papers/2410.18727): 74 non-experts teleoperate three arms through a digital twin. Author-reported, with robot, country and recruitment confounded.