HACTS: a Human-As-Copilot Teleoperation System for Robot Learning

In plain words

Teaching a robot arm by hand usually works like a remote-control car: a person moves a leader device, the robot copies it, and the robot never tells the person where its own arm is. That makes it hard for the person to step in smoothly while the robot is doing the task on its own.

HACTS builds a cheap leader arm out of 3D-printed parts and small servo motors. The two arms are kept in sync in both directions. When a learned policy drives the robot, the leader arm moves along with it, so the person can grab the leader and correct the motion at any point. The corrections are recorded as extra training data.

The authors use this data in two ways: to train imitation-learning policies, where the corrections help the robot recover from mistakes, and to fine-tune a policy with reinforcement learning, where short human corrections guide online learning.

TL;DR

• HACTS is a leader-follower teleoperation arm with bilateral joint synchronisation: the leader follows the robot and the robot follows the leader §3.1 fig. 2.
• Hardware is 3D-printed with low-cost servo motors; the stated cost is below $300 and the bill of materials totals $247 §4.1 tab. 1.
• Human interventions during policy rollouts are recorded as action-correction data. Author-reported gains: ACT on three real tasks with correction data tab. 2, DP on one OOD task tab. 3, and RLPD-HACTS on CloseBin tab. 4.
• No official code was found, so the design cannot be reproduced from software.

Problem

Most teleoperation devices control the robot in one direction only: the operator sends commands and gets no real-time feedback from the robot §1. If a policy is running autonomously and a person wants to intervene, the robot can make sudden movements or the intervention can lose control §1. Existing bilateral systems that give force or tactile feedback need follower hardware that can sense force or touch, which many robots do not have §1 §2.1. The paper's stated gap is a bilateral, joint-level synchronisation that works on ordinary position-controlled robots §1.

Assumptions: the follower must accept joint position commands and report joint positions §3.2 ("we only need to synchronize position information").

Method

Hardware (leader device). The leader is a miniature arm whose kinematic structure is derived from Denavit–Hartenberg parameters of the UR5 with the link lengths scaled down §3.2. Each joint is a Dynamixel servo that can be read passively and driven actively §3.2. The scale factor is not stated.

Software. A Dynamixel API reads and writes motor positions §3.3. The pipeline has two directions:

1. Leader-to-follower (demonstration collection). The leader joint readings are corrected by an initial offset calibration, then sent to the robot controller as joint targets §3.3 fig. 2.
2. Follower-to-leader (reverse sync, used when a policy drives the robot). The robot's joint positions are first reverse-offset compensated back to raw motor values, then sent to the leader motors, so the leader arm tracks the robot §3.3.
3. A foot pedal switches between autonomous operation and human intervention §3.3.

The pseudocode below restates this text. The paper contains no algorithm box and no equations, so the function names and structure are ours.

# HACTS joint synchronisation, restated from the text  [§3.1] [§3.3]
# No algorithm box in the paper; the offset values and rates are not stated.
# Input:  leader_joints (J,), robot_joints (J,), pedal_mode
# Output: robot_targets (J,), leader_targets (J,)

def sync_step(leader_joints, robot_joints, pedal_mode):
    # Leader-to-follower: demonstration and human-driven control  [§3.3]
    if pedal_mode == "teleoperate":
        # Calibrated leader readings become robot joint targets
        robot_targets = apply_offset_calibration(leader_joints)
        return robot_targets, None
    # Follower-to-leader: policy drives robot, leader mirrors it  [§3.3]
    # Robot readings are restored to raw motor values first
    leader_targets = remove_offset_compensation(robot_joints)
    return None, leader_targets

Imitation learning (ACT and DP). The policies take RGB images and robot proprioception as input and output joint-position control signals §4.2. Human demonstrations come from leader-to-follower teleoperation. Corrections are recorded when a human takes over a policy rollout, then mixed with the original demonstrations for retraining §4.2. ACT is used for three settings and Diffusion Policy for the fourth §4.2.

Reinforcement learning (RLPD-HACTS). The pipeline has three stages §4.4:

1. Train a binary reward classifier on a pretrained ResNet-10 that takes the front and wrist camera images, using 200 positive and about 1,000 negative samples collected with HACTS §4.4.
2. Collect 20 expert trajectories with HACTS and train a behaviour-cloning policy on ResNet-10 embeddings, gripper state and the relative 6D end-effector pose §4.4.
3. Load the pretrained actor and fine-tune with RLPD, with short human corrections collected through HACTS. The classifier is queried every step and returns 1 when the task is complete, else 0 §4.4.

No equations are numbered in the paper.

Hyperparameters

Results

Author-reported. Setup: three real UR5 manipulation tasks for IL (OpenBox, SteamBun, UprightMug) with ACT and Diffusion Policy; one task (CloseBin) for online RL with RLPD §4.2 §4.4. The baseline for IL is a policy trained on the full 100 expert demonstrations (full-ACT) or on the 50 base demonstrations (pre-ACT) §4.2. Success rates are from 10 rollouts per cell §4.2.

ACT, three settings tab. 2:

Settings: EADC is equal-amount data comparison, FCID is failure correction in distribution, ODSS is out-of-distribution static object placement §4.2.

Diffusion Policy, ODDS on OpenBox only tab. 3:

ODDS is out-of-distribution dynamic: the target is moved during execution §4.2.

RLPD-HACTS on CloseBin, 10 trials tab. 4 fig. 5 fig. 6:

Human intervention length falls from a peak to under six steps over online training fig. 6 §4.5.

Paper vs code

No official code. Searches: the paper and its abstract link no repository, and the arXiv source has no code or project URL. GitHub repository search for "HACTS", "human-as-copilot" and "bilateral teleoperation" returned no related repository. The GitHub user accounts with the name x-humanoid (and its variants) were checked: they hold no teleoperation code. The only repository found under them is a project page with no description, which is not code for this paper. Repositories named "HACTS" that exist are unrelated and were not used.

Pitfalls

• Motor count conflicts. The text puts XL430 motors on the first three links §3.2, but the bill of materials lists two XL430 and five XL330 tab. 1. Build from the text's per-link assignment only after confirming the count.
• Workstation setup. Section 4.1 describes two workstations, each with a wrist camera and extra cameras, and then describes one integrated RTX 4090 workstation §4.1. The actual layout is unclear.
• Cost claim. The stated "lower than $300" §4.1 is looser than the $247 total tab. 1. The bill is for the UR5 version only.
• Leader scale factor not stated. The leader's links are scaled from the UR5 Denavit–Hartenberg parameters "by a factor" §3.2, but the value is not given. The leader cannot be reproduced without it.
• Offset compensation is described, not specified. The offset calibration and reverse offset compensation are described qualitatively §3.3, with no formula and no values. The sign convention, units and per-joint offsets are not stated. Joint limits, torque limits and safety stops are not stated.
• Control rate. Only the RL environment rate (10 Hz) is given §4.4. The teleoperation and policy control rates are not stated.
• Recording start rule. For the EADC setting, data recording begins "once the robotic arm reached a certain level of progress" §4.2. The threshold is not stated, and it determines which trajectories count as intervention data.
• Small evaluation. Each IL cell is 10 rollouts, so each success rate moves in 10% steps §4.2. No seeds, variance or confidence intervals are reported. The text's claim that HACTS data replace expert data "without any performance drop" §4.3 rests on single 10-rollout comparisons.
• Unequal data sources. HACTS-ACT gets 50 base demonstrations plus 50 intervention trajectories, and full-ACT gets 100 expert demonstrations §4.2. The paper attributes HACTS's win in EADC to the correction data covering failed states §4.3, which is a claim, not a controlled test.
• Zero baselines in OOD. ODSS uses ACT on all three tasks, while ODDS uses DP on one task only tab. 3. The two OOD results are not comparable, and every baseline scores 0% there §4.3.
• Validation set. Checkpoint selection uses the lowest validation loss §4.2, but the validation split is not described.
• Reward signal ambiguity. The text says the classifier returns 1 or 0 §4.4. The episode-end rule uses a logit threshold of 0.85 fig. 6. The paper does not say whether the policy sees the binary output or the logit, and it reports no classifier accuracy.
• RL baseline missing. The text compares episode length against BC §4.5, but Table 4 has no BC success rate and BC's numbers are not given.
• RLPD internals not stated. Critic architecture, update-to-data ratio, buffer size, optimiser, learning rate and batch size are not stated §4.4.
• Intervention-length definition. The paper defines it as total intervention steps divided by the number of consecutive interventions §4.5. The "consecutive" counting rule is not formalised.
• Generality. The text says the design is compatible with almost any robot with position control §3.2 §2.1, but only the UR5 was used §4.1.
• Reproducibility from the paper alone. Printable parts, the motor-to-joint mapping and the calibration routine are not released. Plan to build the hardware from the BOM and the text, and expect to write the synchronisation code.

Lineage

• Builds on: ACT, the Action Chunking Transformer, which the paper cites as arXiv:2304.13705 (its bibliography) §4.2; Diffusion Policy §4.2, with no arXiv id in the source; RLPD §4.4, an offline-to-online RL method; HIL-SERL, cited as arXiv:2410.21845 in its bibliography §4.4; Gello and ALOHA as low-cost kinematically equivalent leaders §2.1.
• Compared against: Bi-ACT and ALPHA, a bilateral system that needs force or tactile feedback §2.1 [kobayashi2025alpha]; FACTR, a bilateral system with force feedback §2.1 [liu2025factr]; RoboCopilot, described as a concurrent work with a similar human-in-the-loop setup §2.1.
• Superseded by: not stated.