Teleoperation of Humanoid Robots: A Survey

In plain words

Humanoid robots, machines with a human-like body, can be driven remotely by a person when a task is too hazardous for people or too hard for a robot to do on its own. This survey explains how such a remote-control setup is built. The person wears or uses sensors that capture their movement, and software turns that movement into commands the robot can follow while keeping itself balanced. The person gets feedback from the robot, such as pictures, sound, or forces felt through a device, so they can react to what the robot experiences.

The hard parts are balance, since the robot is unstable and has many joints, and the link between person and robot, which may be slow or drop data so that the robot lags behind the person. Two ways to soften this are to let the robot share some decisions with the person, or to let the person feel the robot's balance directly.

The survey also covers how to measure whether such a system is usable and which tasks such robots could be used for.

TL;DR

A 2023 survey of humanoid robot teleoperation. It has no experiments of its own; its numbers come from works it cites §1. It walks through the pipeline: operator sensing and feedback devices §2, retargeting, stabilisation and whole-body control §3, assisted and bilateral modes §4, delayed or lossy channels analysed with passivity §5, and evaluation metrics for usability, situation awareness, workload and presence §6. Read it if you design or evaluate a teleoperated humanoid; it gives no implementation to reuse.

Problem

Teleoperating a humanoid is hard for several reasons the paper lists §1. The robot is highly redundant, with nonlinear, hybrid and underactuated dynamics §1 §3.1. It must keep its balance while the operator drives its limbs §4.3. The operator needs minimal but informative haptic feedback, to cover human errors and communication delays §1. Non-ideal links add delay, packet loss and limited bandwidth §2.1 §5.1. Delay can destabilise bilateral teleoperation, and passivity does not guarantee good performance §5.1 §5.5. The authors also note that autonomy alone does not yet give socially and physically competent behaviour, which is why a human stays in the loop §1.

The paper argues that the field needs an up-to-date survey, because the earlier one it cites is about a decade old §1.

Method

This is a survey, so there is no single algorithm. The paper organises teleoperation into blocks and reviews the methods in each block.

1. Operator sensing §2.2. Human motion is measured with IMU suits, optical systems (depth sensors, motion capture, RGB or stereo cameras), exoskeletons with encoders, treadmills and force sensors. Physiological signals (EMG, EEG, heart rate, eye tracking, skin humidity) can estimate the operator's state §2.2.
2. Retargeting policy choice §2.1. Estimated human and user states select the retargeting policy, and references are sent to the robot §2.1.
3. Retargeting and planning §3.2. Human motion is mapped to robot references. Goal planning uses search methods (A*, D* Lite, RRT, dynamic programming) or reactive methods (MPC or QP, potential fields). Motion generation covers footsteps, upper-body retargeting in task or joint space, and whole-body retargeting §3.2.
4. Stabilizer §3.3 fig. 4. Input references are adjusted for balance using the ZMP criterion, the DCM with ZMP, or contact-wrench control §3.3.
5. Whole-body control §3.4. Velocity or position IK, inverse dynamics and momentum-based control are solved as QPs or with LQR or MPC §3.4.
6. Joint control and state estimation §3.5 §3.6. Joint position or torque tracking, plus estimators (Kalman filters, joint torque estimation, floating-base estimation with IMU and optional exteroceptive sensors) §3.5 §3.6.
7. Feedback to the operator §2.3. Visual (GUIs, VR headsets), haptic (kinesthetic force, tactile, vibrotactile, air pressure, balance feedback) and auditory channels §2.3.
8. Assisted modes §4. Shared control blends the operator input with a robot proposal eq. 5 eq. 6. Supervised and safeguarded control lets the robot act alone or override the operator in hazardous states §4.2. Bilateral control feeds robot dynamics back to the operator; whole-body bilateral coupling is a separate case §4.3 eq. 7. Impedance control and tele-impedance adjust stiffness and damping §4.4.
9. Communication channel §5. Delays, distortion from packet loss and time-stamp issues, and network (passivity) analysis, including wave variables and energy tanks §5.1 §5.2 §5.3 §5.4 §5.5.
10. Evaluation §6. Usability (ISO 9241, SUS), situation awareness (SAGAT), workload (NASA-TLX, SWAT), engagement and presence, plus the autonomy level §6.1 §6.2.
11. Applications §7. Telexistence and telepresence, hazardous environments, manufacturing and research, telenursing, space and service robotics. The card does not summarise these subsections §7.

A schematic of how these blocks connect, drawn from the paper's architecture figure and text, is below. It is not an algorithm from the paper.

# Schematic teleoperation loop  [fig. 2] [§2.1]
# Input:  operator sensor readings, robot state
# Output: joint references for the robot

def teleoperation_step(operator_sensors, robot_state):
    # Measure the operator's motion and, if used, physiology  [§2.2]
    operator_motion = measure_operator(operator_sensors)
    # Select the retargeting policy from the estimated user state  [§2.1]
    policy = select_policy(operator_motion, user_state)
    # Map operator motion to robot references  [§3.2] [§4.3]
    references = retarget(operator_motion, policy)
    # Blend operator input with the robot proposal when assisted  [§4.1]
    if assisted:
        references = blend(references, robot_proposal)
    # Send references and feedback over a possibly lossy channel  [§5]
    send_over_channel(references, feedback)
    # Stabilise the references and track them  [§3.3] [§3.4] [§3.5]
    joint_references = whole_body_control(references, robot_state)
    return joint_references

Key equations, as written in the paper. Symbols are glossed in the lines that follow each equation.

Robot dynamics, with contact wrenches eq. 1 §3.1:

\mathbf{M}(\mathbf{q})\,\dot{\boldsymbol{\nu}} + \mathbf{C}(\mathbf{q},\boldsymbol{\nu})\,\boldsymbol{\nu} + \mathbf{g}(\mathbf{q}) = \mathbf{B}\,\boldsymbol{\tau} + \sum_{k=1}^{n_c} \mathbf{J}_k^{T}(\mathbf{q})\,\mathbf{f}^{c}_k eq. 1

Here M is the inertia matrix, C the Coriolis and centrifugal terms, g gravity, B the actuator selector, tau the joint torques, and f^c_k the k-th contact wrench §3.1.

Linear inverted pendulum, with z0 the constant CoM height eq. 2 §3.1:

\ddot{\mathbf{x}} = \frac{g}{z_0}\,(\mathbf{x} - \mathbf{x}_b) eq. 2

Divergent component of motion (DCM), with time constant b = sqrt(z0/g) eq. 3 §3.1:

\boldsymbol{\xi} = \mathbf{x} + b\,\dot{\mathbf{x}} eq. 3

DCM dynamics, with x_b the ZMP or base point eq. 4 §3.1:

\dot{\boldsymbol{\xi}} = \frac{1}{b}\,(\boldsymbol{\xi} - \mathbf{x}_b) eq. 4

Shared control blend of operator input u and robot proposal r eq. 5 §4.1:

\mathbf{u}^{*} = (1 - \alpha)\,\mathbf{u} + \alpha\,\mathbf{r} eq. 5

Blending weight, where d is the distance to the predicted goal and D a threshold eq. 6 §4.1:

\alpha = \max\!\left(0,\; 1 - \frac{d}{D}\right) eq. 6

Whole-body force feedback, with dimensionless CoM velocities of human (H) and robot (R), external force f_ext' and scale k_H eq. 7 §4.3:

\mathbf{f}_{fb} = k_{H}\left[\left(\dot{\mathbf{x}}'_R - \dot{\mathbf{x}}'_H\right) + \mathbf{f}'_{ext}\right] eq. 7

Passivity of an n-port, with power P = f^T v, stored energy E and non-negative dissipation P_diss eq. 8 §5.4:

\int_0^t \mathbf{f}^T(\tau)\,\mathbf{v}(\tau)\,d\tau = E(t) - E(0) + \int_0^t P_{diss}\,d\tau \geq -E(0) eq. 8

Hyperparameters

The paper sets no hyperparameters of its own. The table lists the numeric values and symbols it gives, with the status of each.

Results

The paper reports no new experiments §1. Its findings are qualitative or summarise cited work:

• At the DARPA Robotics Challenge, most teams used high-level commands and visual aids and did not use bilateral teleoperation, given the link described above §5.1.
• One cited experiment found that time delay raised operator workload and reduced performance §6.3.
• Across cited works, an intermediate autonomy level in which the user stays in the loop was reported to improve situation awareness, reduce workload and help performance under failures §6.2.
• A table lists nine studies by the metrics each uses (usability, situation awareness, workload, engagement or presence, autonomy, subjective, objective, physiological, behavioural). The authors note that behavioural and physiological measures are used less often tab. 2 §6.3.
• The ANA Avatar XPRIZE judges teams on how well they perform tasks and on subjective ratings, across walking, handling objects and social interaction §6.3.

Paper vs code

No official code. The paper's text and arXiv record do not link a code repository. The project page it names, humanoid-teleoperation.github.io, links only the repository of the website itself and a third-party theme. A GitHub search for "humanoid teleoperation survey" returned only the website repository, which belongs to the Humanoid-Teleoperation organisation and holds the survey's webpage, not method code. No repository was cloned or used.

Pitfalls

• This is a survey. Every number in Results and Hyperparameters comes from a work it cites. Check the original paper before implementing anything.
• The equations are summaries of the cited methods, not complete derivations. Eq. 7 and eq. 8 are stated at a high level; the paper refers to the cited works for details §4.3 §5.4.
• Several quantities are symbols with no value: beta in the critical delay, the threshold D in eq. 6, the scale k_H in eq. 7, and the retargeting gain §5.5 eq. 6 eq. 7 §3.2. An implementer must choose or measure them.
• Teleoperation and telexistence (telepresence) are used interchangeably in the paper §2.
• The critical delay for a linear inverted pendulum grows with CoM height. The paper ties it to the LIPM §5.5 eq. 2 and says that delay challenges for balance and velocity in humanoid teleoperation are not yet addressed in the literature §5.5. That claim dates from 2023.
• Passivity does not guarantee good performance, and stability and transparency conflict §5.5. Passivity analysis is a stability tool only.
• The paper states that evaluation metrics are highly task-dependent, so the nine works in the metrics table do not transfer directly to one another §6.1 tab. 2.
• The arXiv LaTeX source comments out three sections (Background, Robot Control, Discussion). The numbering used here is the one the compiled paper shows.

Lineage

• Builds on: the 2019 workshop on teleoperation of humanoid robots, organised by four of this paper's authors, which the paper says it follows §1. The workshop has no arXiv id, so it is not linked here.
• Compared against: Goodrich and Schultz's survey "Human-robot interaction: a survey" §1 and Goodrich, Crandall and Barakova's "Teleoperation and beyond for assistive humanoid robots", which the paper calls a decade old §1.
• Superseded by: not known from this wiki.