Mixed Reality Teleoperation Assistance for Direct Control of Humanoids

In plain words

Driving a humanoid by copying your own arm and hand motions is intuitive, but a human body and a robot body do not map onto each other cleanly. Small mismatches turn into missed grasps and repeated attempts, and the operator gets tired and frustrated. This work lets the operator give the robot a rough version of the motion, and the robot fills in the rest using motions it has learned from practice.

The robot first works out which task the operator has started, using the object it can see. It then bends its stored motion so that it fits the operator's first movements and the object's position. For the last part of a grasp it switches to a stored, object-specific sequence of actions and blends smoothly from one to the other. The operator sees the planned motion as a ghost preview, can accept or reject it, and then moves it step by step with a joystick. The authors report that people completed a door-opening task more often, and faster, with this help than with direct control alone.

TL;DR

• For the Nadia humanoid, the detected task is recognised from the operator's first motion, its learned ProMP is conditioned on that motion and the object pose, and the output is blended into an Affordance Template where one exists §2 §2.3 §2.4 §2.7.
• The ProMPs are learned from fewer than 30 demonstrations recorded in a simulated environment §2. The door task used 20 demonstrations fig. 5 and the punching task 27 fig. 7.
• Author-reported: RMS difference between the assisted and the user's own punching trajectories is 2.35 cm (SD 0.87) for left-hand position tab. 1 §4.1.
• Author-reported feasibility study with eight IHMC members: with assistance, 100% of door-opening trials were completed, against 75.0% (experts) and 58.3% (novices) with direct control. Time on task was 40.2 s and 43.3 s with assistance, against 81.1 s and 86.6 s tab. 2 §4.2.
• No official code was found. The main risks are an unsatisfiable blending function, transposes that do not agree across equations, and a very small study. See Pitfalls.

Problem

Direct control, where the robot copies the operator's own motion, is intuitive. Differences between human and robot kinematics cause errors, multiple attempts and operator frustration §1. The authors also note that camera-based direct control gives poor situational awareness, and judges at the ANA Avatar XPRIZE finals were confused about where the robot was §1. The paper's goal is to raise the efficiency and accuracy of teleoperated tasks while keeping the robot's motion human-like §1 §2.

The method assumes: demonstrations of each task recorded by an operator in simulation §2; object poses from ArUco markers, with a manual override in the interface §2.5; a CAD model per object for display §2.5; and an Affordance Template for each object that needs physical interaction §2.6. Where no affordance exists, the robot runs the ProMP alone §2.7 §4.1.

Method

The system has an offline learning phase and an online phase. Offline, the operator performs each task in several different ways in simulation, and one ProMP is learned per task §2 fig. 2. Online, the system recognises the task from the first motion, conditions the ProMP on the operator's input and the object pose, and passes the mean trajectory to the whole-body controller. If an affordance exists, the trajectory is blended into it §2 fig. 2.

1. ProMP model §2.1 eq. 1. A trajectory is a linear basis-function model with a weight vector w, plus observation noise. The weights are Gaussian with parameters θ = {μ_w, Σ_w} §2.1. Eq. 1 is ξ_y(t) = Φ_t w + ε_ξ, with ε_ξ drawn from N(0, Σ_ξ) eq. 1.
2. Learning §2.2 eq. 2 eq. 3. A phase variable υ in [0, 1] removes the dependence on duration §2.2. Each demonstration i gets its own weight vector by ridge regression, with a very small ridge factor λ eq. 2. The mean μ_w and covariance Σ_w of the weights are the sample mean and sample covariance over the D demonstrations eq. 3. Each ProMP models one body part and is learned in the object frame, so it is not tied to a particular robot frame §2.2.
3. Recognition §2.3 eq. 4. Recognition starts when the derivative of the observed end-effector trajectory exceeds a threshold after the user turns on assistance §2.3. The chosen task k is the one whose ProMP mean is closest to the first n_obs observations, measured as a sum of norms, with n_obs about a quarter or a third of the motion's duration §2.3 eq. 4. During this step the ProMP is time-scaled to the mean demonstration duration §2.3.
4. Conditioning §2.4 eq. 5 eq. 6 eq. 7 eq. 8. The weight posterior is updated by Bayes' rule, using the observed target y*_t and its noise Σ*_y eq. 5. The gain L eq. 8 gives the updated mean eq. 6 and covariance eq. 7. If the motion capture is accurate, Σ*_y can be set close to zero §2.4. Conditioning is what adapts the motion to the operator's preferred way of doing the task §2.4.
5. Object pose §2.5. ArUco markers estimate the object pose. The authors describe this as a temporary solution until a state-of-the-art detector is integrated §2.5. The interface shows a virtual object built from CAD data at the estimated pose, and the operator can move it by hand to match the point cloud from the robot's depth camera and LIDAR §2.5.
6. Affordance Templates §2.6. An AT pairs 3D object geometry with a sequence of robot actions in object-centred coordinates. The user registers waypoint frames on the virtual object, and these are converted into end-effector frames for the robot §2.6.
7. Blending §2.7 eq. 9 eq. 10. The ProMP output and the first sample of the AT trajectory are mixed with a weight alpha that rises from 0 to 1 over the blending samples eq. 9 eq. 10. The transition has to start before the ProMP ends, because the centroid of the object may not match the grasp point in the AT §2.7.
8. Robot controller §2.8. A momentum-based whole-body controller, framed as a quadratic program, tracks a desired rate of change of momentum. It can also track objectives for pelvis height, chest orientation and arm configuration. The QP outputs joint accelerations and contact wrenches, and the actuator torques come from inverse dynamics §2.8. The references for the tracked body segments come from an inverse-kinematics QP that checks feasibility and safety §2.8.
9. Interface §3 fig. 3 fig. 4. The operator can drive the robot with a mouse and keyboard, or with a VR headset and controllers in kinematics streaming §3. In streaming mode a yellow ghost robot shows the IK solution §3. During execution the ghost also shows how closely the robot follows the reference §3. The assistance has four steps: pre-activation, generation, validation and execution fig. 3. In validation a gray ghost preview and spline trajectories are shown, and the operator accepts or rejects them with a controller button §3. In execution the operator tilts a joystick to move along the trajectory, and stopping the tilt pauses it §3. The interface is organised around observability, predictability and directability from Coactive Design §3.

Pseudocode: ProMP learning

# Learn one ProMP from demonstrations  [§2.2] [eq. 2-3]
# Input:  demos: D stacked trajectories xi_i (one per demonstration)
#         basis: Phi evaluated on the phase samples  [eq. 1] [§2.2]
#         ridge: scalar ridge factor, 1e-12 in the paper  [§2.2]
# Output: weight mean mu_w and weight covariance sigma_w
def learn_promp(demos, basis, ridge):
    # One weight vector per demonstration, by ridge regression  [eq. 2]
    weights = []
    for trajectory in demos:
        normal_matrix = transpose(basis) @ basis + ridge
        w_i = inverse(normal_matrix) @ transpose(basis) @ trajectory
        weights.append(w_i)
    # Sample mean of the weight vectors  [eq. 3]
    mu_w = mean(weights)
    # Sample covariance, normalised by D (the number of demonstrations)  [eq. 3]
    sigma_w = zeros(len(mu_w), len(mu_w))
    for w in weights:
        sigma_w += outer(w - mu_w, w - mu_w) / len(demos)
    return mu_w, sigma_w

Pseudocode: recognition and conditioning

# Choose the task, then condition its ProMP  [§2.3] [§2.4] [eq. 4-8]
# Input:  promps for the detected object; observed times and targets;
#         noise: observation covariance Sigma*_y
# Output: conditioned mean and covariance of the chosen ProMP
def recognise_and_condition(promps, times, y_star, noise):
    # Sum of norms over the observed times, for each candidate  [eq. 4]
    distances = []
    for promp in promps:
        total = 0
        for t, y in zip(times, y_star):
            total += norm(y - phi(t) @ promp.mu_w)
        distances.append(total)
    promp = promps[argmin(distances)]
    # Basis matrix at the observed times  [§2.4]
    phi_obs = basis_at(times)
    # Gain; eq. 8 is written with Phi_t, not stacked over times  [eq. 8]
    gain = (promp.sigma_w @ phi_obs
            @ inverse(noise + transpose(phi_obs) @ promp.sigma_w @ phi_obs))
    # Posterior mean and covariance; eq. 6-7 use Phi^T  [eq. 6-7]
    mu_hat = promp.mu_w + gain @ (y_star - transpose(phi_obs) @ promp.mu_w)
    sigma_hat = promp.sigma_w - gain @ transpose(phi_obs) @ promp.sigma_w
    return mu_hat, sigma_hat

Pseudocode: blending into an affordance

# Blend the ProMP output into the Affordance Template  [§2.7] [eq. 9-10]
# Input:  y_promp (N_b + 1, n) ProMP samples; y_at_first (n,) first AT
#         sample; a, b, c: blending parameters (values not stated)
# Output: refined reference (N_b + 1, n)
def blend(y_promp, y_at_first, a, b, c, n_b):
    refined = []
    for i in range(n_b + 1):
        # Normalised sample position x = i / N_b  [§2.7]
        x = i / n_b
        # Logistic blending coefficient  [eq. 10]
        alpha = 1 / (1 + a * exp(-b * (x - c)))
        # Mix of the ProMP sample and the first AT sample  [eq. 9]
        refined.append((1 - alpha) * y_promp[i] + alpha * y_at_first)
    return refined

Hyperparameters

Results

Author-reported. Two evaluations: an RMS comparison on a punching task in simulation or on the robot (not stated which) §4.1, and a door-opening feasibility study with eight IHMC members §4.2. No statistical tests are reported.

Punching: RMS error between the adapted ProMP and the user's test trajectory, mean (SD) over 10 test trajectories tab. 1 §4.1

Door opening: feasibility study, mean (SD) tab. 2 §4.2

DC is direct control and DC-AA is direct control with assistive autonomy tab. 2. A failed attempt is a trial in which the robot could not grasp the handle, unlock the door after turning it, or both §4.2.

Qualitative results. For the door, the updated ProMPs follow the user's intended approach, and the blend moves smoothly from ProMP to AT fig. 5 fig. 6 §4.1. For punching, the updated ProMPs follow the user's chosen technique (jab, hook or uppercut) for the test trajectories shown fig. 7 §4.1. The paper gives no numbers for these figures.

Paper vs code

No official code.

Searches run: the abstract, the body text and the bibliography contain no code link §1 §4.1. The paper's links are to the Nadia product page, supplementary videos and the dataset DOI 10.5281/zenodo.8215964 §1 §4.1. The Zenodo record was read through its description page only, not the archive. It is CC BY 4.0, by the first author, and lists demonstrations, test trajectories, updated ProMPs and "a RMS comparison calculator" in one ZIP. The page does not say whether the calculator is code, and it was not downloaded. The supplementary video was not opened.

GitHub search for promp inside the IHMC organisation (ihmcrobotics) returned ihmcrobotics/promp-java. That repository is a Java binding of a third-party C++ ProMP library and is licensed LGPL-3.0. Its README does not mention Nadia, affordance templates, mixed reality or this paper, so it is not treated as this paper's code. Searches for affordance templates and teleoperation inside the same organisation returned nothing.

Because no code was found, no implementation can be checked against the paper.

Pitfalls

• The blending function cannot meet its own constraints. The paper requires α(0) = 0 and α(1) = 1 §2.7, but the logistic form in eq. 10 has range (0, 1) for finite real a, b and c eq. 10. Setting α(0) = 0 would need a·e^(bc) to be infinite, and α(1) = 1 would need a·e^(−b(1−c)) = 0, which only holds for a = 0, and then α is constant 1. The paper gives no values for a, b and c, and no fix. Any implementation has to choose one, and the choice changes how early the transition happens.
• The transposes do not agree across equations. Eq. 1 uses Φ_t w, with Φ_t written as an n·m matrix eq. 1. Eq. 4 uses Φ_t μ eq. 4. Eq. 6 uses Φ_tᵀ μ eq. 6 and eq. 7 and 8 use Φᵀ Σ Φ eq. 7 eq. 8. The pseudocode above follows each equation as written. A block-diagonal Φ with n·m columns would have shape n × nm, which would make Φ w a valid n-vector, but the paper does not state the shape. Choose one convention and check the shapes.
• Eq. 1 and the noise symbols. The text calls Σ_ξ the observation noise variance §2.1, and eq. 1 draws ε_ξ from N(0, Σ_ξ). Σ_ξ is not used again. The noise used in conditioning is Σ*_y, which the paper does not give a value for except to say it can approach zero §2.4.
• Ridge factor. Eq. 2 adds λ as a scalar to ΦᵀΦ rather than λI eq. 2. Whether the paper meant λI is not stated. With λ = 1e-12 the regularisation has very little effect, so the inverse can be ill-conditioned when the basis matrix is nearly rank-deficient. The basis, its scale and the phase sampling are not stated.
• Maximum likelihood and the normaliser. The text says the parameters are estimated with "a maximum likelihood estimation algorithm" §2.2, but eq. 3 gives the sample mean and a covariance divided by D. Use eq. 3 as written, and note that it is a biased estimator.
• Recognition uses norms, not squared norms. Eq. 4 sums the norms of the residuals over T_obs eq. 4, so it is not the squared-error form that the Gaussian likelihood would give. The paper does not say how the observed timing is converted to the phase variable, and the duration scaling to the mean demonstration length §2.3 is the only time mapping stated.
• Conditioning on several observations. The equations condition on one time t eq. 5 eq. 8. The text says "initial observations" §2.4 and eq. 4 uses n_obs observations. How several observations are stacked, or applied one at a time, is not stated. The pseudocode above stacks them, which is an assumption.
• No real-time speed adaptation. The system cannot respond to changes in the operator's speed during live input §5. The paper names Interaction Primitives as a remedy §5.
• Out-of-distribution observations. Conditioning on observations that deviate strongly from the learned standard deviation can give improper conditioning §5. The authors plan a validation step that maps outliers to the nearest point of the learned model, but this is not implemented in the paper §5.
• Object pose is a stand-in. ArUco markers are described as temporary §2.5. Accurate pose is central to the claim that the system works "without any real-world training data" §2, so the sensitivity to pose error is not measured.
• Demonstrations were recorded in simulation, but §4.1 does not say so. §2 says demonstrations were recorded in a simulated environment §2. §4.1 says only that a single expert performed the task several times §4.1.
• Text and Table 1 disagree on orientation. The text gives "no more than 0.05 rad" with SD 0.02 for orientation §4.1. Table 1's left-hand orientation row is 0.03 (0.01). The 0.05 (0.02) value is the left forearm row tab. 1. This card uses the table values.
• The RMS measures agreement with the user, not success. The RMS compares the adapted ProMP with the same user's test trajectory, after observing about a third of it §4.1. A low value shows agreement with that user's motion. It does not measure whether the punch succeeds.
• Punching was not compared to anything. It has no baseline and no success criterion §4.1. The text does not say whether the punching test trajectories ran on the robot or in simulation.
• Feasibility study is small and undefined in places. Four experts and four novices, three trials per method at each of three positions §4.2. The SD basis (per trial or per person) is not stated. Completion rates are consistent with 12 trials per group, but the paper gives no per-trial data. No significance tests are reported.
• Blending needs an affordance. The punching task has no AT, so no blending is involved §4.1 fig. 7.
• Number of ProMPs per door task is not stated. The door task is said to use learned ProMPs with reference to the door handle frame fig. 5 §4.1, but the number of body-part ProMPs for it is not given. For punching, extra ProMPs cover the forearm and chest orientations §4.1.
• Controller details are missing. No gains, weights, control rate, or joint-limit handling is given for the QP controller or the IK QP §2.8.
• Equation and figure numbers. Equation numbers in this card were counted from the LaTeX source in order of appearance. The two-row align in §2.4 gives eq. 6 and eq. 7. Figure numbers follow source order. Check both against the compiled PDF before citing them elsewhere.

Lineage

• Builds on: Probabilistic Movement Primitives [paraschos2017] §2.1; Affordance Templates [pettinger2020] §2.6; Coactive Design principles [johnson2014coactive] §3; momentum-based whole-body QP control [koolen2016] §2.8; inverse-kinematics QP [jorgensen2019] §2.8. None of these has a card in this wiki.
• Compared against: direct control by the same operator without assistance §4.2. No other method is compared.
• Related shared-control work cited in the introduction: Dragan et al. [dragan2013a], Rakita et al. [rakita2019], Rahal et al. [rahal2019], Javdani et al. [javdani2018], Broad et al. [broad2020] §1.
• Extended by: Interaction Primitives [campbell2017], named as a way to add real-time speed adaptation §5.
• Superseded by: not stated in this wiki.