Effective Virtual Reality Teleoperation of an Upper-body Humanoid with Modified Task Jacobians and Relaxed Barrier Functions for Self-Collision Avoidance

In plain words

A person wearing a VR headset and holding two controllers can steer a humanoid robot's arms, head and waist. The hard part is turning those few tracked points into moves for the many joints of the robot. Here, each tracker is connected to a fixed group of joints: the headset's rotation moves the neck, each hand moves its own arm, and the waist tracker turns the torso. Because every joint answers to one clear input, the operator can predict what the robot will do.

The robot should also not hit itself while it moves. Instead of rejecting a command when a collision is about to happen, the solver treats the closeness of two body parts as a soft constraint. It can then return a best-effort motion rather than no solution.

The authors show the system on a real humanoid. Operators with little experience could do table-top pick-and-place after a short explanation, and the robot could pick up a box with two hands and hand it over. The paper gives no measured numbers for these tests.

TL;DR

• Four 6-DoF VR trackers are mapped one-to-one onto joint groups of the 17-DoF Apptronik Astro by editing each task Jacobian §II fig. 1.
• Self-collision avoidance is a soft, relaxed-barrier term inside a QP-based inverse-kinematics solve, with a quadratic expansion of the barrier §II eq. 3 eq. 4.
• Results are qualitative: pick-and-place by new operators and a two-handed box pickup and handover to a human §III §IV.
• No success rates, timings or collision statistics are reported; there are no baselines in experiments §IV.
• No official code was found.

Problem

With more robot joints than tracker degrees of freedom, many joint solutions match one tracker pose. Earlier work resolved this with posture tasks or hand-tuned task weights. Tuning can be hard, and poor weights can cause oscillations §I. Using more trackers, such as a full-body suit, makes the mapping over-determined and still needs weight tuning §I.

Self-collision is a second problem. Operators should not have to watch for it while they teleoperate. Rejecting self-colliding joint commands causes abrupt pauses during a task. Repulsive potential fields are an alternative, but they can get stuck in local minima in cluttered scenes §I §II.

The approach uses one tracker per tracked body part fig. 1, splits the robot joints into torso pitch, torso yaw, neck, and left and right arm groups §II, and uses a signed distance between collision pairs §II. The paper does not state the tracking hardware, the calibration procedure, or the control rate.

Method

The paper has four numbered sections (§I to §IV) and no numbered subsections. Citations use the roman numerals as printed; the three run-in topics of §II are cited as §II with their names in the text.

Joint partition. The robot joint vector is split into torso pitch, torso yaw, neck, left arm and right arm groups §II. Each tracker is assigned to one or more of these groups.

Modified task Jacobian. For a task with Jacobian J, the paper post-multiplies J by a selection matrix S to remove joint columns, giving J_m = J S §II, footnote. The worked example is a left-hand task that uses only the left-arm columns and ignores the torso and neck §II eq. 2.

Tracker-to-joint assignment. The caption of fig. 1 lists four 6-DoF trackers (headset, left controller, right controller, waist). The headset orientation moves the neck. Each hand controller's pose moves its arm. The waist tracker's vertical axis moves the torso yaw. The headset's forward position moves the torso pitch, which leans the robot fig. 1.

Inverse kinematics. Joint velocities are found with a QP-based IK solver §II. The cost sums three parts: a weighted task error per task, a regularizer per task, and a relaxed-barrier term per collision pair §II eq. 3. The compiled text does not state how joint velocities become joint commands.

Relaxed barrier. Each collision pair has a signed distance h_j(q) between convex shapes §II. The paper uses relaxed barrier functions as soft constraints so that best-effort solutions are returned instead of infeasibility. It also says this skips the inequality evaluation step in most QP solvers §II.

Quadratic expansion. The barrier term is replaced by a second-order expansion in the joint velocity dq §II eq. 4. The expansion uses the value of B at the current distance, its gradient through h, and its second derivative.

Pseudocode for the modified Jacobian §II eq. 2:

# Modified task Jacobian: keep only the joints of one tracker  [§II] [eq. 2]
# Input:  task Jacobian (dim_x, n); joint mask for the tracker's joint set
# Output: modified Jacobian (dim_x, n)
def modified_jacobian(jacobian, joint_mask):
    # The paper writes J_m = J S but does not define S; a diagonal 0/1
    # matrix that keeps the assigned columns is inferred.
    selection = diag(joint_mask)                    # (n, n)
    return jacobian @ selection                     # (dim_x, n)

Pseudocode for one IK solve §II eq. 3 eq. 4:

# One retargeting solve: joint velocities for the next control step
# Input:  joint state q; tasks (error e, gain K, Jacobian J, weight w);
#         collision pairs (signed distance h, its gradient dh_dq)
# Output: joint velocity dq (the decision variable of the cost)
def cost(dq, q, tasks, pairs):
    total = 0
    for task in tasks:
        # Weighted task error  [eq. 3]; eq. 3 prints a norm, and the
        # squared norm is inferred so that the problem stays a QP
        total += task.w * squared_norm(task.K @ task.e - task.J @ dq)
        # Regularizer W_i: "adaptive", its form is not given  [§II]
        total += dq.T @ task.W @ dq
    for pair in pairs:
        # Relaxed barrier per collision pair, quadratic in dq  [§II] [eq. 4]
        total += barrier_quadratic(pair.h, pair.dh_dq, dq)
    return total

# Quadratic approximation of the relaxed barrier  [§II] [eq. 4]
# Input:  signed distance h; its gradient dh_dq (n,); barrier B
# Output: scalar cost in dq
def barrier_quadratic(h, dh_dq, dq):
    # Value at the current distance; constant in dq  [eq. 4]
    value = B(h)
    # Linear term: gradient of B through h  [eq. 4]
    linear = dot(dB_dh(h) * dh_dq, dq)
    # Second-order term from the second derivative of B  [eq. 4]
    curvature = d2B_dh2(h) * outer(dh_dq, dh_dq)     # (n, n)
    quadratic = 0.5 * dq.T @ curvature @ dq
    return value + linear + quadratic

Equations, as printed:

\dot{x} = J(q)\,\dot{q} eq. 1

\dot{x}_{\text{l,hand}} = [0, 0, 0, \partial x / \partial q_{\text{l,arm}}, 0]\,\dot{q} eq. 2

\min_{\dot{q}} \sum_{i=1}^{N_t} w_i \lVert K_{p,i} e_i - J_i \dot{q} \rVert + \dot{q}^T W_i \dot{q} + \sum_{j=1}^{N_c} \tilde{B}_j(h_j(q), \dot{q}) eq. 3

\tilde{B}(h_j, \dot{q}) = B(h_o) + \Big(\frac{\partial B}{\partial h}\frac{\partial h}{\partial q}\Big)^T \dot{q} + \tfrac{1}{2}\,\dot{q}^T \Big(\frac{\partial h}{\partial q}^T \frac{\partial^2 B}{\partial h^2} \frac{\partial h}{\partial q}\Big)\dot{q} eq. 4

Equation 1 is the first row of the task-velocity equation; the paper's second row is unnumbered. Equation 3 is the only place the IK cost is written; the paper does not give B, its parameters, or the gains.

Hyperparameters

Results

No quantitative results are reported. Author-reported and qualitative only. Setup: one Apptronik Astro, a table-top scene, operators wearing VR equipment §III §IV.

• Box packing and pick-and-place in a table-top environment §I §III.
• A two-handed box pickup and handover to a human, with a button that holds the offset between the hands §III.
• Several people with little VR or teleoperation experience performed pick-and-place after a few minutes of explanation §IV.
• The authors state that self-collision avoidance improved operational safety and responsiveness, because potential collisions were resolved by the IK solver §IV. No collision counts or clearance values are given.

No baselines are run. The comparison with rejecting colliding commands and with potential fields is argued in the text §I §II and not measured.

Paper vs code

No official code.

The abstract and arXiv page have no code link. The paper's only linked code is pymanoid, cited as the IK solver library §II, footnote. It is a third-party project, not the authors' code, so it is not used here.

Searches run: the paper and arXiv page for code or project links (none); GitHub repository search for "apptronik" (no repository owned by Apptronik appeared); GitHub search for "relaxed barrier teleoperation" and "modified task jacobians humanoid teleoperation" (no results); GitHub user search for "stevenjorgensen" (two accounts, both with no public repositories). The organization-scoped search returned an error, so the search inside the Apptronik organization could not be run.

Because no code was found, the points below cannot be checked against an implementation. They are gaps, not discrepancies.

Pitfalls

• Tracker count versus joint count. The text says a minimum set of trackers is used with n_j = n_t §I. Four 6-DoF trackers give 24 tracker degrees of freedom, while the Astro has 17 joints §III. The figure assigns headset orientation to the 3 neck joints, headset forward position to torso pitch, waist vertical axis to torso yaw, and each hand to 6 arm joints, which adds to 17 fig. 1. This is an inferred count. The paper does not say which tracker axes are ignored.
• Norm in the cost. Equation 3 writes the task error as a norm, not a squared norm, while the text calls it a weighted least-squares term §II eq. 3. A norm makes the problem non-quadratic, so a QP solver needs the squared form. The squared form is inferred and not stated.
• Typo in equation 4. The constant term is written B(h_o), but the expansion is for h_j eq. 4. Read it as B(h_j), since the other terms use h_j.
• Relative weight of collision and task terms. The barrier and task terms share one cost. The paper gives no weight for the barrier term or the barrier's parameters. Without these values the balance between collision and tracking cannot be reproduced.
• Regularizer undefined. W_i is called adaptive and is cited to Levenberg-Marquardt-style damping §II. Its form is not in the compiled paper. The LaTeX source has a commented-out draft of W_i (the source file main.tex, lines 119 to 121). It is not part of the published method and should not be implemented as written.
• Soft constraints can be violated. A relaxed barrier returns the best effort rather than failing §II. The paper gives no measured minimum clearance or violation rate, so the safety claim in §IV cannot be quantified from the paper.
• Signed distance is not specified. The paper says collision pairs use convex shapes and signed distances §II, but gives no shape models, no distance routine, and no list of pairs. The introduction names FCL only as an approach it rejects §I.
• Joint velocities, not positions. The solver returns joint velocities §II. How they are integrated and rate-limited is not stated.
• Operator interface details. The clutch behaviour, joystick grasp cycling and the offset button are described briefly §III. Their mappings, filters and safety limits are not given.
• No failure cases. The paper reports successes only. Singularities, tracker dropouts and collision-avoidance failures are not discussed.
• Single robot, qualitative evaluation. All evidence comes from one platform and informal demonstrations. The claim that tracker-to-joint allocation makes teleoperation more intuitive is a hypothesis in the conclusion §IV.

Lineage

• Builds on: relaxed barrier functions [feller2016relaxed] and [grandia2019feedback] §II; signed-distance constraints [schulman2014motion] §II; humanoid self-collision with control barrier functions, arXiv:2207.00692 (named as Khazoom et al., 2022) §II; collision-free MPC [chiu2022collision] §II; Levenberg-Marquardt damping [sugihara2011solvability] §II; the VR interface and prior Valkyrie work [jorgensen2022cockpit] §I §III. Only arXiv:2207.00692 is named with an id; it has no card in this wiki.
• Compared against: potential fields used in the IHMC VR interface §I §II and rejecting colliding commands with a collision checker [pan2012fcl] §I. Both are argued against, not tested.
• Superseded by: not stated in this wiki.