DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System

In plain words

DexPilot lets a person control a robot arm and a robot hand by moving their own bare hand in front of four depth cameras. The robot copies the pose of the hand and the shape of the fingers, so the operator can see the result and correct it as they go.

Building this takes three parts: cameras and software that follow the human hand, a mapping that turns the human hand into a robot hand with different joints, and a controller that moves the arm and the fingers. The authors used a glove with coloured patches only at the start, to produce training labels. The system then runs on the bare hand.

They tested it with two trained operators on fifteen tasks, from picking up objects to opening a jar or a wallet. The main limits are the lack of touch feedback, which makes precise tasks hard, and a delay of about one second.

TL;DR

• Vision-only teleoperation of a 23-degree-of-actuation robot (KUKA LBR iiwa7 arm plus Allegro hand), with no glove on the operator at run time §1, §6.
• Hand pose comes from DART, a model-based tracker, seeded by two neural networks trained on labels that a glove-based run produced §5, §6.
• Fingertip-direction retargeting solved per frame with SLSQP maps the human hand onto the Allegro hand §7.1.
• Author-reported: two pilots, 15 tasks, 5 consecutive trials each; keypoint error 9.7 mm on the authors' validation set §6, app. completion times.

Problem

Robotic hands with many actuated joints are expensive to teleoperate. Gloves and motion capture add cost, bulk or a larger footprint §2. Bare-hand tracking is cheap but was not reliable enough for long sessions on its own. The authors report that the model-based DART tracker, without priors, lost the hand in spurious local minima every few minutes §6. Mapping human fingers onto a robot hand with different joint axes is also hard. Earlier retargeting approaches were not good enough for precision grasps such as pinches §7.1.

Method

Section numbers follow the order of the sections in the LaTeX source. So §7.1 is the PDF's VII-A and §7.2 is VII-B. Appendix items are cited by subsection title. Table numbers (Roman) and figure numbers match the PDF.

Hardware §3, fig. 2. The robot is a KUKA LBR iiwa7 R800 arm with a Wonik Allegro hand. The hand has four BioTac tactile sensors at the fingertips and grip tape on the phalanges and palm. Four calibrated, time-synchronised Intel RealSense D415 RGB-D cameras look down at a black-covered table. The paper says depth quality degrades beyond about 1 m. The work volume is 80 × 55 × 38 cm §3.

Three threads on three computers fig. 3. A learning thread produces hand pose and joint priors. A tracking thread runs DART and retargeting. A control thread runs the motion policies and sends joint targets fig. 3.

Step 1, glove-based labelling (first phase) §6, app. GloveNet.

1. The operator wears a black glove with coloured blobs (three on the palm, five on the fingertips). Blob centres are segmented with HSV thresholds.
2. GloveNet (anti-aliased ResNet-50 with a spatial-softmax head) regresses the 2D blob centres per camera from RGB. The authors used about 50K images.
3. Blob centres are lifted to 3D with depth. Each palm blob gets four estimates, one per camera. The estimates are filtered and smoothed over time to give the hand pose.
4. Only the palm blobs are used for pose. The fingertip predictions were judged unstable and dropped app. GloveNet.
5. DART then fits the hand model to the segmented points inside a bounding volume. That volume is set heuristically from the predicted pose §6.

Step 2, bare-hand networks (second phase) §6, fig. 5, fig. 6.

1. The four depth maps are back-projected and fused into one cloud. A plane fit removes the table.
2. Stage 1 is a PointNet++-style network. It takes a uniform subsample of 8192 points and outputs 23 keypoints and a hand segmentation.
3. Stage 2 resamples points on the hand from the raw cloud, using the stage-1 pose and segmentation. It adds random perturbation to that pose. It predicts the 23 keypoints again, trained on keypoint losses only §6.
4. The keypoint read-out uses voting offsets, per-keypoint masks and a weighted average of point coordinates app. PointNet++ keypoint head.
5. JointNet, a two-layer MLP with batch norm and ReLU, maps the 23×3 keypoints to 20 finger joint angles app. JointNet, tab. VI.
6. Both networks are trained on 100K point clouds labelled by DART run with glove priors §6.

Step 3, model-based tracking §5, app. Model-based and Model-free. The hand is an articulated mesh model derived from MANO. It has 20 revolute joints: four per finger, one abduction and three flexion §5. DART is initialised with the network priors. The authors describe a loop where DART runs reliably, its outputs train a network, and that network provides better initialisation for the next round app. Model-based and Model-free.

Step 4, kinematic retargeting §7.1, tab. I. Each retargeting vector r_i points between a fingertip, a palm or a thumb in the human hand model or in the Allegro model. The cost is

\mathcal{C}(q_h, q_a) = \tfrac{1}{2} \sum_{i=0}^{N} s(d_i)\,\lVert r_i(q_a) - f(d_i)\,\hat{r}_i(q_h)\rVert^2 + \gamma \lVert q_a \rVert^2 §7.1, unnumbered

where d_i = ||r_i(q_h)||, r̂_i is the unit vector, and

s(d_i) = \begin{cases} 1, & d_i > \epsilon \\ 200, & d_i \le \epsilon,\ r_i \in \mathcal{S}_1 \\ 400, & d_i \le \epsilon,\ r_i \in \mathcal{S}_2 \end{cases}\qquad f(d_i) = \begin{cases} \beta d_i, & d_i > \epsilon \\ \eta_1, & d_i \le \epsilon,\ r_i \in \mathcal{S}_1 \\ \eta_2, & d_i \le \epsilon,\ r_i \in \mathcal{S}_2 \end{cases} §7.1, unnumbered

S1 holds vectors from a primary finger (index, middle, ring) to the thumb. S2 holds vectors between two primary fingers that are both projected onto the thumb tab. I. Constants: β = 1.6, η1 = 1×10⁻⁴ m, η2 = 3×10⁻² m, γ = 2.5×10⁻³ §7.1. The threshold ε is not stated. The distal joints of the index, middle and ring Allegro fingers are constrained to equal their medial joints §7.1.

# Retargeting cost for one frame  [§7.1]
# Input:  human vectors r_h (N, 3); human unit directions (N, 3)
# Input:  candidate Allegro joint angles q_a
# Output: scalar cost

def retarget_cost(q_a, human_vectors, directions, weights, targets):
    # Allegro task-space vectors r_i(q_a), same pairs as the human side  [§7.1]
    robot_vectors = robot_task_vectors(q_a)            # (N, 3)
    # Scaled human direction, f(d_i) * r_hat_i  [§7.1]
    desired = targets[:, None] * directions            # (N, 3)
    # Per-pair switching weight s(d_i) on the squared error  [§7.1]
    squared_errors = squared_norm(robot_vectors - desired)
    fit = 0.5 * sum(weights * squared_errors)
    # Regulariser that pulls the Allegro hand toward open, weight gamma  [§7.1]
    regulariser = 2.5e-3 * squared_norm(q_a)
    return fit + regulariser

Step 5, solving the retargeting §7.1. The cost is minimised with SLSQP from NLopt, once per frame. The first solve starts at zero angles. Each later solve starts from the previous solution. Forward kinematics between the human and Allegro frames uses the Orocos KDL library. A first-order low-pass filter smooths the output §7.1.

Step 6, motion generation §7.2, fig. 3. Riemannian Motion Policies (RMPs) compute the target Cartesian pose of the Allegro palm from the tracked hand. They also avoid collisions with the table and the operator, modelled as collision planes. Arm joint targets go to a torque-level impedance controller at 200 Hz. Retargeted finger angles go to a torque-level joint controller at 30 Hz §7.2. Robot-to-camera calibration uses the first frame: the pilot starts with the palm flat and the fingers forward, and the robot's assumed start pose mirrors this §7.2.

Hyperparameters

Results

Author-reported. Setup: 15 tasks tab. II, two pilots, five consecutive trials per pilot per task after 3–5 warm-up attempts, no reset §8. The paper reports completion time (CT) and success rate. The success-rate values appear only in a bar chart fig. 14 and are not stated in the text.

Completion times below are copied from the appendix table app. completion times. The appendix task names are the authors' short names. The mapping to Table II is by name and is our inference. Means exclude failed trials. Failures (F) are counted from the same table.

Totals from this table: 9 failed trials out of 150 (6 for Pilot 1, 3 for Pilot 2). The paper's own success-rate figure fig. 14 is not reproduced numerically.

Other reported numbers: keypoint error 9.7 mm and joint error 1.33 degrees per joint on the authors' 7000-image validation set §6. The authors compare the keypoint error with [Ge et al.] only qualitatively, since the datasets differ §6. Brick Gaiting shows the middle finger making seven contacts over about 40 s to rotate the brick 180 degrees fig. 15. For NIST board peg insertion the authors say success rates are typically within 10% under specific conditions on board placement and distance to the user §10, fig. 16.

Paper vs code

No official code.

The paper has no code link. The abstract and the project page (sites.google.com/view/dex-pilot) link no repository. A GitHub search for "dexpilot" returned ankurhanda/dexpilot, owned by co-author Ankur Handa. That repository holds only a README and the paper PDF, with no source code and no LICENSE file, so there is nothing to cite. The other search hits belong to unrelated owners. The DART tracker and the PointNet++ implementation named in the appendix are third-party software [app. Software Tools, app. PointNet++ architecture], and were not used here.

Pitfalls

• Tracker seeding is essential. The authors say DART without priors fails every few minutes §6. Their rate is not measured, so plan for the same failure mode.
• The labelling pipeline needs a glove phase. The bare-hand networks learn from labels produced by DART run with glove priors §6. Without that phase, there is no training data in the paper's recipe.
• Do not use the fingertip glove predictions. They were predicted but dropped as unstable app. GloveNet. The main text also says only palm blobs are used §6.
• Stage-2 perturbation size is not stated §6. Choose it yourself and expect it to matter for robustness to stage-1 error.
• Pose-head shape is ambiguous. The appendix says the classifier head has 2 outputs tab. V. The weight product uses a 23-channel mask with a hand-probability term app. PointNet++ keypoint head. How the two combine is not stated, and no code is available to check.
• The projection threshold ε is missing. The switching weights s and the target f both depend on it §7.1. The authors say projection can keep small objects from releasing and can be turned off §10, fig. 17.
• Units of the distance term. η1 and η2 are given in metres §7.1. The units of d_i and of β d_i are implied rather than stated.
• The vector count N and its contents are not listed in the text. The vectors are shown in a figure only fig. 7, tab. I.
• The distal-joint constraint is described but not specified. The paper says index, middle and ring distal joints equal their medial joints §7.1. It does not say whether that is an equality constraint or a substitution.
• Solver budget. SLSQP is warm-started and runs each frame. No tolerance or iteration cap is given §7.1. The paper reports no per-frame solve time, so whether it fits the 30 Hz rate is unknown.
• Filter. The low-pass filter's order and cut-off are not stated §7.1.
• Latency. The authors report about one second end to end §4 with no breakdown. Budget for it when you judge teleoperation feel.
• Training details are missing. Optimiser, learning rate, batch size, epochs, augmentation for the point-cloud networks and loss weights are not stated. The only numbers given are data volume and training time §6.
• Uniform subsampling drops finger detail. Stage 1 uses 8192 uniformly sampled points, which under-sample the fingers. That is the stated reason for stage 2 §6.
• Failed trials are excluded from the mean. Brick Gaiting pilot 1 gives mean 58 from 48, 67 and 58 seconds, with two failures app. completion times. Report failures separately.
• Small sample, high variance. Two pilots, five trials each. Standard deviations reach 165.88 s (Blocks S, pilot 1). The authors attribute the variation to fatigue, training, creativity and motivation [§8, app. completion times].
• Keypoint error is on the authors' own data. The 9.7 mm figure is not a benchmark result §6.
• Calibration error is not reported. The robot-to-camera transform comes from the first frame and an assumed robot pose §7.2.
• Touch feedback. The system has BioTac sensors and records tactile signals §3, §9, but the teleoperation loop uses no tactile feedback §1, §10.

Lineage

• Builds on: DART, model-based hand tracking (Schmidt et al., RSS 2014) §5; the MANO hand model §5; voting-based keypoint regression (Ge et al., ECCV 2018) §6; PointNet++ (Qi et al., 2017) §6; anti-aliased ResNet-50 app. GloveNet; Riemannian Motion Policies (Ratliff et al., 2018; Cheng et al., 2018) §7.2.
• Compared against: depth-based retargeting to a Shadow hand (Li et al., ICRA 2019) §2, §7.1; glove and motion-capture teleoperation §2.
• Superseded by: not known from this wiki.