Visual IRL for Human-Like Robotic Manipulation

In plain words

Teaching a robot arm by watching a person is slow to set up if every move has to be programmed by hand. This paper takes a video of a person doing a manipulation task, finds the person's body joints and the objects in the scene frame by frame, and infers what the person seems to be trying to achieve.

The paper then has to solve a second problem. A person's arm and a robot arm have different joint layouts and lengths, so copying the joint angles exactly does not reproduce the same hand motion. The authors use a learned mapping from the person's arm to the robot's joints, followed by a small correction step that brings the gripper to the target point while keeping the joint changes small. The aim is motion that looks like the human's, not just motion that reaches the right spot. The method was tested on two tasks: sorting onions by whether they are blemished, and pouring from bottles into containers.

TL;DR

Visual IRL feeds per-frame 3D body keypoints and object positions into adversarial inverse reinforcement learning (AIRL) to get a reward and a policy. A neuro-symbolic model turns the human arm pose into initial robot joint angles, and an inverse-kinematics (IK) refinement moves the gripper to the target while changing the angles as little as possible. The paper reports better learned-behaviour accuracy than a discrete-input baseline and smoother, faster motion than RRT and RRT-connect, on two tasks with two robots. Everything is author-reported on lab-collected data, and no official code was found.

Problem

Learning from observation needs a state and action representation that an IRL method can use. Standard pipelines first recognise discrete states and actions from video, which needs extra training §I. The paper argues that continuous 3D keypoints can serve as state features directly §I.

Transferring a human demonstration to a cobot is a second problem. The robot has a different number of joints (the paper uses a 7 DoF Sawyer and a 6 DoF LBR iisy) and different limb lengths, so there is no exact one-to-one joint match §I §V.A §VI. Path planners such as RRT find any collision-free joint solution and do not keep the human's style of motion §V.D.

Method

The method has two parts: Visual IRL §IV.A, which learns the reward and policy, and the neuro-symbolic dynamics mapping §IV.B, which moves the learned behaviour onto a cobot. Figure 1 shows the whole pipeline fig. 1.

Visual IRL §IV.A

1. Run a 2D pose estimator (YOLOv8 pose) on every RGB-D frame, and convert the 2D keypoints to 3D with the camera's intrinsics §IV.A. The later mapping uses the right hip, shoulder, elbow and wrist §V.D.
2. Where pose estimation is unreliable, a recurrent keypoint predictor estimates the 3D hip, shoulder, elbow and wrist from the current and previous 3D coordinates of the arm joints §IV.A. Its layers are listed in the hyperparameter table.
3. Detect objects with a custom YOLOv8 detector, convert them to 3D, and predict the object's 3D position from its previous position and the previous and current end-effector positions when the detector loses it §IV.A eq. 5.
4. Choose one object per frame as the object being manipulated, using the rules in §V.A. The state is the end-effector position, the object position and the object label. The action is the 3D displacement of the end-effector between two consecutive steps §IV.A §V.A.
5. Run AIRL on these trajectories. The discriminator distinguishes expert pairs from generated ones eq. 2. The reward is updated from the discriminator output as a log-odds eq. 4. The policy is optimised against this reward with TRPO §IV.A.

# AIRL reward from the discriminator  [eq. 2] [eq. 4]
# Inputs: states (N, 9), actions (N, 3); discriminator outputs in (0, 1)
# Output: rewards (N,)

def airl_reward(discriminator, states, actions):
    # Probability that each (state, action) pair is from the expert  [eq. 2]
    probs = discriminator(states, actions)
    # Reward as log-odds of the discriminator output  [eq. 4]
    rewards = log(probs) - log(1 - probs)
    return rewards

Neuro-symbolic dynamics mapping §IV.B fig. 2

6. Predict the human hip, shoulder and elbow 3D positions from the wrist 3D position at each time step, with a small MLP. Multiple human poses with similar wrist positions are averaged into one representative pose §IV.B.
7. Convert the human joint angles (hip, shoulder, elbow, wrist) to initial cobot joint angles through a fixed one-to-one joint correspondence §IV.B fig. 3. The text does not give this mapping as an equation.
8. Compute the cobot end-effector position from four angles (hip, shoulder, elbow, wrist) with a learned restricted forward kinematics (FK) model. All other cobot joints are held at 0° §IV.B.
9. Starting from the initial angles, adjust the four angles with an iterative IK optimisation that uses the learned FK model. The loss trades position error against the change from the initial angles §IV.B eq. 6.
10. Stop when the position error is below 0.01 m, or after at most 10,000 iterations §IV.B.

# Human arm pose to cobot joint angles  [§IV.B] [fig. 2] [fig. 3]
# Input:  wrist_xyz (3,), target_xyz (3,)  per time step
# Output: final_angles (4,): hip, shoulder, elbow, wrist

def transfer_step(wrist_xyz, target_xyz):
    # Human hip, shoulder and elbow 3D from the wrist  [§IV.B]
    human_joints_xyz = human_ik_model(wrist_xyz)
    # Human joint angles are derived from these 3D points; method not stated
    human_angles = angles_from_joints(human_joints_xyz)
    # One-to-one symbolic map to the cobot's joints  [§IV.B] [fig. 3]
    init_angles = symbolic_map(human_angles)
    # Refine so the end effector reaches the target  [§IV.B] [eq. 6]
    final_angles = refine_ik(init_angles, target_xyz)
    return final_angles

Action and target §V.B. For the RRT baselines, the target is computed from the current end-effector position plus the policy's displacement at each step. The paper does not state the same explicitly for the neuro-symbolic model, although the target is an input to its IK step §IV.B.

Hyperparameters

Values are from the paper's text and the stated components. Anything not stated is marked.

Results

Author-reported. Setup: two tasks with real cobots, a Sawyer (7 DoF) for onion sorting and a KUKA LBR iisy (6 DoF) for bottle pouring §V.A. Each task had 10 trials with 4 objects per trial, which gives 40 manipulations per model §V.D. The LBA comparison used 10 trials of 30 ± 5 state-action steps, split between two subjects §V.D.

Pose estimator, tab. 1. Values are mean ± standard deviation. The caption does not state the unit.

Human keypoint predictor, tab. 2. Test set of 500 frames.

Object detector, tab. 3. The caption names four criteria (Precision, Recall, mAP50, mAP50-95). The column headers in the source say "Right Hip", "Right Shoulder", "Right Elbow" and "Right Wrist". The mapping of columns to criteria is not stated (see Pitfalls). Values are mean ± standard deviation.

Object location predictor, tab. 4. Onion sorting task.

Onion sorting, one object, tab. 5. Baseline is RRT [lavalle2001randomized].

Bottle pouring, tab. 6. Baseline is RRT-connect [kuffner2000rrt].

Learned-behaviour accuracy (LBA) for Visual IRL, in the text §V.D: 90.4%, against 83.3% for MAP-BIRL (Choi and Kim, 2011) fed with discrete state-action trajectories. The paper gives no table for this comparison.

Paper vs code

No official code.

No repository is linked from the paper, the LaTeX source, or the bibliography. GitHub searches for "visual irl", "neuro-symbolic cobot human-like manipulation", "neuro-symbolic dynamics mapping" and the author names found no repository that clearly belongs to the authors. Searches for "cobot irl keypoints" and "onion sorting cobot" also returned nothing relevant. Only the paper's text and figures are used here.

Pitfalls

• The cobot IK loss is inconsistent with its text. The text says the model is trained with MSE and Adam §IV.B, but the equation for its loss uses plain norms, not squared errors, and the norm type is not stated eq. 6. Reading the norms as L2 is an assumption.
• Keypoint predictor output size does not match its inputs. The text lists six input joints (hip, shoulder, elbow, wrist, index finger, thumb) §IV.A, but says the output size of 12 "includes the predicted 3D coordinates of the six input joints". Six joints × 3 coordinates is 18. Twelve is 4 × 3. Clarify this before implementing.
• Duplicate rows in tab. 2. The values for Elbow X equal those for Shoulder X. Wrist Y equals Elbow Y, and Wrist Z equals Shoulder Z. These coincidences are unlikely to be independent measurements. Check the source data before trusting per-joint numbers.
• Column headers in tab. 3 do not match its caption. The headers are joint names, but the caption describes four detection criteria. The text names Precision, Recall, mAP50 and mAP50-95. The mapping is not stated.
• Which state features the policy uses is ambiguous. The Introduction describes keypoints as state features §I. The policy is described as mapping states that contain human keypoints to end-effector actions §IV.A. The MDP state is defined as end-effector position, object position and object label §V.A. Both could be right, but the paper does not say which is used for the AIRL reward.
• The reward update is written as an assignment. Equation 4 uses "←". The paper does not say whether the reward is stored and updated or recomputed from the discriminator on each call eq. 4.
• AIRL training details are missing. Discriminator architecture, its loss, its learning rate, the TRPO settings, the discount factor, the number of iterations and the batch size are all not stated §IV.A.
• The symbolic mapping is only shown in a figure. The text describes a "one-to-one" joint correspondence §IV.B and depicts it in fig. 3. No equations are given, and the way human joint angles are computed from the predicted 3D joints is not stated.
• Object selection rule is ambiguous. The text lists three rules §V.A: a confidence threshold, a 20 cm proximity rule, and a fallback to the lowest-Y object. It is not stated whether the proximity check is applied to each candidate in order, or only once, and whether the 20 cm check uses the object centroid or the bounding box.
• Camera-to-cobot transform is not given. The text describes the axis permutation and offset in words §V.A. No numbers appear in the paper body, and the reproduction needs the extrinsic calibration, which is not stated.
• Pouring is a simplification. The pipeline does not model liquid flow or volume. Pouring is triggered by a simple tilt once the bottle is aligned §V.A. The "pouring" metric measures motion, not pouring quality.
• Different baselines per task. RRT is the baseline for onion sorting and RRT-connect for pouring §V.B. The per-task tables therefore cannot be compared across tasks, and neither baseline is the same across both.
• "Power efficiency" is not measured. The conclusion claims advantages in "power efficiency" §V.D. The only efficiency metric reported is end-effector displacement tab. 5 tab. 6.
• Variability is reported without sample counts. The caption of tab. 1 says the ± values are standard deviations. The captions of the other tables do not define them, and the sample they are computed over (trials, objects or frames) is not stated.
• Small, single-lab data. Training and evaluation use 3000 and 500 frames from a single lab setup §V.D. The dataset and the simulation data for the FK model are not released. Sample counts for the FK model are not stated.
• Supplementary video is referenced but not in the card. Several claims point to it §V.D.
• Figure numbering is inferred. Figure and table numbers are derived from the order of the LaTeX environments, following ACM format. The PDF was not checked against this numbering. §IV.A style section numbers are taken from the source headings and match the paper's own cross-reference "section IV.A" §V.D.

Lineage

• Builds on: AIRL (Fu, Luo and Levine, ICLR 2018; no arXiv id given in the paper) §III. The AIRL formulation rests on maximum causal entropy IRL (Ziebart, 2010) §III.
• Compared against: RRT (LaValle and Kuffner, 2001) and RRT-connect (Kuffner and LaValle, 2000) for robot motion §V.B; MAP-BIRL (Choi and Kim, 2011) for LBA §V.D. Object detection and pose estimation are compared against Faster-RCNN and Mediapipe §V.B.
• Related work named in the paper: learning-from-observation models (Asali et al., 2023; Soans et al., 2020), retargeting approaches (Wang et al., 2017; Darvish et al., 2019; Qin et al., 2023) and Simitate (Memmesheimer et al., 2019). These are context and are not used in the method §II.
• Superseded by: not known from the content of this wiki.