wikillm

Paper · 2021-05-13 · cs.RO, cs.LG

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Learns a manipulation task from one human demonstration. A CNN, trained self-supervised from images taken while the camera moves around the object, estimates where the demonstrated interaction begins. The robot moves there along a linear path and replays the demonstrated end-effector velocities. Across 8 real tasks the best variant averages 70.0% success (author-reported).

Imitation learningLearning from demonstrationUncertainty estimationPose (rigid position and orientation of an object)robot-learningimitation-learningstate-estimationself-supervised-learning

arXivPDFMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Teaching a robot a task usually takes many demonstrations, a lot of trial and error, or a model of the object built in advance. This paper asks whether one demonstration can be enough. Its answer is to split the task in two. The first part is the approach: the robot's hand travels through open space to the spot where the real work starts. Nothing touches anything there, so the robot can practise it alone, moving a camera around the object and learning from the pictures where that spot is. The second part is the interaction itself, such as putting a lid on a bottle or swinging a hammer. Here the robot simply repeats the motion the person showed it.

So the robot learns one thing from images, namely where that starting spot is, and it does not learn a full control policy. When its hand reaches the spot, it plays back the recorded motion. The authors report that this works on eight everyday tasks with a real robot, and that the controller is simple enough to analyse.

TL;DR

Coarse-to-Fine Imitation Learning (Johns, 2021) learns a manipulation task from one human demonstration. A CNN estimates the bottleneck pose, the end-effector pose where interaction begins, and is trained self-supervised from images taken while the camera moves around the object. The robot moves to that pose along a linear path, then replays the demonstrated end-effector velocities open loop. With filtering over predictions and a last-inch correction, the average success over 8 real tasks is 70.0% (author-reported, not independently checked).

Problem

Learning from demonstrations alone needs many demonstrations, and methods that add self-exploration need environment resets or task-specific apparatus §I. Transfer methods need prior training on a family of similar tasks, and methods that model a task in a hand-defined state space need prior knowledge of the object or task §I. The paper targets the case of a novel everyday task learned from one human demonstration, with no prior knowledge of the object being manipulated §I.

The approach also has a stated cost. Once the interaction starts, the replayed motion is open loop, so any error in reaching the bottleneck becomes a growing offset from the demonstration during the interaction §III-B. The method therefore relies on the bottleneck estimate being accurate, and on the interaction being short.

Method

The paper splits each task into two phases: a coarse approach, then a fine interaction §III-B. The approach is executed as a linear path using inverse kinematics and needs no demonstration §III-B. The interaction is replayed from one demonstration §III-B. Imitation becomes a state estimation problem: predict the bottleneck pose from an image §I §III-B.

Frames. There are three frames: the robot base R, the end-effector E, and the object's bottleneck B. B is a virtual frame fixed relative to the object. T_EB is B expressed in E §III-A.

Bottleneck definition. During training, the bottleneck is set to the demonstration's initial end-effector pose, so that all approaches converge there §III-B. The demonstration should begin as near the object as possible. It must not start lower than the top of the object, so the robot can avoid hitting it while data is gathered §III-B.

Axiom 1 (replay). A controller whose actions are given in the end-effector frame depends, in principle, only on the end-effector's pose relative to the object. So the local velocities recorded from the bottleneck can be replayed once the robot is there §III-B.

Axiom 2 (vision). The camera image depends only on the camera's pose relative to the object. The camera is rigid on the end-effector, so the image depends on T_EB. Images paired with their T_EB can therefore train a regressor from image to T_EB §III-B.

Self-supervised data. After the demonstration, the training pairs are D = {(I, T_EB)}, with T_EB computed from forward kinematics and the recorded bottleneck eq. 1 §III-C. Each approach starts from a random pose above the object and moves in a straight line at a constant speed toward a pose sampled near the bottleneck, recording pairs on the way §III-C. The regressor is a CNN trained with supervised learning on the pose directly. No camera calibration is needed as long as the camera stays rigid on the end-effector §III-C.

T_{EB} = T_{ER}\,T_{RB} eq. 1

Reduced prediction space. The object is assumed to be on a tabletop, where it moves by horizontal translation and rotation about the vertical axis. The end-effector is therefore constrained to point straight down, and to rotate only about the vertical axis, both during data collection and during the approach at test time. The prediction space then has three dimensions: two horizontal translations and one rotation about the vertical axis §III-C. The bottleneck is the demonstration's initial position with a vertical orientation. The rotation R between that bottleneck and the initial pose is recorded and applied at the end of the approach, so the interaction can still be 6-DOF §III-C.

Last-inch network. A second dataset E is collected with the height fixed at the bottleneck height h, so each trajectory moves only horizontally and rotates about the vertical axis. A second network g is trained on E for a final, close-range correction §III-C alg. 1.

Pseudocode
# Training from one demonstration  [alg. 1]
# Input: one demonstration; camera rigid on the end-effector
# Output: f(I), g(I), replay velocities U, rotation R, height h
def train_from_demonstration(demonstration):
    # Record initial pose and end-effector velocities U  [§III-B] [alg. 1]
    initial_pose, velocities_U = record(demonstration)
    # Bottleneck: initial position with a vertical orientation  [§III-C]
    bottleneck = with_vertical_orientation(initial_pose)
    # Rotation from bottleneck to initial pose, applied at test  [§III-C]
    rotation_R = rotation_between(bottleneck, initial_pose)
    # Height of the bottleneck; the switch from f to g uses it  [§III-C]
    height_h = height_of(bottleneck)
    # Coarse data D for f, from approach trajectories  [§III-C] [alg. 1]
    dataset_D = collect_approach_pairs(bottleneck, height_h)
    # Fine data E for g, with the height fixed at h  [§III-C] [alg. 1]
    dataset_E = collect_fixed_height_pairs(bottleneck, height_h)
    # f regresses T_EB directly from the image  [§III-C]
    f = train_cnn(dataset_D)
    # g makes the final close-range estimate  [§III-C] [alg. 1]
    g = train_cnn(dataset_E)
    return f, g, velocities_U, rotation_R, height_h
Pseudocode
# One approach trajectory per loop pass; pairs are recorded while moving
# [§III-C] [alg. 1]
def collect_approach_pairs(bottleneck, height_h):
    pairs = []
    # Number of trajectories: 50 in the experiments  [§IV-C]
    for trajectory in range(n_trajectories):
        # Start from a random pose above the object  [§III-C] [alg. 1]
        robot.move_to(random_start_pose())
        # Target sampled near the bottleneck; ranges 5 cm, 5 deg  [§IV-A]
        target = sample_near(bottleneck, 0.05, 5.0)
        while robot.height() > height_h:
            image = camera.capture()
            # Forward kinematics gives T_ER  [eq. 1]
            pose_ER = forward_kinematics()
            # T_EB = T_ER T_RB  [eq. 1]
            pose_EB = compose(pose_ER, bottleneck)
            pairs.append((image, pose_EB))
            # Constant-speed straight-line motion  [§III-C] [alg. 1]
            robot.move_linear_towards(target)
    return pairs

Sequential state estimation. At test time the object is stationary, so every prediction estimates the same bottleneck pose. The method therefore fuses predictions over time, treating each prediction as a Gaussian with a standard deviation. Two fusion rules are given §III-D:

  • Batch: at each step, re-estimate from all predictions so far, weighting each by the inverse of its variance eq. 2 §III-D.
  • Filtering: at each step, combine the previous estimate and its variance with the new prediction and its variance. The initial estimate is the first prediction, and the initial variance is the Prior uncertainty eq. 3 eq. 4 §III-D. The paper describes this as a Bayes filter derived from a Kalman filter with zero process noise §III-D.

\bar{x}_t = \frac{\sum_{\tau=0}^{t} \hat{x}_{\tau}/\hat{\sigma}_{\tau}^2}{\sum_{\tau=0}^{t} 1/\hat{\sigma}_{\tau}^2} eq. 2

\bar{x}_t = \frac{\bar{x}_{t-1}/\bar{\sigma}_{t-1}^2 + \hat{x}_t/\hat{\sigma}_t^2}{1/\bar{\sigma}_{t-1}^2 + 1/\hat{\sigma}_t^2} eq. 3

\bar{\sigma}_t^2 = \frac{1}{1/\bar{\sigma}_{t-1}^2 + 1/\hat{\sigma}_t^2} eq. 4

Pseudocode
# Batch estimate over all predictions so far  [§III-D] [eq. 2]
# predictions: per-step pose estimates; sigmas: their standard deviations
def batch_estimate(predictions, sigmas):
    # Inverse-variance weight for each prediction  [eq. 2]
    weights = [1 / sigma ** 2 for sigma in sigmas]
    weighted_sum = sum(w * x for w, x in zip(weights, predictions))
    return weighted_sum / sum(weights)
Pseudocode
# One filter step: fuse the previous estimate with a new prediction
# [§III-D] [eq. 3] [eq. 4]
def filter_update(estimate, variance, prediction, prediction_var):
    # Inverse-variance weighted mean  [eq. 3]
    new_estimate = (
        (estimate / variance + prediction / prediction_var)
        / (1 / variance + 1 / prediction_var)
    )
    # Variance of the fused estimate  [eq. 4]
    new_variance = 1 / (1 / variance + 1 / prediction_var)
    return new_estimate, new_variance

Uncertainty sources. Each prediction needs a standard deviation. Three options are compared §III-D:

  • Dropout: keep dropout active at inference, and use the standard deviation of repeated predictions §III-D.
  • Predicted: train a small network to predict the prediction error from the predicted pose, using errors measured on the validation split §III-D.
  • Prior: use the validation error of f as one constant for all predictions §III-D.

Execution. At test time the robot runs the approach loop until the end-effector is at height h. At each step it captures an image, computes the bottleneck pose from f, updates the fused estimate, and moves linearly toward it §III-E alg. 2. The bottleneck pose is T_RB = T_RE T_EB eq. 5 §III-E. At height h, the robot takes one final estimate from g, moves to that pose, rotates by R, and replays the recorded velocities U §III-E alg. 2.

T_{RB} = T_{RE}\,T_{EB} eq. 5

Pseudocode
# Testing  [alg. 2]
# Inputs from training: f, g, replay velocities U, rotation R, height h
def run_task(f, g, velocities_U, rotation_R, height_h):
    estimate = None
    # Approach while above height h; refine the estimate each step  [alg. 2]
    while robot.height() > height_h:
        image = camera.capture()
        # f gives T_EB; eq. 5 maps it to the base frame T_RB  [§III-E] [eq. 5]
        pose_RB = compose(forward_kinematics(), f(image))
        # Fuse with earlier predictions: batch or filter  [§III-D] [alg. 2]
        estimate = fuse(estimate, pose_RB)
        # Straight-line motion toward the current estimate  [§III-E] [alg. 2]
        robot.move_linear_towards(estimate)
    # Final estimate from g at height h  [§III-E] [alg. 2]
    pose_RB = compose(forward_kinematics(), g(camera.capture()))
    robot.move_to(pose_RB)
    # Re-orient by R, then replay the demonstration's velocities  [alg. 2]
    robot.rotate_by(rotation_R)
    robot.replay(velocities_U)

Hyperparameters

Values as stated in the paper, with the section that gives them. "Not stated" means the paper does not give the value. The data-collection loop that these counts feed into is the one in alg. 1.

NameValueWhereNotes
Robot7-DOF Sawyer§IV-A
Control rate30 Hz§IV-Athe robot operates at 30 Hz
Image input64 × 64 RGB, camera on the end-effector§IV-Ahigher resolution judged unnecessary §IV-A
Network (f and g)4 conv layers with ReLU and max-pool; then 4 fully connected layers with ReLU and Dropout§IV-Afilter counts, kernel sizes, widths and Dropout rate not stated
LossMean squared error; translation/rotation balance§IV-Abalance from [kendall2015posenet], computed once on a small dataset, then fixed; value not stated
Validation split20% of data for early stopping§IV-A
Optimiser, learning rate, batch size, epochsnot stated—
Task space40 × 40 cm region on the table, below the robot§IV-A
Training start height50 cm above the table§IV-A
Training start orientationsampled uniformly over a 90° range§IV-A
Test object placementrandom in the task space, rotated over a 90° range relative to training§IV-A
Test start height50 cm above the centre of the task space§IV-A
Target sampling for franges of 5 cm and 5° relative to the bottleneck§IV-Awhether the ranges are symmetric is not stated
Target sampling for granges of 2 cm and 20° relative to the bottleneck§IV-Asame caveat
Trajectories per dataset (imitation)50 for D and 50 for E§IV-C
Target-reaching setup7 objects; 50 trajectories per object for D; 5 poses per object§IV-B
Imitation test setup8 tasks; 20 object poses per task§IV-C
Bottleneck orientationend-effector vertical, rotation only about the vertical axis§III-C
Prior uncertaintyvalidation error of f, used as a constant§III-Dthe value is not stated
Dropout at inferencenumber of passes and rate not stated§III-D
Predicted-uncertainty networkarchitecture and inputs not stated§III-Ddescribed only as a small network
Approach speedconstant; value not stated§III-C

Results

Author-reported. Real-world setup: a 7-DOF Sawyer robot and a camera on the end-effector, with 8 everyday tasks and 20 object poses per task. Success is judged by eye §IV-C. The baseline is visual servoing, which estimates the bottleneck from the current image only §IV-C. A residual reinforcement learning baseline, built on a demonstration-based policy, did not solve any task even after tuning, and no numbers are reported for it §IV-C.

Target reaching (estimating the bottleneck pose with the approach trajectory; 7 objects, 5 poses each; mm for position, degrees for orientation) tab. I:

Estimation methodMean PosMean OriMin PosMin OriMax PosMax Ori
Oracle0.30.00.10.00.70.1
First Image13.99.03.71.325.717.7
Best Image (Dropout)4.96.42.32.58.210.7
Best Image (Predicted)13.612.05.54.624.924.2
Visual Servoing5.76.63.43.18.911.3
Batch (Prior)5.25.81.92.09.810.4
Batch (Dropout)7.55.92.91.212.712.1
Batch (Predicted)5.86.42.32.210.211.3
Filtering (Prior)5.25.71.91.89.510.6
Filtering (Dropout)7.66.23.11.712.512.7
Filtering (Predicted)5.86.92.52.311.012.8

The paper's table caption states that errors are averaged over the 7 objects, each with 5 poses, and that the three best values per column are emphasised, excluding the Oracle tab. I. This card does not reproduce the emphasis.

The paper's reading is that the best sequential methods beat plain visual servoing; that Prior uncertainty beats the two dynamic uncertainty estimates; and that filtering does at least as well as batch §IV-B. The oracle row measures the controller's own error. Its ground truth was set by eye §IV-B.

Imitation (success rate in % over 20 object poses per task) tab. II:

MethodBottlePlateScrewdriverLidHammerScoopKnifePlugAverage
Visual Servoing65252065355515535.6
Visual Servoing + Correction10095701004095101065.0
Filtering85251085509001044.4
Filtering + Correction100806010065100104570.0

The average is the mean over the 8 tasks tab. II. The paper states that the full method (filtering with last-inch correction) succeeded on all 20 poses for three tasks §IV-C, which matches the three 100 entries for that row tab. II. The Plug task was solved in 9 of 20 poses §IV-C. The paper also argues that the last-inch correction significantly raised success rates. For the Knife task, which had the lowest success with the full method, the paper attributes the difficulty to a very thin slot and a textureless rack, which make orientation prediction hard §IV-C.

Paper vs code

No official code.

Search, in order: the arXiv abstract page lists no code URL, and its comments field reads "Published at ICRA 2021. Webpage and video". The project page at robot-learning.uk shows a "[Code]" link next to BibTeX, but its target could not be read, and no GitHub URL is visible on the page. A GitHub repository search for the title returns several repositories, none owned by an author. The one whose name matches the title exactly is rylynnrex/coarse-to-fine-imitation- learning, a third-party repository, so it is not used. The author's GitHub account has no public repositories. A guessed name for the lab's GitHub organisation returns 404.

The unresolved "[Code]" link is the one open item. Check it by hand before treating the card as final.

Pitfalls

  • Scalar or vector uncertainty. The estimate x̂ is a 3-vector (horizontal position and vertical orientation), but the paper gives one standard deviation per prediction §III-D. The paper does not say whether this is a scalar for all three components or one value per component. Table I reports position in mm and orientation in degrees tab. I, so a single scalar variance treats the two quantities as if they were in the same units. Choose and document one option.
  • Frame of the fused estimate. The paper says every prediction estimates the same value because the object is stationary §III-D. That holds for T_RB, not for T_EB, which changes as the robot moves. The text fuses x̂ = f(I) directly and then gives T_RB = T_RE T_EB eq. 5 §III-E. It does not state whether f's output is converted to the base frame before fusion. The pseudocode above converts it, which is the only reading that keeps the estimate fixed.
  • First step of the filter. The paper sets the initial estimate to the first prediction and the initial variance to the Prior value §III-D. It does not say whether the update in eq. 3 and eq. 4 also runs for the first prediction, which would fuse it with itself and with the prior.
  • Prior uncertainty value. The Prior is the validation error of f §III-D, but the value, and how position and orientation errors are combined into one number, are not stated.
  • Dropout and predicted-uncertainty details. The number of stochastic passes and the dropout rate are not stated §III-D. The predicted-uncertainty network is described only as a small network trained on validation errors §III-D; its inputs, targets and size are not given.
  • Last-inch estimate is not fused. g's estimate is used directly for the final move §III-E alg. 2. The paper does not say whether it is combined with the earlier fused estimate.
  • Loss balance is not given. The translation/rotation weights were computed once with the method of [kendall2015posenet] on a small dataset and fixed §IV-A. The values are not in the paper. An implementation has to recompute them or choose its own.
  • Training details are missing. The optimiser, learning rate, batch size, number of epochs, layer widths, kernel sizes and dropout rate are not stated §IV-A. The paper mentions image augmentation only as a possible way to reduce the approximation in its second axiom §III-B, and does not report using it.
  • Demonstration and object constraints. Equation 1 assumes the object does not move between the demonstration and data collection. If it does, the paper says to reposition it or to collect the data before the demonstration §III-C. The demonstration must also not start below the top of the object §III-B.
  • Open-loop replay. Errors at the bottleneck grow during the interaction and are not corrected §III-B. Results on long or precise interactions are the most sensitive to this.
  • Vertical-orientation assumption. The method assumes a tabletop, a vertical end-effector and rotation only about the vertical axis §III-C. The paper does not state how the 90° rotation range wraps around, or the sign convention for the rotation.
  • Sampling speed and rate. The approach speed is "constant" but its value is not stated §III-C. Algorithm 1 records one pair per loop pass alg. 1, but the loop rate for data collection is not given.
  • Sampling ranges. The target ranges (5 cm and 5° for f, 2 cm and 20° for g) are stated as ranges, not as ± values §IV-A. Whether they are symmetric around the bottleneck is not stated.
  • Evaluation is by eye. Success is judged by eye §IV-C, and the paper does not say how many attempts were made per pose or how many runs there were. The reported counts are the only evidence.
  • Oracle ground truth is approximate. The reference bottleneck for target reaching was set by eye, with the end-effector placed by hand §IV-B.
  • Residual RL baseline has no numbers. The residual reinforcement learning baseline solved no tasks, but the paper gives no table or per-task result for it §IV-C.
  • Narrow setup. The experiments use one robot, 64 × 64 images, and a tabletop with a 40 × 40 cm task space §IV-A. The paper does not test other robots, other cameras or other workspace sizes. For the Plug task, it says the success rate would likely have been lower had the socket not been free to slide along the table during the interaction §IV-C.

Lineage

  • Builds on: Bayesian dropout as an approximation to uncertainty §III-D (Gal et al., 2016, bibliography key gal2016dropout), inverse-variance weighting §III-D, a Bayes filter from the probabilistic robotics literature §III-D, and PoseNet-style loss weighting §IV-A (Kendall et al., 2015). None of these has an arXiv id in the paper, so none is linked.
  • Compared against: visual servoing, which uses the current image only §IV-B §IV-C; residual reinforcement learning following Schoettler et al. (2020) §IV-C.
  • Not compared: behavioural cloning and meta-learning, which the paper excludes because they need more than one demonstration or prior training on similar tasks §IV-C.
  • Superseded by: not stated in this card.

Source and license

Adapted from “Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration” by Edward Johns (arXiv:2105.06411), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.