LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

In plain words

A robot that learns one task after another tends to forget the earlier ones. People want robots that keep learning new jobs over their working life without losing the old skills, but there has been no shared, simulated test bed for this kind of robot manipulation. This paper builds one. It contains 130 tasks in four groups, such as putting a bowl on a plate or opening a drawer. Every task has a set of demonstrations recorded by a person steering the robot.

The authors then run a set of standard learning methods on the tasks, using three different robot "brains" (neural network designs). They measure how often the robot succeeds, how fast it picks up a new task, and how much it forgets. Two of the results stand out. The simplest method, which just keeps training on the new task, learns new tasks best. And starting from a model that was first trained on other tasks can make the later learning worse.

TL;DR

• LIBERO is a simulated benchmark for lifelong imitation learning in robot manipulation, with 130 language-conditioned tasks in four suites §1 §4.2.
• Each task has 50 teleoperated demonstrations. Policies are trained with behaviour cloning and scored on success rate §4.4 §5.1.
• Forward transfer, negative backward transfer and AUC are defined on success rates and computed over a 0-50 epoch learning curve §5.1 eq. 3.
• Author-reported: sequential finetuning has the best forward transfer, and basic supervised pretraining can hurt §5.2 tab. 2 fig. 5.

Problem

Lifelong learning is studied mostly for declarative knowledge, such as classifying images or understanding text. Robot decision-making also needs procedural knowledge, such as how to open a drawer, and no suitable benchmark existed for measuring how each kind transfers §1 §3.

The paper frames the setting as a sequence of K tasks learned by one task-conditioned policy. Earlier tasks' demonstrations are not fully available when task k is learned. The surrogate objective is behaviour cloning on demonstrations rather than reinforcement learning §2.2 eq. 1 eq. 2. The authors state that sparse-reward reinforcement learning is hard enough to motivate demonstrations §2.2.

The paper identifies five research topics: transfer of different knowledge types, neural architecture, lifelong learning algorithms, robustness to task order, and use of pretrained models §3.

Method

Overview

LIBERO is built on the RoboSuite simulator §4.1. Its tasks are generated from a pipeline: language templates are extracted from Ego4D annotations, a scene is chosen, and the initial object distribution and goal predicates are written in PDDL §4.1 fig. 2. The pipeline can in principle produce unlimited tasks. The released benchmark fixes 130 of them in four suites §4.2.

Steps

1. Task templates. Ego4D language annotations are summarised into commonly used templates, which generate instructions over objects available in the simulator §4.1.
2. Scene and initial state. A scene layout matching the instruction is sampled. Object categories, placements and initial states are written as PDDL, giving the initial state distribution mu_0 §4.1 fig. 2.
3. Goal. The goal is a conjunction of unary predicates (for example Open(X)) and binary predicates (for example On(A, B) or In(A, B)). The episode ends when all predicates hold §4.1.
4. Suites. LIBERO-Spatial, LIBERO-Object and LIBERO-Goal have 10 tasks each. They vary spatial layout, object type and task goal respectively. LIBERO-100 has 100 tasks, split into 90 short-horizon tasks (LIBERO-90, used for pretraining) and 10 long-horizon tasks (LIBERO-Long, used for downstream evaluation) §4.2.
5. Demonstrations. Each task has 50 trajectories collected by human experts with a 3Dconnexion SpaceMouse §4.4.
6. Lifelong sequence. Tasks arrive one after another. The policy is conditioned on the task. Each task is trained with behaviour cloning, a Gaussian-mixture negative log-likelihood §2.2 eq. 2 §4.4.
7. Policy networks. Three designs, each taking an image, the robot state and a BERT language embedding §4.4.
   • BC-RNN: ResNet encoder with FiLM language conditioning, then an LSTM §4.4.
   • BC-T: ResNet encoder, then a transformer decoder over visual tokens with the language embedding as an extra token §4.4.
   • BC-ViT: ViT encoder, then a transformer decoder; the language embedding is a token in both §4.4. All three output a Gaussian-mixture action distribution §4.4. The paper says the networks are matched at about 13.5 GFLOPS app. D.
8. Algorithms. Experience replay (ER), elastic weight consolidation (EWC) and PackNet, plus sequential finetuning (SeqL) as a lower bound and multitask learning (MTL) as an upper bound §4.3 app. A.2.
9. Metrics. Each task's success is measured at epochs 0, 5, ..., 50. The best epoch sets the point at which the agent is treated as having stopped learning. Forward transfer, negative backward transfer and AUC are then computed from the success matrix §5.1 eq. 3.

# Lifelong imitation learning: one policy, tasks in a fixed order  [§2.2]
# Input:  tasks 1..K, each with 50 demonstrations; a policy architecture
# Output: policy after task K; success matrix c[i][j]

def run_lifelong(policy, tasks):
    # 1. Learn the tasks one after another; earlier demos are not revisited
    for task_index, task in enumerate(tasks):
        curve = learn_one_task(policy, task)
        # 2. Evaluate on every task seen so far  [code: libero/lifelong/main.py#L214-L236]
        c[task_index] = evaluate_on_all_seen_tasks(policy, tasks)
    return c

# Behaviour cloning on one task, with checkpoint selection  [§2.2] [app. D]
# Epoch 0 is a zero-shot evaluation; epochs 1..50 train on the demos
# Output: 11 success values (epochs 0, 5, ..., 50) and the best checkpoint

def learn_one_task(policy, task):
    # 1. Fresh AdamW (lr 1e-4) and cosine schedule down to 1e-5  [app. D]
    optimizer = adamw(policy.parameters(), lr=1e-4)
    scheduler = cosine(optimizer, t_max=50, eta_min=1e-5)
    best_success = -1.0
    curve = []
    for epoch in range(0, 51):
        if epoch > 0:
            # 2. Minimise the BC loss over batches of 32  [eq. 2] [app. D]
            for batch in minibatches(task.demos, size=32):
                optimizer.zero_grad()
                loss = policy.nll(batch)
                loss.backward()
                clip_grad_norm(policy.parameters(), max_norm=100)
                optimizer.step()
            scheduler.step()
        # 3. Every 5 epochs, roll out 20 episodes and keep the best  [§5.1]
        if epoch % 5 == 0:
            success = rollout_success(policy, task, episodes=20)
            curve.append(success)
            if success > best_success:
                best_success = success
                save_checkpoint(policy)
    # 4. Restore the best checkpoint; it is the one reported  [app. D]
    load_checkpoint(policy)
    return curve

# Experience replay: a batch of new windows plus replayed old ones  [app. A.2]
# Input:  new-task batch of 32 windows; buffer of earlier-task windows
# Output: loss for the joint batch

def er_observe(new_batch, buffer):
    # 1. Draw 32 windows uniformly at random from the buffer  [app. A.2]
    replay_batch = sample_uniform(buffer, size=32)
    # 2. Concatenate new and replayed windows (64 in total)
    batch = concat(new_batch, replay_batch)
    # 3. Standard BC loss and update on the joint batch  [eq. 2]
    return bc_loss(batch)

# EWC penalty and Fisher update, as the code implements them  [app. A.2]
# Differs from the paper's formulas: see Paper vs code
# Input:  Fisher estimate fish, parameters saved after the last task

def ewc_penalty(params, saved_params, fish, e_lambda=5e4):
    # 1. Fisher-weighted squared distance to the saved parameters
    return e_lambda * sum(fish * (params - saved_params) ** 2)

def ewc_update_fisher(fish_old, fish_new, gamma=0.9):
    # 2. Decay the old estimate and add the new one (code: no (1 - gamma))
    return gamma * fish_old + fish_new

# PackNet pruning of one weight layer after a task  [app. A.2]
# Input:  weights of one Conv2d or Linear layer; mask gives each weight's task
# Output: updated mask; pruned weights are released for later tasks

def prune_layer(weights, mask, task_id, keep=0.25):
    # 1. Take the weights that belong to the current task
    owned = weights[mask == task_id + 1]
    # 2. Keep the largest 25% by magnitude  [app. A.2]
    #    Code cuts at rank round(0.75 * count)
    #    [code: libero/lifelong/algos/packnet.py#L42]
    cutoff = kth_smallest(abs(owned), round((1 - keep) * owned.size))
    # 3. Release the rest; the kept weights are frozen after fine-tuning
    mask[(abs(weights) <= cutoff) & (mask == task_id + 1)] = 0
    return mask

# Lifelong metrics from the success matrix  [§5.1] [eq. 3]
# c[i][j]: success on task j after learning task i (best epoch of task i)
# curve[k]: the 11 learning-curve successes of task k (epochs 0 to 50)
# Tasks are 1-based, as in the paper; the last task has no NBT

def lifelong_metrics(c, curve, K):
    # 1. Forward transfer per task: mean of its 11 learning-curve points
    fwt = [mean(curve[k]) for k in range(1, K + 1)]
    # 2. Negative backward transfer: peak minus later success, per task
    nbt = [mean(c[k][k] - c[t][k] for t in range(k + 1, K + 1))
           for k in range(1, K)]
    # 3. AUC per task: learning curve plus later success, averaged
    auc = [(fwt[k - 1] + sum(c[t][k] for t in range(k + 1, K + 1)))
           / (K - k + 1)
           for k in range(1, K + 1)]
    # 4. Average each per-task metric over the tasks
    return mean(fwt), mean(nbt), mean(auc)

Hyperparameters

Training and evaluation settings are the same for every algorithm and architecture, except where a row says otherwise. The paper's appendix lists the architecture tables app. A.1. The code is the authors' release at the commit in the frontmatter.

Results

Author-reported, mean and standard error over three seeds tab. 1 tab. 2. Each run is scored on 20 rollouts per task app. D.

Architecture comparison with ER and PackNet tab. 1. FWT is higher-is-better, NBT is lower-is-better, AUC is higher-is-better.

Author findings: BC-T and BC-ViT beat BC-RNN on average. The best architecture depends on the algorithm and the suite §5.2.

Algorithm comparison with BC-T tab. 2.

Author findings: SeqL has the best FWT on all four suites §5.2. From tab. 2, PackNet has the best AUC on LIBERO-Object and LIBERO-Goal and the lowest NBT on both. The authors also report that ER is robust across suites §5.2. The MTL row shows AUC only; FWT and NBT are blank in the paper.

Pretrained language embeddings on LIBERO-Long, BC-T with ER tab. 3. No difference is statistically significant.

Task order and pretraining are shown only as figures. Figure 4 plots ER and PackNet with BC-T on five task orders; the authors report that order changes results, significantly for PackNet §5.2 fig. 4. Figure 5 compares five algorithm and architecture combinations with and without pretraining on LIBERO-Long; the authors report that basic supervised pretraining can hurt downstream lifelong learning §5.2 fig. 5. The text gives no numbers for either figure.

Full per-suite results across all algorithms and architectures are in app. E.1. Per-task success for the multitask baseline is in app. E.3.

Paper vs code

Code: repository at commit 8f1084e README.md. The README names the paper and its authors. The abstract points to the project website for code [abstract]. The LICENSE file is MIT LICENSE. Paths below are relative to the repository root.

• Optimizer. Paper says Adam app. D. Code builds torch.optim.AdamW with weight decay 1e-4 adam_w.yaml and base.py.
• EWC penalty. Paper writes the penalty as (lambda/2) F (theta - theta*)^2 app. A.2. Code omits the 1/2: the penalty is e_lambda * fish * (params - checkpoint)**2 ewc.py and ewc.py. For the same lambda = 5 x 10^4, the code's coefficient is twice the paper's stated one.
• EWC Fisher update. Paper says F~k = gamma F{k-1} + (1 - gamma) F_k app. A.2. Code does fish *= gamma then fish += new_fish, so the new term has weight 1, not (1 - gamma) ewc.py. The paper's Fisher is an expectation over (s, a) pairs. Code averages the squared gradient of each batch's mean log-likelihood over batches ewc.py.
• ER buffer unit. Paper says the buffer stores "up to 1000 trajectories" app. A.2. Code's buffer returns the first n_memories = 1000 items of each earlier task's sequence dataset er.py datasets.py. Each item is a window of length 10 default.yaml, not a trajectory. Selection is by index, not random.
• Replay batch. Paper says 32 trajectories per step app. A.2. Code samples 32 windows, the training batch size er.py.
• PackNet pruning. Paper keeps the top 25% by magnitude app. A.2. Code matches: prune_perc = 0.75 removes the bottom 75% of the current task's weights packnet.py packnet.yaml.
• PackNet evaluation timing. The appendix says all evaluation metrics are computed before the fine-tuning stage app. A.2. FWT and the learning curves are taken before fine-tuning base.py. The success matrix is evaluated after end_task, which includes the 50-epoch fine-tuning, and that model is the best one from fine-tuning main.py packnet.py.
• Data split. Paper says 50 demonstrations per task §4.4. Code uses all of them for training. The config key train_dataset_ratio: 0.8 is defined default.yaml but not referenced by any Python file. The code has no held-out split.
• Seeds. Paper uses 100, 200, 300 app. D. Code defaults to 10000 config.yaml, and the README leaves SEED as a placeholder README.md. The three paper seeds are not in the repo.
• Metrics code. Paper defines negative backward transfer and AUC §5.1 eq. 3. The training loop returns the per-task forward-transfer value and saves the learning curves base.py. I found no code that computes NBT or AUC across tasks.
• Code-only details. Image size 128 x 128, gradient clipping at 100, augmentation, sequence length 10, and 5 GMM modes are in the configs but not in the paper text. See Hyperparameters.

Pitfalls

• Checkpoint selection uses the reported episodes. Each task's best checkpoint is picked by success over the same 20 fixed initial states that are then reported base.py metric.py. Reported success is therefore optimistic. Compare methods only under this same protocol.
• Coarse success values. Each run's success rate is a multiple of 0.05 (20 episodes) metric.py. The reported values are means over three seeds, so single-run results move in steps of 0.05.
• ER buffer covers fewer trajectories than the paper suggests. The buffer is the first 1000 windows of each task, not 1000 trajectories datasets.py. Because each trajectory produces many windows, the number of demonstrations covered depends on their length, which the paper does not state. I inferred this from the windowing; I did not run it.
• EWC coefficient. The code applies twice the penalty of the paper's stated formula for the same lambda, and its Fisher update lacks the (1 - gamma) term (see Paper vs code). A reimplementation that follows the paper's equations will not match the reported EWC numbers.
• PackNet numbers mix two timings. FWT and AUC come from before fine-tuning, while the success matrix comes from after fine-tuning. Keep them apart when comparing with other work.
• NBT for the last task is undefined. The paper's NBT averages over tau > k, which is empty for the final task (K - k = 0) eq. 3. The paper does not say how to handle this. Average over the K - 1 tasks that have a later task, or state the choice.
• Epoch 0 counts toward FWT. The zero-shot evaluation at epoch 0 is in the 11-point average base.py. An untrained policy scores near zero there, so it lowers FWT. Keep it when reproducing the paper's FWT.
• Hidden evaluation cost. Each evaluation runs 20 rollouts of up to 600 steps per task. The loop evaluates every seen task after each new task, and PackNet adds 50 fine-tuning epochs with evaluation every 5 epochs main.py packnet.py. The paper gives no runtime per run beyond the hardware app. B.
• Task order is not listed. The paper runs five task orders for Figure 4 but does not list them in the text §5.2 fig. 4. The config has only task_order_index default.yaml.
• Pretraining protocol. The pretraining run on LIBERO-90 is described in the text §5.2. The code loads a checkpoint through pretrain_model_path main.py, but I did not verify the exact pretraining script used for the paper.

Lineage

• Builds on: behaviour cloning §2.2, BC-RNN from robomimic §4.4, BC-T from VIOLA and BC-ViT from ViLT §4.4.
• Compared against: experience replay (ER), elastic weight consolidation (EWC) and PackNet, with sequential finetuning and multitask learning as bounds §4.3 app. A.2.
• Related benchmarks: ContinualWorld, CORA, and other robot-learning benchmarks named in the related-work section §6.