wikillm

Paper · 2022-06-22 · cs.LG, cs.AI, cs.CV, cs.RO

Behavior Transformers: Cloning k modes with one stone

Behavior Transformer (BeT) clones unlabeled, multimodal demonstrations. K-means turns each action into one of k bins plus a continuous residual; a MinGPT transformer predicts the bin with focal loss, and an offset head corrects the residual. Author-reported: highest block-push success and the highest probability of completing 2 to 5 kitchen tasks in a row; CARLA roughly tied.

Imitation learningAction tokenizationk-means clusteringFocal lossbehavior-cloningaction-discretizationmultimodal-demonstrationsrobot-manipulation

arXivPDFCodeMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Teaching a machine by showing it examples works well when every situation has one right move. Real recordings are messier. At the same fork, one driver turns left and another turns right, and a plain learner that averages its examples ends up heading for the middle.

Behavior Transformer keeps those choices apart. It first groups all recorded actions into a small set of typical actions. Each real action then becomes two things: which typical action is closest, and a small correction from it. A transformer that reads the recent observations predicts the typical action, and a second output predicts the correction. When the model runs, it draws one typical action according to its probabilities, so it commits to one of the recorded styles instead of blending them.

The method learns from demonstrations alone, without reward labels and without trying things in the environment.

TL;DR

  • Clones unlabeled demonstrations that contain several distinct ways of doing a task, without rewards §1 §2.
  • Splits each action into a k-means bin and a continuous residual; a MinGPT transformer predicts the bin from a window of recent observations, and a second head predicts the residual for the chosen bin §2.1–§2.3.
  • Author-reported: best block-push success and the most kitchen tasks completed in chains of 2 to 5, against six baselines; CARLA is roughly tied tab. 1.
  • The official code (MIT) follows the architecture in tab. 4 but differs in the offset-loss form, batch size, two evaluation context lengths and the number of evaluation steps. See Paper vs code.

Problem

Behavior cloning with mean-squared error is maximum likelihood under a unimodal Gaussian, so it assumes one mode per observation §2. Pre-collected human or expert data often has several modes §1. Earlier multimodal approaches need a complex generative model, an exponential number of bins, complicated training, or slow test-time optimisation §1. The target setting has no reward labels and no online interaction with the environment §2.

Method

The paper trains a discrete-plus-residual policy in three parts and samples actions at test time §2. Notation: observation o, action a, history length h, number of bins k.

  1. Objective. Find parameters that maximise the likelihood of the observed action given the observation §2, eq. 1. Under a unimodal Gaussian this reduces to MSE §2.

    \theta^* := \argmax_\theta \prod_t \mathbb P(a_t \mid o_t; \theta) eq. 1

  2. Action discretisation (fixed after training). Run k-means on all training actions to get centres A_1,\dots,A_k §2.1 alg. 1. The bin of an action is its nearest centre by L2 distance; the residual is the action minus that centre §2.1. The original action is recovered as the centre plus the residual §2.1. This encoder and decoder are not updated again §2.1. fig. 3 (A) shows the step.

  3. Sequence model with focal loss. Feed the last h observations to a MinGPT decoder §2.2 app. B.4. The input layer is replaced by a linear projection, and otherwise the architecture follows MinGPT app. B.4. At each position the model outputs a categorical distribution over the k bins §2.2. The class loss is focal loss against the ground-truth bin of each position §2.2 alg. 1:

    \mathcal L_{focal}(p_t) = -(1-p_t)^\gamma \log(p_t) §2.2

    The text does not give the value of \gamma §2.2 app. B.2.

  4. Offset head. For each observation the model also outputs k residual vectors, one per bin, each of size dim(A) §2.3. Only the ground-truth bin's residual receives a loss, using the squared-error form in eq. 2. This stops the model fitting the true action through every bin §2.3 app. B.2. The total loss is \mathcal L_{focal} + \alpha \mathcal L_{mt}, where \alpha makes the two terms of similar size at initialisation. Its value is not stated alg. 1 app. B.2. fig. 3 (B) shows the training step.

    \text{MT-Loss} = \sum_{j=1}^k \mathbb I[\lfloor a \rfloor = j] \cdot \|\langle a \rangle - \langle \hat a^{(j)} \rangle\|_2^2 eq. 2

  5. Test-time sampling. At time t, feed the last h observations §2.4. Take the bin distribution at the final position, sample one bin from it §2.4 alg. 1, and output that bin's centre plus the predicted residual for that bin §2.4. fig. 3 (C) shows the rollout.

Pseudocode
# Behavior Transformer, training and sampling  [alg. 1] [§2]

def fit_bin_centres(all_actions, k):
    # Cluster every training action into k centres  [§2.1] [alg. 1]
    centres = kmeans(all_actions, k)                  # (k, dim(A))
    return centres

def encode_action(action, centres):
    # Bin = nearest centre by L2 distance  [§2.1]
    distances = norm(action - centres, axis=-1)       # (k,)
    bin_index = argmin(distances)
    # Residual = action minus its bin centre  [§2.1]
    residual = action - centres[bin_index]            # (dim(A),)
    return bin_index, residual

def training_loss(obs_window, action_window, centres, alpha, gamma):
    # Encode each action in the window of length h  [§2.1]
    bin_targets, residual_targets = encode_actions(action_window, centres)
    # Per position: k class logits and k residuals, each dim(A)  [§2.2] [§2.3]
    class_logits, residuals = minGPT(obs_window)      # (h, k), (h, k, dim(A))
    # Focal loss on the bin distribution  [§2.2]
    class_loss = focal_loss(class_logits, bin_targets, gamma)
    # Residual loss only for the ground-truth bin  [§2.3] [eq. 2]
    chosen = residuals[arange(h), bin_targets]        # (h, dim(A))
    offset_loss = sum(square_norm(chosen - residual_targets))
    # alpha sets the two terms to similar size at initialisation  [alg. 1]
    return class_loss + alpha * offset_loss

def sample_action(obs_history, centres):
    # Take the last h observations  [§2.4]
    logits, residuals = minGPT(obs_history)           # (h, k), (h, k, dim(A))
    # Sample a bin from the distribution at the final position  [§2.4]
    probs = softmax(logits[-1])                       # (k,)
    bin_index = sample_categorical(probs)
    # Output the bin centre plus its predicted residual  [§2.4]
    action = centres[bin_index] + residuals[-1, bin_index]
    return action

Hyperparameters

Environment-dependent values from the paper tab. 4 app. B.2. The paper says these are reference values rather than the only working settings; many nearby settings train well provided training does not overfit app. B.2.

HyperparameterPoint-massCARLABlock-pushKitchenSource
Layers1346tab. 4
Attention heads2446tab. 4
Embedding width2025672120tab. 4
Dropout probability0.10.60.10.1tab. 4
Context size h210510tab. 4
Training epochs104035050tab. 4
Batch size641286464tab. 4
Number of bins k2; 3322464tab. 4

Shared across all BeT experiments tab. 5 app. B.2:

NameValueSource
OptimizerAdamtab. 5
Learning rate1e-4tab. 5
Weight decay0.1tab. 5
Betas(0.9, 0.95)tab. 5
Gradient clip norm1.0tab. 5

Values only in the code, not stated in the paper:

NameBlock-pushCARLAKitchenSource
Offset loss scale \alpha100000.0500.01000.0mingpt_blockpush_best.yaml mingpt_carla_best.yaml mingpt_kitchen_best.yaml
Focal \gamma2.02.02.0mingpt_blockpush_best.yaml mingpt_carla_best.yaml mingpt_kitchen_best.yaml
Training batch size (code)25612864train_blockpush.yaml train_carla.yaml train_kitchen.yaml
Training context (code)51010train_blockpush.yaml train_carla.yaml train_kitchen.yaml
Evaluation context (code)3103eval_blockpush.yaml eval_carla.yaml eval_kitchen.yaml
Evaluation steps (code)300700700eval_blockpush.yaml eval_carla.yaml eval_kitchen.yaml
Optimizer class (code)AdamWAdamWAdamWmodel.py

The point-mass configuration is not in the repository.

Results

Author-reported. Setup: CARLA is evaluated over 100 rollouts; block-push and kitchen over 1,000 rollouts tab. 1. Block-push columns give the probability of reaching one or two blocks and of pushing one or two blocks to their squares. Kitchen columns give the probability of completing n tasks within the allotted steps; the caption says 280 timesteps tab. 1.

Table 1, performance compared with baselines tab. 1:

BaselinesCARLA SuccessReach R1Reach R2Push P1Push P2Kitchen 1Kitchen 2Kitchen 3Kitchen 4Kitchen 5
RBC0.980.6700000000
1-NN0.990.490.050.0100.900.720.440.170
LWR10.500.060010.830.520.210
VAE00.600.050010000
Flow0.030.590.02000.040000
IBC0.250.980.040.0100.990.870.610.240
BeT (Ours)0.9810.990.960.710.990.930.710.440.02

The paper bolds LWR's CARLA score, LWR's and VAE's kitchen-1 scores, and BeT's other best values tab. 1.

Table 2, multimodality of the learned behaviour. CARLA reports left and right turn probability at the intersection, ignoring out-of-distribution rollouts. Block-push reports which block is reached first and each block's target. Kitchen reports the empirical entropy of task strings tab. 2:

BaselinesCARLA LeftCARLA RightBlock first RedBlock first GreenPush red→RedPush red→GreenPush green→RedPush green→GreenKitchen task entropy
RBC00.980.410.2500000
1-NN00.990.240.250000.012.12
LWR010.260.260.0100.010.012.29
VAE000.270.3300000.72
Flow000.310.2900000.08
IBC0.120.130.480.50000.010.012.41
BeT (Ours)0.340.640.540.460.430.440.410.402.47
Demonstrations0.500.500.500.500.500.500.500.502.96

Table 3, ablations, relative to full BeT (1.0 = BeT's average success on the task) tab. 3 §3.5 app. C:

AblationCARLABlock pushKitchen
No offsets0.940.950.78
No binning0.940.250.68
No history0.650.950.88
MLP trunk0.9000.05
Temporal convolution0.720.010.26
LSTM0.030.030.04
GPT-MDN0.300.830.86
Uniform quantisation0.900.960.90

Compute, author-reported: models have roughly 10^4 to 10^6 parameters and train within an hour on block-push on one desktop GPU, against about 14 hours for IBC §3.5. One rollout evaluation takes 1.65 s for BeT and 17.70 s for IBC §3.5. Per-action time on kitchen is 2.8 ms for BeT, 52 ms for IBC and 0.5 ms for an MLP app. B.2. Hardware: RTX 3080 for state inputs, RTX 8000 for images app. B.2.

Baselines are described in app. B.1. The paper reports that its own trajectory transformer baseline completed no kitchen tasks, and that IBC needed early stopping on CARLA and block-push app. B.1.

Paper vs code

Searched: the abstract links the project page (notmahi.github.io/bet, which redirects to mahis.life/bet) § abstract. That page links github.com/notmahi/bet, owned by the first author. Code is at commit 3d9d42352c0a0a6fb170369e95dde4ec3cfea951, MIT licence. Discrepancies:

  • Offset loss form. Paper: squared L2 norm, summed over the bin index, for the ground-truth bin eq. 2. Code: F.mse_loss on the selected residual, averaged over the batch and the action dimensions, multiplied by a scale and added to the class loss mingpt.py.
  • Offset scale. Paper gives no value for \alpha alg. 1. Code sets it per environment to 100000 (block-push, "actions are very small"), 500 (CARLA) and 1000 (kitchen) mingpt_blockpush_best.yaml mingpt_carla_best.yaml mingpt_kitchen_best.yaml.
  • Focal \gamma. Paper gives no value §2.2. Code uses 2.0 in every environment mingpt_blockpush_best.yaml.
  • Bin initialisation. Algorithm 1 says centres are initialised randomly in the action space alg. 1. Code draws k actions from the data as initial centres, runs a fixed 50 k-means iterations, and re-seeds empty clusters k_means.py. The paper does not state the iteration count.
  • Block-push batch size. Paper: 64 tab. 4. Code: 256 train_blockpush.yaml.
  • Block-push evaluation context. Paper: 5 tab. 4. Code evaluates with 3 eval_blockpush.yaml.
  • Kitchen evaluation context. Training uses 10 tab. 4. Code evaluates with 3 eval_kitchen.yaml. CARLA matches at 10 eval_carla.yaml.
  • Kitchen rollout length. Paper caption: 280 timesteps tab. 1. Code: 700 eval_kitchen.yaml.
  • Optimizer. Paper: Adam tab. 5. Code: AdamW with weight decay applied to one parameter group and not to the other model.py.
  • Hold-last-bin option. Code can keep the previous sampled bin for several steps (action_update_every). Eval configs set it to 1, so it is inactive there eval_blockpush.yaml. The paper does not describe it.
  • Matching values. Layers, heads, embedding width, dropout, bin counts and epochs in tab. 4 match the training configs for block-push, CARLA and kitchen. Point-mass has no config in the repository.

Pitfalls

  • \alpha is environment-specific and not in the paper. Without a scale, the offset term can be negligible or dominant. The code's values reflect the action magnitudes of each environment. Normalise actions or sweep \alpha; the paper gives no rule.
  • Gamma and loss reduction. Focal \gamma is missing from the paper text. The code averages the focal loss over positions loss_fn.py.
  • Residual sign. Encoding is residual = action minus centre; decoding is centre plus residual §2.1 k_means.py. Mixing the two signs gives a systematic bias.
  • Discretisation error. The discrete bins limit fidelity; the paper attributes the residual head to correcting that §2.3. Bin count is a trade-off: more bins raise the class loss and lower the offset loss app. C.2.
  • Stochastic test-time behaviour. Actions are sampled from the bin distribution at every step §2.4. Repeated rollouts vary. The paper's checklist marks error bars as not applicable; timing is averaged over three runs [checklist] app. B.2.
  • Context length differs between training and evaluation in the code. See Paper vs code. Match training context unless you are testing that choice.
  • Text and table disagree. The text describes BeT as ahead of every other method everywhere except CARLA, where LWR is slightly ahead §3.3. Table 1 also has 1-NN ahead of BeT on CARLA (0.99 vs 0.98), and LWR and VAE ahead of BeT on kitchen with one task (1.00 vs 0.99) tab. 1.
  • Citation mismatch. The main text cites minGPT with the GPT-3 reference §2.1 §2.2; the appendix cites the minGPT repository app. B.4.
  • Dataset specifics. Kitchen uses the Relay Policy Learning data rather than D4RL and zeroes the last 30 observation dimensions app. A. Block-push demonstrations come from a hard-coded controller with four modes app. A.
  • Dataset size. Kitchen uses 566 demonstrations, block-push 1,000 and CARLA 100 §3.1.
  • Costs the abstract hides. Visual inputs use a frozen ImageNet ResNet-18 encoder §3.1. The largest block-push model trains within an hour on a single desktop GPU, but the paper gives no training time for the other environments §3.5.

Lineage

  • Builds on: MinGPT §2.2 app. B.4; focal loss §2.2; the masked multi-task loss from object detection §2.3.
  • Compared against: MLP with MSE (RBC), nearest neighbour, locally weighted regression, VAE (inspired by SPiRL), normalising flow (inspired by PARROT), implicit behavioural cloning (IBC), and a trajectory transformer §3.2 app. B.1.
  • Discussion of limitations: §5.

Source and license

Adapted from “Behavior Transformers: Cloning k modes with one stone” by Nur Muhammad (Mahi) Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, Lerrel Pinto (arXiv:2206.11251), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/notmahi/bet at commit 3d9d42352c0a (MIT). Any quoted code is shown under that license.