Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

In plain words

PerAct teaches a robot arm to do tabletop jobs from a sentence such as "open the top drawer" plus a few example demonstrations per task. Instead of learning from flat camera images, it first builds a 3D grid of small cubes around the workspace from depth cameras, so every point of the scene has a known position.

The network then asks one question per step: in which cube should the gripper go next, which way should it turn, and should it open or close? The answer is scored over the whole grid at once. A Transformer that reads the grid through a small set of learned summary vectors keeps this affordable, and the language instruction tells it which job is wanted.

The authors argue that this 3D framing needs less data than image-to-action policies. They report results on 18 simulated tasks and 7 real-robot tasks, with the caveats listed below.

TL;DR

• Behaviour cloning from keyframe demonstrations, with a 100^3 voxel grid of RGB-D input and a CLIP language goal §3.
• Each step predicts a translation voxel, 5-degree rotation bins per axis, a gripper-open flag and a collision flag, all trained with cross-entropy §3.3–3.4.
• Perceiver Transformer with 2048 latents of dimension 512 keeps the 8000-patch input tractable §3.3.
• Author-reported: one multi-task agent beats the C2FARM-BC 3D U-Net in 25 of 36 task/demo-count cells §4.2, tab. 1.
• Pitfalls: batch size, training length and several optimizer settings differ between paper and code; the crop augmentation in the code does not run for PerAct (see Paper vs code).

Problem

Image-to-action policies need a lot of data to learn hand-eye coordination and to handle 6-DoF pose, and single-view observations make this harder §4.2. The paper argues that voxel-based action detection gives a structural prior, so that the agent learns 6-DoF actions with fewer demonstrations §1; §3. The setting assumes few expert demonstrations per task, a known workspace volume §3.2, calibrated RGB-D cameras app. B, and a sampling-based motion planner that executes the discretized actions §5.

Method

Overview. The agent takes a language goal and a voxelized observation of the scene and outputs one discretized action: translation, rotation, gripper open state and collision-avoidance mode §3.3; fig. 2. The action is executed by a motion planner, and the loop repeats until the goal is reached §3.3. The policy uses only the current observation, with no history app. L.

Demonstrations and keyframes §3.1–3.2. An action is a 6-DoF pose, a gripper-open flag and a collision flag §3.1. Keyframes are extracted from each demonstration with a heuristic: an action is a keyframe when joint velocities are near zero and the gripper state has not changed §3.2. The paper does not state the velocity threshold. Each datapoint becomes "predict the next keyframe action" §3.2; app. F.

Voxelization §3.2; app. B. The observation is a 100^3 voxel grid covering a 1.0 m^3 volume, i.e. 1 cm voxels §3.2; §4.2. Each voxel has 10 channels: RGB (zero-mean normalized), a point, an occupancy flag and a 3-value position index app. B. Translation is the voxel nearest the gripper fingers §3.2. Rotation is discretized into 5-degree bins per axis §3.2. Gripper open and collide are binary §3.2.

Encoding §3.3; app. B.

1. Language goal: CLIP text encoder, 77 tokens with zero padding, 512 dimensions per token; a linear layer maps them to 128 app. B.
2. Voxels: a 1x1x1 3D convolution lifts 10 channels to 64; a 5x5x5 convolution with stride 5 makes 20^3 patches app. B.
3. Proprioception: 4 scalars (gripper open, two finger positions, timestep) go through a linear layer to 64 and are tiled over the patch grid app. B.
4. Patches are flattened to 8000 tokens of width 128, the language tokens are appended, and learned positional embeddings are added. The input is 8077 x 128 app. B.

Perceiver Transformer §3.3; app. B; fig. 2. Cross-attention from 2048 randomly initialized, trained latents (dimension 512) to the input; then six self-attention layers on the latents; then cross-attention back to the input size. The output is upsampled with a 3D convolution and trilinear upsampling to a 100^3 grid with 64 channels, concatenated with the 64-channel encoder feature (skip connection), and reduced back to 64 channels §3.3; app. B.

Decoding and action selection §3.3; app. B.

• Translation: a 1x1x1 convolution gives one logit per voxel of the 100^3 grid.
• Rotation, gripper and collide: the voxel features are max-pooled over space to a 64-vector, then three independent linear heads. The rotation head has 216 logits (72 bins of 5 degrees for each of the three Euler axes); gripper and collide each have 2 logits.
• At inference, each Q-function is maximized independently, as an argmax over voxels (translation) or bins (rotation, gripper, collide) §3.3.

Loss §3.4. The four heads are trained as classifiers. The total loss is the sum of four cross-entropy terms, one for each head, against one-hot targets (translation over the voxel grid, rotation per axis, gripper, collide). The weights are 1.0 in the code PERACT_BC.yaml. The paper's loss has no extra weighting terms §3.4, unnumbered equation.

Sampling and augmentation §3.4; app. E. Training samples one datapoint per task per batch, so batches are uniform over tasks §3.4. Voxel observations and keyframe actions are perturbed together by SE(3) transforms: translation up to ±0.125 m on each axis, yaw up to ±45 degrees, and no roll or pitch app. E. Perturbations that push the action out of the grid are dropped app. E.

# PerAct training step  [§3.3] [§3.4]
# Input:  voxels (B, 100, 100, 100, 10); goal tokens (B, 77, 512)
# Output: scalar loss

def training_step(voxels, goal_tokens, keyframe_actions):
    # Perturb the observation and its target together  [app. E]
    voxels, keyframe_actions = se3_augment(voxels, keyframe_actions)

    # Encode the voxel grid into 8000 patch tokens  [app. B]
    patch_tokens = encode_voxels(voxels)              # (B, 8000, 128)

    # Encode the goal and join the two sequences  [app. B]
    language_tokens = linear_512_to_128(goal_tokens)  # (B, 77, 128)
    sequence = concat(patch_tokens, language_tokens)  # (B, 8077, 128)

    # Latent bottleneck and decoding  [§3.3] [app. B]
    voxel_features = perceiver(sequence)              # (B, 100, 100, 100, 64)

    # One logit set per head  [app. B]
    translation_logits = conv1x1_to_one(voxel_features)  # (B, 100^3)
    rotation_logits = three_linear_heads(max_pool(voxel_features))
    gripper_logits = linear_to_two(max_pool(voxel_features))
    collide_logits = linear_to_two(max_pool(voxel_features))

    # Sum of four cross-entropy losses  [§3.4]
    loss = cross_entropy(translation_logits, keyframe_actions.translation)
    loss = loss + cross_entropy(rotation_logits, keyframe_actions.rotation)
    loss = loss + cross_entropy(gripper_logits, keyframe_actions.gripper)
    loss = loss + cross_entropy(collide_logits, keyframe_actions.collide)
    return loss

Lineage of the input. The voxel formulation follows C2FARM-BC by James et al., which the authors use as the 3D U-Net baseline and whose keyframe and voxel setup PerAct adopts §3.2; §4.2. The authors state that their codebase is built on the ARM repository app. B. Keyframes follow James et al. app. F.

Hyperparameters

Values the paper does not state are marked "not stated". Code values come from conf/method/PERACT_BC.yaml and conf/config.yaml unless cited otherwise.

Results

Author-reported. Setup: RLBench simulation with a Franka Panda, four RGB-D cameras, 128 x 128 images, 25 test episodes per task and binary success §4.1. Baselines are a BC-Z-style image-to-action agent with CNN or ViT encoders and FiLM conditioning on CLIP features, and C2FARM-BC, a 3D U-Net that works on voxels and is conditioned on CLIP at the bottleneck §4.2.

Simulation, multi-task, success rate (%), 25 episodes per task tab. 1.

Author-reported summary: PerAct beats C2FARM-BC in 25 of 36 task/demo-count cells; the average improvement is 1.33x with 10 demos and 2.83x with 100 demos §4.2; tab. 1. Without language, the agent performs at chance §4.2. Three high-precision tasks (insert peg, stack cups, place cups) are near zero for all multi-task agents §4.2.

High-precision tasks, multi-task vs single-task, 100 demos tab. 5.

Global vs local receptive field, open drawer, 100 demos §4.3; fig. 4. PerAct is the only agent above 70% success. All U-Net variants sit near chance, about 33% §4.3.

Real robot, one multi-task agent trained from scratch on 53 demos, 7 tasks §4.4; tab. 2. Success rate is over the small test counts listed.

Most real-robot failures were wrong gripper-open predictions, and the agent sometimes exploits dataset biases §4.4.

Sensitivity §4.2; app. G; tab. 4. The full table is not transcribed here. The text reports that rotation perturbations generally help robustness, that fewer Perceiver latents suffice for simple short-horizon tasks, and that some tasks solve with a 32^3 grid while high-precision tasks need 100^3 §4.2. Read tab. 4 for the exact per-task values.

Ablations on open drawer §4.2; fig. 3. The skip connection speeds up training slightly. The Perceiver is crucial. Keyframe extraction is essential, since random or fixed-interval keyframes give zero success §4.2.

Paper vs code

Code: https://github.com/peract/peract at commit 02fb87681c5a47be9dbf20141bedb836ee2d3ef9 (linked from the project page, which the paper names §1; peract.github.io).

• Batch size. Paper uses 16 §3.4; the code's default replay batch is 8 config.yaml.
• Training length. Paper uses 600K iterations §3.4; the code default is 40000 config.yaml. Which value produced the reported results is not stated.
• Crop augmentation. The code has a crop-shift branch, applied only when the voxel level index is above zero launch_utils.py. PerAct uses a single level, voxel_sizes: [100] PERACT_BC.yaml. The branch therefore does not run for PerAct. The paper does not describe a crop augmentation.
• Dropout. The code uses input and attention dropout 0.1 PERACT_BC.yaml. The paper does not state dropout.
• Learning rate and weight decay. Code uses 5e-4 and L2 1e-6 PERACT_BC.yaml; the paper gives neither.
• Schedule. The code has a cosine-with-restarts warmup scheduler qattention_peract_bc_agent.py, disabled in the config PERACT_BC.yaml. The paper states no schedule. Its default warmup in the agent is 20000 steps, against 3000 in the config qattention_peract_bc_agent.py.
• Attention heads. The code uses 8 latent heads and 1 cross-attention head PERACT_BC.yaml. The paper does not state heads.
• Optimizer. The code uses LAMB with betas (0.9, 0.999) qattention_peract_bc_agent.py. The paper names LAMB and does not give betas.
• Loss. Four cross-entropy terms with weight 1.0 each, matching the paper qattention_peract_bc_agent.py.
• Rotation bins. 360 // 5 = 72 bins per axis qattention_peract_bc_agent.py, matching the paper.
• Workspace. Scene bounds of 1.0 m per axis match the 1.0 m^3 volume at 100^3 config.yaml.
• Demo augmentation. The config sets demo_augmentation_every_n: 10 PERACT_BC.yaml. The paper's appendix does not give this number.

Pitfalls

• Unnumbered equations. The paper's equations are unnumbered (equation*, align*). Cite them by section, not by equation number §3.3; §3.4.
• Table numbers. Table numbers here follow the order of the tables in the LaTeX source (tab. 1 RLBench, tab. 2 real robot, tab. 3 task descriptions, tab. 4 sensitivity, tab. 5 high precision). Check them against the compiled PDF before publishing.
• Keyframe threshold. The keyframe rule uses "near zero" joint velocities, with no numeric threshold given §3.2.
• Spatial heads. Only the translation head keeps the voxel layout. Rotation, gripper and collide heads max-pool over space first, so they see no explicit location §3.3; app. B.
• Argmax at inference. Each head is maximized on its own. The joint action is never searched over combinations §3.3.
• Motion planner. Discrete actions are executed by a sampling-based planner. Dexterous or multi-finger control is out of scope §5.
• No history. The policy sees only the current observation. Counting, ordering and partial observability tasks are not handled app. L.
• Evaluation. Objects are the same at train and test time, so generalization to unseen objects is not tested §4.1. Test-time scenes differ in pose and attributes §4.1.
• Sim vs real. Real-robot evaluation uses the last checkpoint and no validation set app. C. Sample sizes are small (5 to 10 episodes) tab. 2.
• Cost. Training took 16 days on 8 V100 GPUs for simulation §3.4. The 100^3 input has 8000 patches before the latent bottleneck §3.3.
• Resolution. The 1 cm voxel is coarser than the U-Net's finest level (0.47 cm) §4.2. Some high-precision tasks need the full 100^3 grid §4.2.
• Fourier positional features made results much worse than learned embeddings app. K.
• Upsampling inside the latent self-attention did not work app. K.

Lineage

• Builds on: C2FARM-BC (James et al.), the coarse-to-fine voxel action detector used as the 3D U-Net baseline and as the source of the keyframe and voxel setup §3.2; §4.2. No card for it exists yet.
• Builds on: Perceiver (Jaegle et al.) for the latent Transformer §3.3.
• Builds on: CLIP for the language encoder §3.3.
• Compared against: BC-Z-style image-to-action baselines and C2FARM-BC §4.2.
• Superseded by: not stated.