FAST: Efficient Action Tokenization for Vision-Language-Action Models

In plain words

Robot policies built on large language-style models often write their actions out one small piece at a time, the way a language model writes words. The usual way to turn a robot's smooth movements into those pieces is to cut each joint's value into a fixed set of levels and write one piece per joint per moment. When the robot moves quickly, neighbouring moments look almost identical, so the model learns to repeat the last piece and stops learning anything useful.

FAST changes the representation before the model sees it. It treats a short stretch of motion like a sound clip and uses a frequency transform, the same family of tool that JPEG uses for pictures, so that smooth motion becomes a few large numbers and many near-zero ones. The near-zero numbers are dropped, the rest is written as one sequence, and a standard compression step shortens it further. The model then predicts this short sequence, and the decoder turns it back into motion.

The authors report that this makes such models trainable on fast, dexterous tasks where the old scheme failed, and that it lets them train a strong generalist policy with much less compute than a diffusion-based model. They also release a pretrained tokenizer meant to work on many robots without retraining.

TL;DR

• Compresses each action chunk with a DCT, a scale-and-round step and a learned byte-pair encoding (BPE) §5.1 §5.2.
• Per-timestep binning produces many highly correlated tokens at high control frequency; this is the failure the paper targets §4 fig. 4.
• A universal tokenizer trained on about 1M one-second chunks is reported to match dataset-specific tokenizers §5.3 §6.3.
• With pi0, the autoregressive variant matches diffusion pi0 on the authors' generalist evaluation with about 5x fewer training GPU hours §6.6 fig. 13.
• Inference is slower: about 750 ms per chunk versus about 100 ms for diffusion pi0 on an RTX 4090, as reported §6.5.

Problem

Autoregressive VLAs usually map each action dimension at each timestep to one of 256 bins and concatenate them, so a one-second chunk at 50 Hz can require hundreds of tokens §3 §5.2. For smooth signals, the next token is then almost determined by the previous one, so the next-token loss gives little signal and models can settle for copying the last action §1 §4. The paper shows this in a toy spline-interpolation task where binning error rises as the sampling rate increases, while the DCT-compressed targets stay low §4 fig. 4. On the authors' real-robot tasks, binning produces no progress on Table Bussing (20 Hz) and T-Shirt Folding (50 Hz) §6.2 fig. 6.

Method

Scope. The method is a tokenizer for action chunks. The backbone is unchanged; the paper tests it with pi0 (PaliGemma-3B) and OpenVLA (Prismatic-7B) §6.1 app. A.3.

Normalization. For each action dimension, map the 1st and 99th percentile of the training data to -1 and 1, using quantiles so that rare outliers do not set the scale §5.2.

Transform. Apply a discrete cosine transform to each action dimension separately over the time axis of the chunk §5.1 §5.2 fig. 3.

Quantize. Multiply the DCT coefficients by a scale gamma and round to integers. Most high-frequency entries become zero §5.2 alg. 1.

Flatten. Put the coefficients into one sequence with the lowest-frequency row first and, within a row, all action dimensions in order. The paper reports that this order matters for training; the other order (all frequencies of one dimension first) was not chosen §5.2.

Compress. Train a BPE on the flattened integer sequences of the training set, with vocabulary size 1024. BPE removes the zero runs and merges frequent coefficient patterns §5.2 alg. 1.

Decode. Invert each step: BPE tokens to integers, divide by gamma, inverse DCT. The paper calls every step invertible §5.2.

Universal tokenizer. Fit the same pipeline once on about one million one-second chunks from many robots, padded to 32 action dimensions. The data mix is listed in tab. 2 app. A.1. The learned BPE then stays fixed for new robots §5.3.

Policy training. Action tokens replace rarely used entries of the VLM vocabulary, with the same chunk length (one second) for all tasks; the proprioceptive state is binned into 256 bins and written as text §6.1 app. A.3. Training uses next-token cross-entropy on the action tokens only tokenizer.py pi0_fast.py.

# FAST encoder  [alg. 1]  [§5.2]
# Input:  actions (H, D), normalized to about [-1, 1]
# Output: list of action tokens

def fast_encode(actions, scale, bpe):
    # 1. DCT along time, separately for each action dimension  [§5.2]
    coeffs = dct(actions, axis=time)                    # (H, D)
    # 2. Scale and round; most high-frequency entries become 0  [§5.2]
    quantized = round(scale * coeffs)                   # (H, D) ints
    # 3. Low frequencies first, all dimensions per frequency  [§5.2]
    flat = quantized.flatten()                          # row-major
    # 4. Byte-pair encoding over the flat integers  [§5.2] [alg. 1]
    return bpe.encode(flat)

# FAST decoder  [§5.2]  (inverse of the encoder)
# Input:  tokens, scale, bpe, horizon H, dimensions D
# Output: actions (H, D)

def fast_decode(tokens, scale, bpe, horizon, dims):
    # 1. Expand BPE tokens back to the integer coefficient sequence  [§5.2]
    flat = bpe.decode(tokens)
    # 2. Reshape to (frequency, dimension) and undo the scaling  [§5.2]
    coeffs = flat[: horizon * dims].reshape(horizon, dims) / scale
    # 3. Inverse DCT along time, per dimension  [§5.2]
    return idct(coeffs, axis=time)                      # (H, D)

Denormalization after decoding is the inverse of the quantile map. The DCT normalization convention is not in the paper; see Pitfalls.

Hyperparameters

Not stated: the BPE implementation, the DCT normalization convention, the number of training steps for the single-dataset experiments, and the exact text template for the prompt.

Results

Author-reported. Setup: pi0 or OpenVLA backbones; six real robots plus LIBERO in simulation; comparisons to per-timestep binning and to FSQ §6.1. The per-task success curves are plotted and their values are not stated in the text, so cite the figures rather than quoting numbers.

Token counts per one-second chunk tab. 1, at comparable reconstruction error:

Other reported results:

Data scale: the generalist run uses 903M timesteps from the authors' data plus about 9.1% open-source data §6.6. The paper's headline scale is 10k hours of robot data §1.

Paper vs code

The official code is the openpi repository, under Physical-Intelligence, at commit 215abfb217dbac7d5f1273282331b9b1866c0479, which is later than the paper's configurations. The repository includes the FASTTokenizer wrapper and the pi0-FAST model and training configs; it does not contain the DCT and BPE implementation. That lives in the released tokenizer on Hugging Face (physical-intelligence/fast), which the wrapper loads with AutoProcessor.from_pretrained(..., trust_remote_code=True) tokenizer.py.

• Scale 10 and BPE vocabulary 1024 are the Hugging Face processor's defaults, matching §5.2 [hf: physical-intelligence/fast, processing_action_tokenizer.py].
• DCT normalization: the Hugging Face processor applies an orthonormal DCT (norm="ortho") along time [hf: physical-intelligence/fast, processing_action_tokenizer.py]. The paper does not state this.
• Chunk length for DROID: the paper uses 15-step chunks app. A.4; the pi0_fast_droid config uses action_horizon=10 config.py, and the full-DROID FAST config uses 16 config.py.
• Optimizer schedule: the paper uses a constant 5e-5 after 1k warmup steps and EMA 0.999 app. A.3. The FAST defaults are cosine decay with peak 2.5e-5 optimizer.py and EMA 0.99 config.py. The full-DROID FAST config sets a 1k-step warmup with peak 5e-5 and decay to 5e-5 config.py but does not override EMA config.py.
• AdamW settings match the paper: b1 0.9, b2 0.95, clip norm 1.0 optimizer.py. Weight decay is set to 1e-10 rather than zero optimizer.py.
• Sampling: the FAST sampler defaults to greedy decoding (temperature 0.0) pi0_fast.py. The paper's 0.7 temperature for bimanual tasks app. A.3 does not appear in the code I read, so the code does not confirm it.
• Maximum token length: the paper does not state one; the model default is 250 pi0_fast.py, and the LIBERO config uses 180 for single-arm robots with a note that about 250 suits bimanual robots config.py.
• Token placement: the paper says action tokens overwrite the least-used VLM tokens §6.1. The code maps action token ids to the tail of the PaliGemma vocabulary, skipping the last 128 entries and counting down from the top, without an overwrite step tokenizer.py tokenizer.py.
• Proprioceptive state: the 256-bin discretization over [-1, 1] matches the paper app. A.3 tokenizer.py. The code also fixes a text template, "Task: ..., State: ...;" followed by "Action: ... |", which the paper does not give tokenizer.py.
• Quantile normalization in code maps the 1st and 99th percentile to -1 and 1 with a small epsilon in the denominator transforms.py. It is enabled for all non-pi0 models config.py; the default flag is off config.py.
• Decoding fallback: if "Action: " is missing from the output, the code returns zeros for the whole chunk tokenizer.py.

Pitfalls

• Lossy by design. The scale gamma trades reconstruction accuracy for compression. The reported compression numbers hold at comparable reconstruction error §6.2 app. A.2 fig. 14. A different gamma changes both the token count and the decoded actions.
• Flatten order matters. The paper reports a clear effect on training, and it uses lowest-frequency-first ordering §5.2. Reversing it is not a neutral choice.
• Truncation is silent in the wrapper. If the prompt, state and action tokens exceed the maximum length, the wrapper cuts the sequence and only logs a warning tokenizer.py. Action tokens at the end are then lost, and the decoded chunk is wrong.
• Variable length output. The number of tokens differs by chunk, so a decoder must cope with truncated or invalid sequences. The paper reports about 30 tokens per arm per chunk for its datasets §6.2, but not the distribution.
• Per-dataset BPE is learned. Fitting takes minutes for a dataset §5.3. The integer range depends on the training data; the Hugging Face fit sets the alphabet from the min and max quantized values [hf: physical-intelligence/fast, processing_action_tokenizer.py]. Tokenizers fitted on different data are not interchangeable.
• Slow inference. Decoding needs 30 to 60 tokens per chunk and the full 2B language model, versus 10 diffusion steps and a 300M action expert for diffusion pi0 §6.5. The paper notes that speculative decoding and quantization are not tried.
• Temperature for bimanual tasks. The paper needed a temperature of 0.7 to move bimanual policies out of the home position, because some training episodes start with long stationary stretches app. A.3. Greedy decoding may stall there.
• Universal tokenizer and eval overlap. The authors say the universal tokenizer was trained on data that includes their own real-robot evaluation tasks §6.1. Compression on the held-out robot set §6.3 is the cleaner test.
• Results are mostly in plots. Per-task success values and the DROID quantitative suite are shown in figures fig. 6 fig. 11 fig. 13 and in a table of 44 trials across 16 tasks scored by a progress rubric tab. 3. Do not read these as success rates.
• DROID evaluation is partly qualitative. The "zero-shot" claim rests on qualitative tests across three campuses, where success is not measured §6.2 fig. 7 app. A.5.
• Action dimension padding. The universal tokenizer pads actions to 32 dimensions before tokenization app. A.1. Pad values and their effect on compression are not stated.

Lineage

• Builds on: byte-pair encoding for text §2; the DCT and time-series compression as used in JPEG §5.1; the binning scheme used by RT-2 and OpenVLA, which it replaces §3 §2.
• Compared against: per-timestep binning (RT-2, RT-2-X, OpenVLA) §6.1; FSQ, a learned quantizer on the same chunks §6.1 §6.2; diffusion pi0 §6.5 §6.6.
• Built on by: the pi0-FAST configs in the official openpi repository config.py.