--- kind: concept slug: action-tokenization title: Action tokenization summary: >- Turning continuous values such as robot actions or motion clips into discrete tokens that a sequence model can predict. Covered: numbers written into a frozen LLM's prompt, per-timestep binning, a DCT-plus-BPE compressor (FAST), and a learned VQ-VAE motion tokenizer for humanoid text-to-motion (UH-1). tags: [robot-learning, tokenization, vision-language-action] papers: ["2307.04721", "2310.08864", "2412.14172", "2501.09747"] related: [imitation-learning, in-context-learning, learning-from-demonstration, vision-language-action-model, action-chunking, transfer-learning, vector-quantization] status: draft updated_at: 2026-10-25 --- ## Overview Action tokenization maps real-valued signals, such as joint positions or end-effector poses, to a sequence of discrete symbols. A sequence model then predicts those symbols with the usual next-token objective, and a decoder maps the predicted symbols back to real values. The sources cover different points in this design. One writes numbers directly into the prompt of a frozen LLM, with no learned tokenizer [arxiv:2307.04721 §5] [arxiv:2307.04721 §6]. Another bins each action dimension into a fixed set of values and trains a policy on the resulting tokens [arxiv:2310.08864 §4.2]. A third trains a tokenizer for the action chunks of a vision-language-action policy [arxiv:2501.09747 §5.2]. The fourth learns a codebook over humanoid motion clips and generates motion as codebook indices from a text command [arxiv:2412.14172 §4]. The shared requirement is that the tokens can be turned back into the signal. The FAST paper describes each of its steps as invertible [arxiv:2501.09747 §5.2]. These sources also show that the discretisation choice changes results: digit spacing changes how the prompt tokenises [arxiv:2307.04721 §4], and the scale in FAST trades reconstruction accuracy against token count [arxiv:2501.09747 §6.2] [arxiv:2501.09747 app. A.2]. ## Variants ### Plain numbers in the prompt (no learned tokenizer) The model is given a few input-output sequences as text and asked to continue the pattern. Every number is written as a short string of tokens. The paper uses an alphabet of digits, space-prefixed so that neighbouring digits do not merge, and also tests random alphabets of made-up tokens [arxiv:2307.04721 §4] [arxiv:2307.04721 tab. 1]. Robot values are rounded to integers from 0 to 100, or 0 to 300 for the whiteboard task, and each integer is one token [arxiv:2307.04721 §5]. The paper reports that a frozen text-davinci-003 solves 85 of 800 ARC problems [arxiv:2307.04721 tab. 1]. The card does not say whether these are exact-match scores. Card: [Large Language Models as General Pattern Machines](/papers/2307.04721). ### Per-timestep binning Each action dimension at each timestep is mapped to one of a fixed set of bins, and the bins are concatenated. The autoregressive baseline in the FAST paper uses 256 bins per dimension [arxiv:2501.09747 §3]. Because each timestep adds one token per dimension, the count grows with control frequency. For one-second chunks the paper reports 35, 105, 140 and 700 tokens for four datasets at 5 Hz to 50 Hz [arxiv:2501.09747 tab. 1]. On two real-robot tasks, at 20 Hz and 50 Hz, this scheme made no task progress in the authors' single-dataset experiments [arxiv:2501.09747 §6.2] [fig. 6]. Open X-Embodiment uses the same idea for one policy trained on pooled robot data. Its RT-1-X model splits each action into 8 dimensions and bins each one into 256 uniform bins [arxiv:2310.08864 §4.2]. The text normalises each dataset's actions before binning [arxiv:2310.08864 §4.1], but the released code clips to fixed bounds, and the normalisation statistics are not in that repository. The bin range is therefore set by an input the paper does not give. The reported results compare whole policies, not the tokenizer. RT-1-X beats each dataset's original method on 4 of 5 small-data domains, but it underfits the large-data domains and does not beat RT-1 there [arxiv:2310.08864 §5.1] [arxiv:2310.08864 tab. 1]. Card: [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](/papers/2310.08864). ### Frequency-domain compression (FAST) The action chunk is normalised per dimension with the 1st and 99th percentile mapped to -1 and 1 [arxiv:2501.09747 §5.2]. A discrete cosine transform is applied along time to each dimension. The coefficients are multiplied by a scale of 10 and rounded, so most high-frequency entries become zero. The result is flattened with the lowest frequency first and compressed with a byte-pair encoding of 1024 entries [arxiv:2501.09747 §5.2] [arxiv:2501.09747 alg. 1]. The same pipeline can be fitted once on about one million chunks from many robots, and the authors report that this universal tokenizer matches dataset-specific ones on their policy evaluations [arxiv:2501.09747 §5.3] [arxiv:2501.09747 §6.3]. Card: [FAST: Efficient Action Tokenization for Vision-Language-Action Models](/papers/2501.09747). ### Learned vector-quantized motion tokens (UH-1) A 1-D convolutional encoder maps a clip of humanoid motion to latent vectors, and each latent is snapped to its nearest entry in a learned codebook. Each token covers four frames, so the sequence is about a quarter as long as the clip [arxiv:2412.14172 §4] [arxiv:2412.14172 app. D.1]. The paper gives a codebook of 2048 entries [arxiv:2412.14172 app. D.3]. A causal Transformer reads a CLIP embedding of the text command and predicts the next token, with an end token to stop [arxiv:2412.14172 §4] [arxiv:2412.14172 app. D.2]. A decoder maps the generated tokens back to a motion clip. Binning uses fixed bins and FAST a fixed transform, whereas here the codebook itself is learned, so the tokens form a vocabulary of motion primitives [arxiv:2412.14172 §4]. The paper reports that a larger codebook helps: FID falls from 0.539 with 512 codes to 0.463 with 2048 [arxiv:2412.14172 §5.4] [arxiv:2412.14172 fig. 9]. On a HumanoidML3D variant the author-reported FID is 0.445 for UH-1 against 0.582 for MDM [arxiv:2412.14172 tab. 1]. The tokenizer is trained on Humanoid-X, a set of 163,800 clips that were retargeted to a robot and tracked by an RL policy before use [arxiv:2412.14172 §3]. Card: [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172). ## When to use which | Situation | Use | Why | |---|---|---| | Frozen LLM, no training budget, low-dimensional signal | Plain numbers in the prompt | No tokenizer to fit. Each value costs one model query, which the in-context paper names as a blocker for real systems [arxiv:2307.04721 §7] | | Training an autoregressive policy on high-frequency, many-dimension actions | FAST | Binning token counts grow with frequency and dimension, while FAST stays low [arxiv:2501.09747 tab. 1]; binning made no progress on the 20 Hz and 50 Hz tasks [arxiv:2501.09747 §6.2] | | Training an autoregressive policy on slow, low-dimension actions | Per-timestep binning, or FAST | Binning is the simpler scheme and needs no fitted BPE. At 5 Hz the paper's gap is smallest, 35 naive tokens against 20 [arxiv:2501.09747 tab. 1]. The paper gives no task comparison at this rate, so this row rests on token counts only | | Pooling many robot datasets into one shared action format, one policy trained on the mix | Per-timestep binning, as in RT-1-X | Reported to transfer across datasets: RT-1-X beats each dataset's original method on 4 of 5 small-data domains, though it underfits the large ones [arxiv:2310.08864 §5.1] [arxiv:2310.08864 tab. 1]. The per-dataset numbers are only in a bar plot [arxiv:2310.08864 fig. 3] | | Deploying one tokenizer across many robots | Universal FAST tokenizer | Reported to match dataset-specific tokenizers and to use fewer tokens on unseen robot datasets [arxiv:2501.09747 §6.3] [fig. 8] | | Latency-critical control with a large VLM | Check before choosing autoregressive FAST | On the authors' hardware, FAST decoding took about 750 ms per chunk versus about 100 ms for a diffusion policy [arxiv:2501.09747 §6.5]. The paper does not compare FAST with binning on latency | | Generating whole humanoid motions from text, with a large motion dataset | Learned VQ tokens, as in UH-1 | The codebook is learned, and the authors report a better FID than MDM on their benchmark [arxiv:2412.14172 tab. 1]. The reported gain depends on codebook size [arxiv:2412.14172 fig. 9]. The card records no comparison of this tokenizer with binning or FAST, and no latency figure | All FAST results are author-reported and mostly shown in plots, so they should be read as trends rather than success rates [arxiv:2501.09747 §6]. The UH-1 results are FID and success rates on its own benchmark and robot, so they do not transfer to the other rows. ## Implementation sketch The first two encoders below share the same outer shape: normalise, transform, round to integers, then serialise. They differ in the transform and in the last step. The third replaces the fixed transform with a learned encoder. The decoders run the steps in reverse. The sources do not give the exact code for the parts marked as inputs. ```pseudocode # Variant A: per-timestep binning [arxiv:2501.09747 §3]; same scheme in [arxiv:2310.08864 §4.2] def bin_encode(actions, lo, hi, n_bins): # Clip each value to the range; lo and hi are inputs, not stated in the papers clipped = clamp(actions, lo, hi) # Map each value to an integer bin, one token per dimension per timestep # Truncates, so decoding to the lower bin edge biases values low (Open X card) bins = floor((clipped - lo) / (hi - lo) * (n_bins - 1)) # Concatenate per timestep [arxiv:2501.09747 §3] return bins.flatten() ``` ```pseudocode # Variant B: DCT, rounding and BPE (FAST) [arxiv:2501.09747 §5.2] def fast_encode(actions, scale, bpe): # Normalise each dimension, 1st and 99th percentile to -1 and 1 [§5.2] # actions is assumed normalised already: (H steps, D dimensions) # DCT along time, one transform per dimension [arxiv:2501.09747 §5.2] coeffs = dct(actions, axis=0) # Scale and round; most high-frequency entries become zero [alg. 1] quantized = round(scale * coeffs) # Lowest frequency first, all dimensions per frequency [§5.2] flat = quantized.flatten() # Byte-pair encoding, vocabulary 1024, fitted on training data [§5.2] return bpe.encode(flat) ``` ```pseudocode # Variant C: learned VQ tokens (UH-1) [arxiv:2412.14172 §4] [eq. 10] def vq_encode(clip, encoder, codebook): # Encode the clip with a 1-D convolutional encoder [app. D.3] # One latent per four frames [app. D.1] latents = encoder(clip) # (T/4, C) # Snap each latent to its nearest codebook entry; the index is the token [eq. 10] distance = norm(latents[:, None] - codebook[None], axis=-1) # (T/4, K) return argmin(distance, axis=-1) # (T/4,) codebook indices ``` Practical note: the official open-source wrapper for FAST cuts the sequence at a maximum token length, which drops trailing action tokens and decodes to a wrong chunk. The FAST paper does not state a maximum length, so this comes from the code, not the paper; see the [FAST card](/papers/2501.09747). Decoders must handle invalid or truncated sequences. The UH-1 release also caps generated sequences, at 51 tokens by default, which is about 204 frames at four frames per token. The paper does not state a maximum length, so this too comes from the code; see the [UH-1 card](/papers/2412.14172). Decoding there takes the top-1 token at each step, with no sampling option. Open X-Embodiment's released code emits 11 tokens per step where its paper describes 8, and its inference script builds the model with 512 bins where the paper says 256 [arxiv:2310.08864 §4.2]. Check the checkpoint's output size before you train against it; see the [Open X card](/papers/2310.08864). ## Contradictions - Open X-Embodiment's paper and code differ. The text gives 8 action dimensions and 256 bins [arxiv:2310.08864 §4.2]. The released inference code uses 11 tokens per step and 512 bins. The text normalises each dataset before binning [arxiv:2310.08864 §4.1], while the code clips to fixed bounds. - Normalisation range inside the in-context paper: the main text maps Marker in Cup positions to 0 to 100 [arxiv:2307.04721 §6], while the appendix says 0 to 200 [arxiv:2307.04721 app. C.1]. The compiled PDF was not checked. - UH-1's paper and code differ on the tokenizer. The paper gives a 2048-entry codebook and an embedding loss plus a commitment loss [arxiv:2412.14172 app. D.1] [arxiv:2412.14172 app. D.3]. The inference code defaults to 512 codes and uses an exponential-moving-average codebook with a commitment loss only. The released checkpoints are therefore not the paper's configuration; see the [UH-1 card](/papers/2412.14172), Paper vs code. - Open X-Embodiment's and UH-1's tokenizers have different aims. The first bins actions to a fixed grid and the second learns its vocabulary. The two cards report no comparison between them. - The four papers report results on different tasks: ARC and sinusoid completion with a frozen LLM in one [arxiv:2307.04721 §4] [arxiv:2307.04721 §5]; real-robot policy training with a vision-language backbone in FAST [arxiv:2501.09747 §6.1]; whole-policy evaluation over pooled robot datasets in Open X-Embodiment [arxiv:2310.08864 §5]; and text-to-motion FID on a humanoid benchmark in UH-1 [arxiv:2412.14172 tab. 1]. Their numbers are not comparable. None of the four papers compares its tokenizer with the others, and the UH-1 card records no comparison with binning or FAST. ## Papers - [Large Language Models as General Pattern Machines](/papers/2307.04721): writes numbers and random-token sequences into the prompt of a frozen LLM and tests completion on ARC, sinusoids and robot trajectories. - [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](/papers/2310.08864): pools 60 robot datasets into one format and trains RT-1-X, which bins each action into 8 dimensions of 256 bins, and RT-2-X on the mixture. - [FAST: Efficient Action Tokenization for Vision-Language-Action Models](/papers/2501.09747): DCT-plus-BPE tokenizer for action chunks, with a universal variant and comparison against per-timestep binning and diffusion policies. - [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172): VQ-VAE tokenizer over humanoid motion clips and a text-conditioned Transformer that generates the tokens, trained on the Humanoid-X dataset.