--- kind: concept slug: vector-quantization title: Vector quantization summary: >- Vector quantization (VQ) snaps each vector to its nearest codebook entry, so a generator can predict discrete codes, a planner can use landmarks, or a stored set of vectors can shrink. Covers residual and multi-scale stacks, bit-wise codes, frozen tokenizers, action tokens and k-means codecs. Helps pick a variant by budget. tags: [tokenization, generative-models, vq-vae, latent-representation, compression, vector-search, planning, speech] papers: ["2102.12092", "2104.10157", "2205.10643", "2206.08916", "2209.00647", "2212.04493", "2212.05199", "2301.00704", "2306.14795", "2308.10897", "2312.00063", "2401.08281", "2404.02905", "2406.17074", "2411.07760", "2412.04431", "2412.14172"] related: [variational-autoencoder, latent-diffusion, masked-image-modeling, masked-language-modeling, discrete-diffusion, classifier-free-guidance, in-context-learning, multimodal-language-model, autoregressive-image-generation, gaussian-splatting, post-training-quantization, nearest-neighbor-search, k-means-clustering, text-to-image-generation, instruction-tuning, motion-prediction, language-model-pretraining, hierarchical-reinforcement-learning, offline-reinforcement-learning, goal-conditioned-reinforcement-learning, action-tokenization, motion-retargeting, discrete-speech-units, self-supervised-learning] status: draft updated_at: 2026-11-01 --- ## Overview Vector quantization (VQ) replaces each continuous latent vector with the index of its nearest entry in a learned codebook. The result is a grid of integers. A second model can then predict those integers with the same cross-entropy tools used for text, which is why several of the papers below build a language-model-style prior over codes. Most variants here are the quantizer inside generative tokenizers. Two variants use the same lookup for storage instead: post-training k-means shrinks a trained 3D scene, and product quantization shrinks a vector index for search. One uses the codes as a state abstraction for planning: QPHIL maps states to landmarks, and a planner chooses a route through them. One uses the codes only as targets for a contrastive loss, not as a generative vocabulary [arxiv:2205.10643 §3.3.2]. This page does not cover model-weight quantization, which is covered by [post-training quantization](/concepts/post-training-quantization). The shared core is an encoder, a snap to the nearest code, a decoder, and a loss whose main parts are a reconstruction term and a term that pulls encoder outputs toward their codes. Some variants add a third term, as the landmark tokenizer does [arxiv:2411.07760 §4.1]. The snap has no gradient, so a straight-through estimator passes the gradient across it [arxiv:2312.00063 §3.1] [arxiv:2104.10157 §2.1]. In a generative tokenizer, the invariant every variant keeps is that a decoder must rebuild the input from the codes. The quality of the tokenizer therefore bounds the quality of anything generated from its codes [arxiv:2212.05199 §3.1] [arxiv:2206.08916 §4.6]. The variants differ on three choices. The first is how many quantizers stack: one codebook, several residual layers, one shared codebook applied at several scales, or bits with no lookup at all. The second is how codes are written and predicted: in raster order, in parallel masked rounds, or scale by scale. The third is whether the quantizer is trained at all, or frozen from a pretrained tokenizer. A fourth is what the codes are for: tokens to generate, landmarks to plan with, or stored attributes. ## Variants ### One codebook, autoregressive prior VideoGPT trains a 3D-convolutional VQ-VAE with axial attention. It maps a clip to a grid of codes, then trains a GPT-style transformer prior over those codes [arxiv:2104.10157 §3]. Codebooks have 512 or 1024 entries, with embedding size 64 or 256 depending on the dataset [arxiv:2104.10157 tab. 8]. On BAIR, the codebook sweep shows FVD flat at about 104 from 256 to 4096 codes, while reconstruction (R-FVD) improves from 18.2 to 11.3 [arxiv:2104.10157 tab. 6]. Reported results are FVD 103.3 on BAIR [arxiv:2104.10157 tab. 1] and Inception Score 24.69 on UCF-101 [arxiv:2104.10157 tab. 2]. A separate ablation splits the encoder channels across several codebooks. That is not residual stacking, and FVD worsens across its rows [arxiv:2104.10157 tab. 7]. Card: [VideoGPT: Video Generation using VQ-VAE and Transformers](/papers/2104.10157). ### One codebook, parallel masked decoding MAGVIT tokenizes a 16-frame 128×128 clip into 4×16×16 codes from a 1024-entry codebook [arxiv:2212.05199 §3.1] [arxiv:2212.05199 app. B.2]. Its tokenizer is initialised by inflating a 2D VQ trained on ImageNet [arxiv:2212.05199 §3.1]. A bidirectional transformer learns masked token prediction. Generation fills the codes in 12 rounds, keeps the most confident guesses, and re-masks the rest [arxiv:2212.05199 §3.2] [arxiv:2212.05199 alg. 1]. On BAIR frame prediction, 12 masked rounds over 1024 tokens reach FVD 76, against 91 for autoregressive decoding over the same tokens in 1024 steps [arxiv:2212.05199 §4.5]. The paper reports the autoregressive TATS baseline as about 60 times slower at inference [arxiv:2212.05199 §4.3]. Card: [MAGVIT: Masked Generative Video Transformer](/papers/2212.05199). ### One codebook, masked decoding with a frozen text encoder Muse uses the same kind of masked prediction for text-to-image. A VQGAN with downsampling factor 16 turns a 256×256 image into 16×16 tokens from an 8192-entry codebook. A second VQGAN with factor 8 gives 64×64 tokens for the 512×512 stage [arxiv:2301.00704 §2.2] [tab. 6]. A frozen T5-XXL encoder supplies the text condition, and a transformer predicts the masked tokens with cross-entropy at the masked positions [arxiv:2301.00704 §2.1] [§2.3]. Training draws a masking rate per example from an arccosine density, with mean about 0.64 [arxiv:2301.00704 §2.6]. Sampling starts fully masked and commits the most confident tokens at each step, so 256 base tokens take 24 steps [arxiv:2301.00704 §2.8]. The 3B model reports zero-shot COCO FID 7.88 [arxiv:2301.00704 tab. 2]. No code or weights were released. The paper does not give the cosine formula or the temperature behind its headline results, so an implementation has to choose them [arxiv:2301.00704 §2.8] [§5]. Card: [Muse: Text-To-Image Generation via Masked Generative Transformers](/papers/2301.00704). ### One codebook, joint text and image stream DALL-E trains a discrete VAE with one codebook of 8192 entries. It maps each 256×256 image to a 32×32 grid of codes, so 1024 image tokens [arxiv:2102.12092 §2.1]. A 12-billion-parameter transformer then reads the caption tokens followed by the image tokens as one stream [arxiv:2102.12092 §2.2]. When the transformer is trained, each image becomes tokens by taking the argmax over the encoder's logits, not by sampling. The authors note that the bound strictly calls for sampling [arxiv:2102.12092 §2.2]. Sampling draws 512 candidate grids and keeps the top k by a contrastive reranker [arxiv:2102.12092 §2.6]. Reported results are author-reported: 90% of raters preferred its realism over DF-GAN on MS-COCO [arxiv:2102.12092 §3.1]. The transformer is not released, only the dVAE (see the card). Card: [Zero-Shot Text-to-Image Generation](/papers/2102.12092). ### Residual stacks of codebooks MoMask quantizes each motion with a residual VQ-VAE. Each of six layers quantizes what the layers below it missed, and the decoder sums the layer codes [arxiv:2312.00063 §3.1] [arxiv:2312.00063 eq. 1]. Each layer has a 512-entry codebook [arxiv:2312.00063 §4]. Quantization dropout switches off the last few layers with probability q during training, so the early layers carry more of the motion [arxiv:2312.00063 §3.1]. A masked transformer models the base layer, and a residual transformer predicts each finer layer conditioned on the layers below [arxiv:2312.00063 §3.2] [arxiv:2312.00063 §3.3]. The ablation shows a depth trade-off. Reconstruction FID on HumanML3D falls from 0.091 with no residual layers to 0.014 with six or seven [arxiv:2312.00063 tab. 2]. Generation FID is best at five layers (0.051) and worse at six and seven (0.076 and 0.084) [arxiv:2312.00063 tab. 2]. The paper says generation quality starts to degrade above five layers [arxiv:2312.00063 §4.2]. Quantization dropout at q = 0.2 gives generation FID 0.051, against 0.091 at q = 0 [arxiv:2312.00063 tab. 2]. Card: [MoMask: Generative Masked Modeling of 3D Human Motions](/papers/2312.00063). ### Multi-scale codes from one shared codebook VAR quantizes an image at ten scales, from 1×1 up to 16×16, with one shared codebook of 4096 entries [arxiv:2404.02905 §3.2] [arxiv:2404.02905 §4]. At each scale the residual is pooled to that size, the nearest codes are found, and their reconstruction is subtracted before the next scale [arxiv:2404.02905 alg. 1]. A GPT-style transformer predicts each whole token map in one pass, conditioned on the coarser maps [arxiv:2404.02905 §3.2]. The paper's ablation moves from raster order to next-scale order at a similar size, and FID falls from 18.65 to 5.22 [arxiv:2404.02905 tab. 3]. The 2.0B model reports FID 1.92, or 1.73 with rejection sampling, against 18.65 for the VQGAN raster baseline [arxiv:2404.02905 tab. 1]. Card: [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](/papers/2404.02905). ### Bit-wise quantization without a codebook Infinity has no codebook lookup. Each channel is reduced to a sign and the code is normalised, so the vocabulary is 2^d without being stored. The text-to-image model uses d = 32 [arxiv:2412.04431 §3.2] [arxiv:2412.04431 §4.5]. The output head predicts each bit as a two-way decision, not a softmax over the whole vocabulary. The paper's example is that a 2^32-way classifier at width 2048 would need about 8.8 trillion weights [arxiv:2412.04431 §3.3]. On a 2^16 vocabulary, the bit-wise head reaches FID 3.83, against 4.49 for a conventional head [arxiv:2412.04431 tab. 5]. On ImageNet 256 with stride 16, reconstruction rFID is 0.61 at 2^32 and 0.33 at 2^64, against 0.87 for a continuous SD VAE [arxiv:2412.04431 tab. 4]. The paper also flips training bits (bit-wise self-correction), which lowers FID to 3.48 from 9.76 with teacher forcing [arxiv:2412.04431 tab. 7]. That is a training change, not a quantizer change. The entropy-penalty weight is not stated [arxiv:2412.04431 §3.2]. Card: [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](/papers/2412.04431). ### Large shared codebook inside a multi-task sequence model Unified-IO encodes images as 256 codes from an ImageNet-pretrained VQ-GAN. The VQ-GAN has compression ratio 16 and a 16384-entry codebook, and its codes join the shared text vocabulary [arxiv:2206.08916 §3.1] [arxiv:2206.08916 §3.2]. Generated code sequences are decoded back to images by the same VQ-GAN decoder [arxiv:2206.08916 §3.1]. The authors say the fixed tokenizer limits image generation and depth and normal quality, and that a stronger tokenizer would help [arxiv:2206.08916 §4.1] [arxiv:2206.08916 §4.6]. Card: [Unified-IO: A unified model for vision, language, and multi-modal tasks](/papers/2206.08916). ### Motion codes as words in a language-model vocabulary MotionGPT follows the Unified-IO pattern for motion. A VQ-VAE with 512 codes of dimension 512 encodes a motion clip, with time downsampled by 4 [arxiv:2306.14795 §3.1] [§4.1]. Each code becomes a word in the vocabulary of a Flan-T5 encoder-decoder, and start and end tokens wrap each motion [arxiv:2306.14795 §3.2]. The paper names EMA updates and codebook reset for codebook use [arxiv:2306.14795 §3.1]. Training has three stages: the tokenizer, span corruption over text and motion, then instruction tuning on prompt templates [arxiv:2306.14795 §3.3]. One model covers text-to-motion, motion-to-text, prediction and in-between [arxiv:2306.14795 tab. 3] [tab. 4] [tab. 5]. The author-reported text-to-motion FID on HumanML3D is 0.232 for the task-tuned model, while T2M-GPT reports 0.116 in the same table [arxiv:2306.14795 tab. 3]. The authors describe the results as competitive on most metrics, not best on all [arxiv:2306.14795 §4.2]. Card: [MotionGPT: Human Motion as a Foreign Language](/papers/2306.14795). ### Motion codes interleaved with timed words The listener model uses the same kind of motion tokenizer, with 256 codes and one token per 8 frames [arxiv:2308.10897 §3.2] [arxiv:2308.10897 app. B]. The difference is placement. Each word goes before the motion token whose 8-frame window follows the word's end time, so text and motion line up in time [arxiv:2308.10897 §3.3]. A GPT-2 Medium, initialised from text-only pretraining, predicts the next motion token. Motion tokens are new vocabulary entries with random initial embeddings, and the output layer is replaced by a projection onto the 256 motion codes [arxiv:2308.10897 §3.3]. During training, input motion tokens are corrupted with probability 0.5 [arxiv:2308.10897 app. B]. Author-reported results on the Trevor Noah listener: the full model has FD 18.22, against 22.81 when the language model is not text-pretrained [arxiv:2308.10897 tab. 2]. The variant without interleaving reaches FD 19.03, and the authors say the gap to the full model is not significant [arxiv:2308.10897 tab. 2] [§6.1]. In a human A/B test, listeners preferred the method over nearest-neighbour retrieval 70.1% of the time [arxiv:2308.10897 §5.2]. The model uses the speaker's transcript only, not audio or video, so reactions cued by prosody or the speaker's face are not modelled [arxiv:2308.10897 §6.4]. Card: [Can Language Models Learn to Listen?](/papers/2308.10897). ### Frozen codes as masked-prediction targets MAE-VQGAN keeps a VQGAN tokenizer frozen and uses its codes as targets. A ViT-L masked autoencoder predicts a distribution over 1024 codes for each hidden patch, trained with cross-entropy [arxiv:2209.00647 §3.1]. The predicted codes are decoded by the frozen VQGAN decoder [arxiv:2209.00647 §3.1]. Trained on arXiv figures, code targets beat pixel targets on segmentation (27.83 against 17.42 on split 0) and on colour LPIPS (0.40 against 0.55). Pixel targets are better on colour MSE (0.43 against 0.67) [arxiv:2209.00647 tab. 1]. The ImageNet-only version of the same model is far lower [arxiv:2209.00647 tab. 1]. Card: [Visual Prompting via Image Inpainting](/papers/2209.00647). ### Quantized targets for contrastive speech pretraining wav2vec 2.0 masks the convolutional features of speech before a Transformer. The unmasked features are quantised into discrete targets, using a Gumbel-softmax quantiser over several codebooks at once (product quantisation). A diversity term encourages every codebook entry to be used [arxiv:2205.10643 §3.3.2]. The loss is InfoNCE with cosine similarity. It is applied only at masked steps, and the negatives are other masked steps [arxiv:2205.10643 §3.3.2]. The review does not give the number of groups or entries [arxiv:2205.10643 §3.3.2]. The codes serve only as targets, so this variant has no prior over codes to sample from. HuBERT is a different case. Its targets are offline k-means clusters, with no learned quantiser inside the model. Its cluster count is not stated [arxiv:2205.10643 §3.4.2]. Card: [Self-Supervised Speech Representation Learning: A Review](/papers/2205.10643), which covers both models as a survey and reports no new method. ### Quantized latent under diffusion SDFusion compresses truncated signed-distance grids with a 3D VQ-VAE, then runs a conditional diffusion model in that latent space [arxiv:2212.04493 §3.1] [arxiv:2212.04493 §3.2]. The authors say that diffusion on a high-resolution 3D grid is too expensive, which is the reason for the latent stage [arxiv:2212.04493 §3]. The main text states neither the latent dimension nor the codebook size [arxiv:2212.04493 §3.1], and the paper gives no weight for the VQ and commitment losses [arxiv:2212.04493 §3.1]. Reported results are a lower UHD than AutoSDF on ShapeNet completion, 0.0557 against 0.0567 [arxiv:2212.04493 tab. 1], and Pix3D chamfer distance 1.852 [arxiv:2212.04493 tab. 2]. Card: [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](/papers/2212.04493). ### Learned landmarks for hierarchical planning QPHIL quantizes states, not generated outputs. A VQ-VAE maps each state to one of a small set of landmark tokens. A transformer plans a sequence of landmarks toward a goal, and two offline IQL policies follow it: one moves between landmarks, and a goal policy takes over for the exact goal once the plan is used up [arxiv:2411.07760 §4] [§4.3]. The tokenizer adds a temporal triplet loss, so states close in time receive close latents [arxiv:2411.07760 §4.1]. Its planner is trained by teacher forcing on the points where the token changes, and the authors say conditioning on the full compressed history helps [arxiv:2411.07760 §4.2]. The paper's table lists codebook sizes from 24 to 96 by maze, but does not name the window or map its loss coefficients to the text's symbols [arxiv:2411.07760 app. B.2]. The method assumes the low-level policy can reach any state inside a landmark from any other state of it [arxiv:2411.07760 §4.2]. Stitching splices two trajectories that share a pair of landmarks, which adds plans the data does not contain [arxiv:2411.07760 §4.2]. Author-reported on AntMaze-Ultra with diverse data, QPHIL without stitching reaches 70.3 ± 6.9, against 52.6 ± 8.7 for HIQL without representation [arxiv:2411.07760 tab. 1]. On the new AntMaze-Extreme map with diverse data, the stitched variant reaches 39.5 ± 13 and the unstitched one 12.5 ± 8.5, against 21.9 ± 6.6 for HIQL without representation [arxiv:2411.07760 fig. 5]. Card: [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](/papers/2411.07760). ### Action tokens for text-conditioned humanoid control UH-1 uses a 1-D convolutional encoder-decoder with a vector-quantized bottleneck. Each token covers four frames, and the codebook has 2048 entries [arxiv:2412.14172 §4] [app. D.1]. A causal transformer reads a CLIP embedding of the command and predicts the next token, with an end token to stop [arxiv:2412.14172 §4] [app. D.2]. The paper does not state the tokenizer's loss weights or its optimiser [arxiv:2412.14172 app. D.1]. Two control modes sit on top of the same tokens. In text-to-keypoint mode, the model emits keypoints and a goal-conditioned policy tracks them in closed loop. In text-to-action mode, the model emits actions for open-loop control, and the lower body then needs a separate locomotion policy [arxiv:2412.14172 §4] [§5.3]. On HumanoidML3D, the author-reported FID is 0.445 against 0.582 for MDM and 0.667 for T2M-GPT [arxiv:2412.14172 tab. 1]. On a real Unitree H1-2, the authors report 70% to 100% success per command [arxiv:2412.14172 tab. 3]. Card: [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172). ### Post-training k-means codebooks for a stored representation Not a generative tokenizer. Reduced 3D Gaussian Splatting trains the scene first, then clusters each attribute group, except position, into 256-entry k-means codebooks, and stores one byte per value [arxiv:2406.17074 §4.3]. The twenty groups are opacity, the three scale components (one shared codebook), the two rotation parts, base colour and 15 spherical-harmonic groups [arxiv:2406.17074 §4.3]. Positions and codebook entries are stored as 16-bit floats [arxiv:2406.17074 §4.3]. There is no prior over codes and no training through the quantizer, so the codebook only needs to be fitted once. Its effect is one part of a larger pipeline that also prunes Gaussians and drops spherical- harmonic bands. The paper reports a file of 29 MB against 744 MB for the 3DGS re-run on Mip-NeRF360, with PSNR 27.10 against 27.42 [arxiv:2406.17074 tab. 1]. The paper's own ablation finds half-float storage without codebooks smaller than the k-means version, 29 MB against 38 MB on Mip-NeRF360, with PSNR 27.10 against 27.17 [arxiv:2406.17074 tab. 4]. The reference code is under a non-commercial licence, so this page cites the paper only. Card: [Reducing the Memory Footprint of 3D Gaussian Splatting](/papers/2406.17074). ### Product quantization for search, not generation Faiss compresses a vector set so that approximate nearest-neighbour search fits in memory. A single k-means codebook over the whole vector is the best fit for the error, but it is impractical beyond about 3-byte codes. So a product quantizer splits the vector into M sub-vectors, each with its own k-means codebook of K entries [arxiv:2401.08281 §4.1]. The notation PQ6x10 means six sub-vectors with 2^10 entries each [arxiv:2401.08281 §4.1]. Distances are estimated from the stored codes, usually by asymmetric distance computation, which compares the raw query with the decoded vectors [arxiv:2401.08281 §4]. Product quantization is paired with an inverted file: a coarse quantizer picks the nearest lists, and only those are scanned [arxiv:2401.08281 §5.1]. There is no prior over codes, so this variant does not generate anything. The paper's largest index, 1.5 trillion vectors, uses a different codec: PCA to 72 dimensions and 6-bit scalar quantization, with about 1 s per query, author- reported [arxiv:2401.08281 §7.1]. Card: [The Faiss library](/papers/2401.08281). ## When to use which The choice depends on the decoding budget, the pretraining budget, and how much detail one codebook can carry. | Situation | Use | Why | |---|---|---| | Few sampling rounds, not one token per step | Masked decoding over one codebook, as in [MAGVIT](/papers/2212.05199) | 12 rounds reach FVD 76 on BAIR, against 91 for 1024-step autoregressive decoding [arxiv:2212.05199 §4.5] | | Simplest baseline, and a full-attention prior is affordable | Autoregressive prior, as in [VideoGPT](/papers/2104.10157) | One code at a time, with a small set of settings; BAIR FVD 103.3 [arxiv:2104.10157 tab. 1]. Its released code differs from the paper in several settings, see the card | | Need more detail than one codebook carries | Residual layers, as in [MoMask](/papers/2312.00063) | Reconstruction FID falls from 0.091 to 0.014 with residual layers [arxiv:2312.00063 tab. 2]. Generation worsens above five layers, so stop there | | Huge vocabulary, and no large softmax | Bit-wise head, as in [Infinity](/papers/2412.04431) | Avoids a 2^32-way classifier [arxiv:2412.04431 §3.3]. Its text-to-image results use next-scale generation, not raster order [arxiv:2412.04431 §3.1] | | Pretrained image tokenizer, no budget to retrain it | Frozen codes as targets, as in [MAE-VQGAN](/papers/2209.00647) | Code targets help segmentation and colour LPIPS. Pixel targets keep colour MSE better [arxiv:2209.00647 tab. 1] | | One output vocabulary shared by text and vision | A fixed large codebook, as in [Unified-IO](/papers/2206.08916) | Images and text share one softmax [arxiv:2206.08916 §3.2]. The ImageNet-only tokenizer caps image quality [arxiv:2206.08916 §4.6] | | 3D shapes where the raw grid is too large to diffuse | Quantized latent under diffusion, as in [SDFusion](/papers/2212.04493) | Diffusion runs in the compressed latent [arxiv:2212.04493 §3] | | A trained representation must shrink, with no retraining | Post-training k-means codebooks, as in [Reduced 3DGS](/papers/2406.17074) | Disk size falls about 25 times on Mip-NeRF360 [arxiv:2406.17074 tab. 1]. Codebooks are decoded at load time, and the paper's ablation finds half-float storage smaller here [arxiv:2406.17074 tab. 4] | | A vector index must fit in memory, for search only | Product quantization with an inverted file, as in [Faiss](/papers/2401.08281) | Graph indexes suit below about 1M vectors when memory is not limiting. Compressed inverted files suit larger sets [arxiv:2401.08281 §5.3]. Tune by measuring on your own data [arxiv:2401.08281 §3.4] | | Text-to-image with a frozen language encoder, and fewer steps than autoregressive decoding | Masked decoding over one codebook, as in [Muse](/papers/2301.00704) | 256 base tokens in 24 steps [arxiv:2301.00704 §2.8]. Author-reported 0.5 s per batch against 6.4 s for Parti-3B, both timed internally on TPU-v4, batch size not stated [arxiv:2301.00704 tab. 3] | | Speech encoder pretraining with discrete targets, and no generative use of the codes | Quantized targets with a contrastive loss, as in wav2vec 2.0 (reviewed in [Self-Supervised Speech Representation Learning](/papers/2205.10643)) | The review names wav2vec 2.0 and HuBERT as the go-to choices [arxiv:2205.10643 §7]. Its results are author-reported and use different fine-tuning recipes, so they are not directly comparable [§4.5] | | One language model for several motion tasks | Codes as words in a language-model vocabulary, as in [MotionGPT](/papers/2306.14795) | One model covers four motion tasks [arxiv:2306.14795 §3.3]. It does not beat T2M-GPT on text-to-motion FID [arxiv:2306.14795 tab. 3], so choose it for breadth, not for the best single metric | | Motion driven by a speaker's transcript, with a text-pretrained model | Motion codes interleaved with timed words, as in [Can Language Models Learn to Listen?](/papers/2308.10897) | Text pretraining lowers FD from 22.81 to 18.22 on the Trevor Noah listener [arxiv:2308.10897 tab. 2]. Interleaving adds little there, and audio and video are not used [§6.4] | | Image generation where raster decoding is too slow | Multi-scale codes with next-scale prediction, as in [VAR](/papers/2404.02905) | Ten prediction steps against one per token [arxiv:2404.02905 tab. 1]. FID 5.22 against 18.65 for raster order, at a similar size [arxiv:2404.02905 tab. 3] | | Text and images in one transformer stream, with reranking | Joint discrete VAE prior, as in [DALL-E](/papers/2102.12092) | Reranking 512 candidates keeps the best matches to the caption [arxiv:2102.12092 §2.6]. The transformer is not released, so an implementer writes it | | Goal reaching over long distances from recorded trajectories, with sparse goal rewards | Landmark planning, as in [QPHIL](/papers/2411.07760) | AntMaze-Ultra diverse 70.3 without stitching, against 52.6 for HIQL without representation [arxiv:2411.07760 tab. 1]. Needs the zone assumption [§4.2], and stitching matters on the extreme diverse map (39.5 against 12.5) [fig. 5] | | Text command to humanoid motion, with motion data retargeted to the robot | Action tokens and a text-conditioned transformer, as in [UH-1](/papers/2412.14172) | FID 0.445 against 0.582 for MDM on HumanoidML3D [arxiv:2412.14172 tab. 1]. Real-robot results cover one robot, the Unitree H1-2 [tab. 3] | **Sizing the codes.** More capacity helps reconstruction faster than it helps samples. In VideoGPT, the largest latent (16×32×32) gives the best reconstruction, but the 8×32×32 latent gives the best samples. The paper attributes this to limited compute for the prior [arxiv:2104.10157 tab. 5] [arxiv:2104.10157 §4.7]. MoMask shows the same pattern with residual layers [arxiv:2312.00063 tab. 2]. UH-1 shows the same direction for text-to-motion: FID falls from 0.539 with 512 codes to 0.463 with 2048 [arxiv:2412.14172 §5.4]. Pick the size by the generation metric, not the reconstruction metric. ## Implementation sketch The common core is the nearest-code lookup, the tokenizer loss, the codebook update, and a prior over the codes. Variants change only one of these. Lines are Python-flavoured pseudocode, not a port of any one repository. ```pseudocode # Nearest-code lookup with straight-through gradient [arxiv:2104.10157 §2.1] def quantize(latents, codebook): # Squared distance from each latent to every code (N, K) dist = pairwise_sq_distance(latents, codebook) indices = argmin(dist, axis=-1) codes = codebook[indices] # Forward pass uses the code; gradient skips the snap # [arxiv:2312.00063 §3.1] straight = latents + stop_gradient(codes - latents) return straight, indices ``` ```pseudocode # One tokenizer training step [arxiv:2104.10157 §2.1] [arxiv:2312.00063 §3.1] def tokenizer_step(x, encoder, decoder, codebook, beta): latents = encoder(x) straight, indices = quantize(latents, codebook) # Reconstruction term against the input [arxiv:2104.10157 §2.1] loss = reconstruction_error(decoder(straight), x) # Commitment: pull encoder outputs toward their codes. The weight beta # differs across papers, so take it from the card, not from this sketch loss = loss + beta * mse(latents, stop_gradient(codebook[indices])) return loss ``` ```pseudocode # Codebook update by moving average, with dead-code reset # [arxiv:2104.10157 §2.1] def ema_codebook_update(codebook, counts, sums, latents, indices, decay): counts = decay * counts + (1 - decay) * count_per_code(indices) sums = decay * sums + (1 - decay) * sum_per_code(latents, indices) codebook = sums / (counts[:, None] + eps) # Codes that stop being used are re-seeded from current latents [§4.1] dead = counts < threshold codebook = where(dead[:, None], random_latents(latents), codebook) return codebook, counts, sums ``` ```pseudocode # Prior over codes by masked decoding in rounds [arxiv:2212.05199 alg. 1] def masked_decode(prior, cond, n_codes, rounds): tokens = full(n_codes, MASK) for r in range(rounds): logits = prior(tokens, cond) sampled = sample(logits) free = tokens == MASK tokens = where(free, sampled, tokens) # Re-mask a cosine-shaped share of the least confident codes # [arxiv:2212.05199 §3.2] remask = cos(pi / 2 * (r + 1) / rounds) tokens = remask_lowest(tokens, logits, sampled, free, remask) return tokens ``` Where the variants differ: - **Residual stacks:** run `quantize` on the residual left by the previous layer and sum the codes [arxiv:2312.00063 §3.1]. - **Bit-wise:** replace the `argmin` with a sign per channel, and replace the codebook update with nothing [arxiv:2412.04431 §3.2]. - **Frozen codes:** skip `tokenizer_step` and the codebook update, and use the frozen encoder's indices as targets [arxiv:2209.00647 §3.1]. - **Codes as words:** give each code an id after the text vocabulary, and wrap each motion in start and end ids [arxiv:2306.14795 §3.2]. To interleave by time instead, place each word before the motion token whose window follows the word's end [arxiv:2308.10897 §3.3]. - **Autoregressive prior:** replace `masked_decode` with a sampler that writes one code at a time, conditioned on the codes already written [arxiv:2104.10157 §3]. - **Contrastive targets:** quantize the unmasked features with a Gumbel-softmax product quantiser and a diversity term, then score the masked positions with InfoNCE against negatives from other masked steps [arxiv:2205.10643 §3.3.2]. - **Diffusion:** keep the tokenizer, but train a denoiser on the latent instead of a prior over codes [arxiv:2212.04493 §3.2]. - **Post-training k-means:** skip `tokenizer_step` and the codebook update. After training, cluster each attribute group with k-means and store the indices and centres [arxiv:2406.17074 §4.3]. - **Multi-scale:** pool the residual to each scale before `quantize`, and subtract the upsampled reconstruction before the next scale [arxiv:2404.02905 alg. 1]. - **Product quantization:** split the vector into M sub-vectors and run a k-means per sub-vector. Store the M indices, and skip the training loss and the prior [arxiv:2401.08281 §4.1]. - **Landmarks:** quantize states, not outputs. Add the temporal triplet term to the tokenizer loss, then train the planner on the compressed token sequence, collapsing repeated tokens [arxiv:2411.07760 §4.1] [§4.2]. - **Action tokens:** encode each clip to one token per four frames, and train a causal transformer to predict the tokens given a text embedding, with an end token [arxiv:2412.14172 §4]. ## Contradictions No direct contradiction on the core claims. The following are the disagreements the cards support. They are not resolved here. - **Codebook size and generation.** VideoGPT's BAIR sweep shows FVD flat from 256 to 4096 codes [arxiv:2104.10157 tab. 6]. Infinity reports reconstruction rFID falling as the implicit vocabulary grows from 2^32 to 2^64 [arxiv:2412.04431 tab. 4]. The two measure different things on different data, so they do not settle whether more codes help generation. MAGVIT uses 1024 codes where TATS used 16384, and reports better FVD, but it changes the decoder too, so the codebook size is not isolated [arxiv:2212.05199 app. A.1]. - **Codes against pixels for colour.** Code targets win on colour LPIPS and segmentation, and pixel targets win on colour MSE [arxiv:2209.00647 tab. 1]. The metrics disagree about which target is better, so the question is open. - **Commitment weight.** Of the papers that use a commitment term, VideoGPT states β = 0.25 [arxiv:2104.10157 tab. 8], and the listener model states 0.02 for its embedding term [arxiv:2308.10897 app. B]. MoMask and SDFusion leave the weight unstated [arxiv:2312.00063 eq. 2] [arxiv:2212.04493 §3.1]. MotionGPT gives no weights in the paper, while its code uses 0.02 [arxiv:2306.14795 §3.1]. The listener paper writes its losses as sums, while its code averages them, so the same weight acts on a different scale [arxiv:2308.10897 §3.2]. The cards record code values that differ widely. Do not carry a value from one paper to another. - **Within-paper numbers.** MoMask reports HumanML3D FID 0.045 in its main table and 0.051 in its ablation table, and KIT-ML FID 0.204 and 0.228 [arxiv:2312.00063 tab. 1] [arxiv:2312.00063 tab. 2]. Cite the table you use. VAR's repository README lists 1.97 and 1.80 for its d30 rows, where Table 1 gives 1.92 and 1.73 [arxiv:2404.02905 tab. 1]. The VAR card explains the sampling settings. - **Text against table, large maze.** QPHIL's text says it outperforms all other methods on the larger maps [arxiv:2411.07760 §5.2]. In its Table 1, HIQL with representation scores 88.2 ± 5.3 on large diverse, above both QPHIL variants at 80.25 and 80.5 [arxiv:2411.07760 tab. 1]. The claim does not hold for that row. ## Papers - [Self-Supervised Speech Representation Learning: A Review](/papers/2205.10643): survey; quantized targets in wav2vec 2.0 and the VQ-VAE as a generative model. - [VideoGPT: Video Generation using VQ-VAE and Transformers](/papers/2104.10157): one codebook with an autoregressive prior over video codes. - [Visual Prompting via Image Inpainting](/papers/2209.00647): frozen VQGAN codes as masked-prediction targets for an MAE. - [Unified-IO: A unified model for vision, language, and multi-modal tasks](/papers/2206.08916): one large VQ-GAN codebook in a shared text-and-vision vocabulary. - [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](/papers/2212.04493): quantized 3D latent under latent diffusion. - [MAGVIT: Masked Generative Video Transformer](/papers/2212.05199): one 3D codebook with parallel masked decoding over video tokens. - [MoMask: Generative Masked Modeling of 3D Human Motions](/papers/2312.00063): residual vector quantization with masked and residual transformers. - [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](/papers/2412.04431): bit-wise quantization with a per-bit classifier and next-scale generation. - [Reducing the Memory Footprint of 3D Gaussian Splatting](/papers/2406.17074): post-training k-means codebooks for 3DGS attributes, with pruning and SH band reduction. - [Zero-Shot Text-to-Image Generation](/papers/2102.12092): discrete VAE with one 8192-entry codebook, feeding a joint text and image transformer. - [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](/papers/2404.02905): multi-scale codes from one shared codebook, predicted scale by scale. - [The Faiss library](/papers/2401.08281): product quantization with inverted files for approximate nearest-neighbour search. - [Muse: Text-To-Image Generation via Masked Generative Transformers](/papers/2301.00704): one VQGAN codebook with masked parallel decoding, conditioned on frozen T5-XXL text. - [MotionGPT: Human Motion as a Foreign Language](/papers/2306.14795): motion codes added to a Flan-T5 vocabulary, with instruction tuning across motion tasks. - [Can Language Models Learn to Listen?](/papers/2308.10897): motion codes from a VQ-VAE interleaved with timed words for a text-pretrained GPT-2. - [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](/papers/2411.07760): landmark codes from a VQ-VAE with a temporal triplet loss, planned by a transformer over offline policies. - [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172): 2048-entry action tokenizer under a text-conditioned transformer, for humanoid motion.