Vector quantization

Overview

Vector quantization (VQ) replaces each continuous latent vector with the index of its nearest entry in a learned codebook. The result is a grid of integers. A second model can then predict those integers with the same cross-entropy tools used for text, which is why several of the papers below build a language-model-style prior over codes. Most variants here are the quantizer inside generative tokenizers. Two variants use the same lookup for storage instead: post-training k-means shrinks a trained 3D scene, and product quantization shrinks a vector index for search. One uses the codes as a state abstraction for planning: QPHIL maps states to landmarks, and a planner chooses a route through them. One uses the codes only as targets for a contrastive loss, not as a generative vocabulary 2205.10643 §3.3.2. This page does not cover model-weight quantization, which is covered by post-training quantization.

The shared core is an encoder, a snap to the nearest code, a decoder, and a loss whose main parts are a reconstruction term and a term that pulls encoder outputs toward their codes. Some variants add a third term, as the landmark tokenizer does 2411.07760 §4.1. The snap has no gradient, so a straight-through estimator passes the gradient across it 2312.00063 §3.1 2104.10157 §2.1. In a generative tokenizer, the invariant every variant keeps is that a decoder must rebuild the input from the codes. The quality of the tokenizer therefore bounds the quality of anything generated from its codes 2212.05199 §3.1 2206.08916 §4.6.

The variants differ on three choices. The first is how many quantizers stack: one codebook, several residual layers, one shared codebook applied at several scales, or bits with no lookup at all. The second is how codes are written and predicted: in raster order, in parallel masked rounds, or scale by scale. The third is whether the quantizer is trained at all, or frozen from a pretrained tokenizer. A fourth is what the codes are for: tokens to generate, landmarks to plan with, or stored attributes.

Variants

One codebook, autoregressive prior

VideoGPT trains a 3D-convolutional VQ-VAE with axial attention. It maps a clip to a grid of codes, then trains a GPT-style transformer prior over those codes 2104.10157 §3. Codebooks have 512 or 1024 entries, with embedding size 64 or 256 depending on the dataset 2104.10157 tab. 8. On BAIR, the codebook sweep shows FVD flat at about 104 from 256 to 4096 codes, while reconstruction (R-FVD) improves from 18.2 to 11.3 2104.10157 tab. 6. Reported results are FVD 103.3 on BAIR 2104.10157 tab. 1 and Inception Score 24.69 on UCF-101 2104.10157 tab. 2. A separate ablation splits the encoder channels across several codebooks. That is not residual stacking, and FVD worsens across its rows 2104.10157 tab. 7. Card: VideoGPT: Video Generation using VQ-VAE and Transformers.

One codebook, parallel masked decoding

MAGVIT tokenizes a 16-frame 128×128 clip into 4×16×16 codes from a 1024-entry codebook 2212.05199 §3.1 2212.05199 app. B.2. Its tokenizer is initialised by inflating a 2D VQ trained on ImageNet 2212.05199 §3.1. A bidirectional transformer learns masked token prediction. Generation fills the codes in 12 rounds, keeps the most confident guesses, and re-masks the rest 2212.05199 §3.2 2212.05199 alg. 1. On BAIR frame prediction, 12 masked rounds over 1024 tokens reach FVD 76, against 91 for autoregressive decoding over the same tokens in 1024 steps 2212.05199 §4.5. The paper reports the autoregressive TATS baseline as about 60 times slower at inference 2212.05199 §4.3. Card: MAGVIT: Masked Generative Video Transformer.

One codebook, masked decoding with a frozen text encoder

Muse uses the same kind of masked prediction for text-to-image. A VQGAN with downsampling factor 16 turns a 256×256 image into 16×16 tokens from an 8192-entry codebook. A second VQGAN with factor 8 gives 64×64 tokens for the 512×512 stage 2301.00704 §2.2 tab. 6. A frozen T5-XXL encoder supplies the text condition, and a transformer predicts the masked tokens with cross-entropy at the masked positions 2301.00704 §2.1 §2.3. Training draws a masking rate per example from an arccosine density, with mean about 0.64 2301.00704 §2.6. Sampling starts fully masked and commits the most confident tokens at each step, so 256 base tokens take 24 steps 2301.00704 §2.8. The 3B model reports zero-shot COCO FID 7.88 2301.00704 tab. 2. No code or weights were released. The paper does not give the cosine formula or the temperature behind its headline results, so an implementation has to choose them 2301.00704 §2.8 §5. Card: Muse: Text-To-Image Generation via Masked Generative Transformers.

One codebook, joint text and image stream

DALL-E trains a discrete VAE with one codebook of 8192 entries. It maps each 256×256 image to a 32×32 grid of codes, so 1024 image tokens 2102.12092 §2.1. A 12-billion-parameter transformer then reads the caption tokens followed by the image tokens as one stream [arxiv:2102.12092 §2.2]. When the transformer is trained, each image becomes tokens by taking the argmax over the encoder's logits, not by sampling. The authors note that the bound strictly calls for sampling 2102.12092 §2.2. Sampling draws 512 candidate grids and keeps the top k by a contrastive reranker 2102.12092 §2.6. Reported results are author-reported: 90% of raters preferred its realism over DF-GAN on MS-COCO 2102.12092 §3.1. The transformer is not released, only the dVAE (see the card). Card: Zero-Shot Text-to-Image Generation.

Residual stacks of codebooks

MoMask quantizes each motion with a residual VQ-VAE. Each of six layers quantizes what the layers below it missed, and the decoder sums the layer codes 2312.00063 §3.1 2312.00063 eq. 1. Each layer has a 512-entry codebook 2312.00063 §4. Quantization dropout switches off the last few layers with probability q during training, so the early layers carry more of the motion 2312.00063 §3.1. A masked transformer models the base layer, and a residual transformer predicts each finer layer conditioned on the layers below 2312.00063 §3.2 2312.00063 §3.3.

The ablation shows a depth trade-off. Reconstruction FID on HumanML3D falls from 0.091 with no residual layers to 0.014 with six or seven 2312.00063 tab. 2. Generation FID is best at five layers (0.051) and worse at six and seven (0.076 and 0.084) 2312.00063 tab. 2. The paper says generation quality starts to degrade above five layers [arxiv:2312.00063 §4.2]. Quantization dropout at q = 0.2 gives generation FID 0.051, against 0.091 at q = 0 2312.00063 tab. 2. Card: MoMask: Generative Masked Modeling of 3D Human Motions.

Multi-scale codes from one shared codebook

VAR quantizes an image at ten scales, from 1×1 up to 16×16, with one shared codebook of 4096 entries 2404.02905 §3.2 2404.02905 §4. At each scale the residual is pooled to that size, the nearest codes are found, and their reconstruction is subtracted before the next scale [arxiv:2404.02905 alg. 1]. A GPT-style transformer predicts each whole token map in one pass, conditioned on the coarser maps 2404.02905 §3.2. The paper's ablation moves from raster order to next-scale order at a similar size, and FID falls from 18.65 to 5.22 2404.02905 tab. 3. The 2.0B model reports FID 1.92, or 1.73 with rejection sampling, against 18.65 for the VQGAN raster baseline 2404.02905 tab. 1. Card: Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.

Bit-wise quantization without a codebook

Infinity has no codebook lookup. Each channel is reduced to a sign and the code is normalised, so the vocabulary is 2^d without being stored. The text-to-image model uses d = 32 2412.04431 §3.2 2412.04431 §4.5. The output head predicts each bit as a two-way decision, not a softmax over the whole vocabulary. The paper's example is that a 2^32-way classifier at width 2048 would need about 8.8 trillion weights 2412.04431 §3.3. On a 2^16 vocabulary, the bit-wise head reaches FID 3.83, against 4.49 for a conventional head 2412.04431 tab. 5. On ImageNet 256 with stride 16, reconstruction rFID is 0.61 at 2^32 and 0.33 at 2^64, against 0.87 for a continuous SD VAE 2412.04431 tab. 4. The paper also flips training bits (bit-wise self-correction), which lowers FID to 3.48 from 9.76 with teacher forcing 2412.04431 tab. 7. That is a training change, not a quantizer change. The entropy-penalty weight is not stated 2412.04431 §3.2. Card: Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.

Large shared codebook inside a multi-task sequence model

Unified-IO encodes images as 256 codes from an ImageNet-pretrained VQ-GAN. The VQ-GAN has compression ratio 16 and a 16384-entry codebook, and its codes join the shared text vocabulary 2206.08916 §3.1 2206.08916 §3.2. Generated code sequences are decoded back to images by the same VQ-GAN decoder 2206.08916 §3.1. The authors say the fixed tokenizer limits image generation and depth and normal quality, and that a stronger tokenizer would help 2206.08916 §4.1 2206.08916 §4.6. Card: Unified-IO: A unified model for vision, language, and multi-modal tasks.

Motion codes as words in a language-model vocabulary

MotionGPT follows the Unified-IO pattern for motion. A VQ-VAE with 512 codes of dimension 512 encodes a motion clip, with time downsampled by 4 2306.14795 §3.1 §4.1. Each code becomes a word in the vocabulary of a Flan-T5 encoder-decoder, and start and end tokens wrap each motion 2306.14795 §3.2. The paper names EMA updates and codebook reset for codebook use 2306.14795 §3.1. Training has three stages: the tokenizer, span corruption over text and motion, then instruction tuning on prompt templates 2306.14795 §3.3. One model covers text-to-motion, motion-to-text, prediction and in-between 2306.14795 tab. 3 tab. 4 tab. 5. The author-reported text-to-motion FID on HumanML3D is 0.232 for the task-tuned model, while T2M-GPT reports 0.116 in the same table 2306.14795 tab. 3. The authors describe the results as competitive on most metrics, not best on all 2306.14795 §4.2. Card: MotionGPT: Human Motion as a Foreign Language.

Motion codes interleaved with timed words

The listener model uses the same kind of motion tokenizer, with 256 codes and one token per 8 frames 2308.10897 §3.2 2308.10897 app. B. The difference is placement. Each word goes before the motion token whose 8-frame window follows the word's end time, so text and motion line up in time 2308.10897 §3.3. A GPT-2 Medium, initialised from text-only pretraining, predicts the next motion token. Motion tokens are new vocabulary entries with random initial embeddings, and the output layer is replaced by a projection onto the 256 motion codes 2308.10897 §3.3. During training, input motion tokens are corrupted with probability 0.5 2308.10897 app. B.

Author-reported results on the Trevor Noah listener: the full model has FD 18.22, against 22.81 when the language model is not text-pretrained 2308.10897 tab. 2. The variant without interleaving reaches FD 19.03, and the authors say the gap to the full model is not significant 2308.10897 tab. 2 §6.1. In a human A/B test, listeners preferred the method over nearest-neighbour retrieval 70.1% of the time [arxiv:2308.10897 §5.2]. The model uses the speaker's transcript only, not audio or video, so reactions cued by prosody or the speaker's face are not modelled 2308.10897 §6.4. Card: Can Language Models Learn to Listen?.

Frozen codes as masked-prediction targets

MAE-VQGAN keeps a VQGAN tokenizer frozen and uses its codes as targets. A ViT-L masked autoencoder predicts a distribution over 1024 codes for each hidden patch, trained with cross-entropy 2209.00647 §3.1. The predicted codes are decoded by the frozen VQGAN decoder 2209.00647 §3.1. Trained on arXiv figures, code targets beat pixel targets on segmentation (27.83 against 17.42 on split 0) and on colour LPIPS (0.40 against 0.55). Pixel targets are better on colour MSE (0.43 against 0.67) 2209.00647 tab. 1. The ImageNet-only version of the same model is far lower 2209.00647 tab. 1. Card: Visual Prompting via Image Inpainting.

Quantized targets for contrastive speech pretraining

wav2vec 2.0 masks the convolutional features of speech before a Transformer. The unmasked features are quantised into discrete targets, using a Gumbel-softmax quantiser over several codebooks at once (product quantisation). A diversity term encourages every codebook entry to be used 2205.10643 §3.3.2. The loss is InfoNCE with cosine similarity. It is applied only at masked steps, and the negatives are other masked steps 2205.10643 §3.3.2. The review does not give the number of groups or entries 2205.10643 §3.3.2. The codes serve only as targets, so this variant has no prior over codes to sample from.

HuBERT is a different case. Its targets are offline k-means clusters, with no learned quantiser inside the model. Its cluster count is not stated 2205.10643 §3.4.2. Card: Self-Supervised Speech Representation Learning: A Review, which covers both models as a survey and reports no new method.

Quantized latent under diffusion

SDFusion compresses truncated signed-distance grids with a 3D VQ-VAE, then runs a conditional diffusion model in that latent space 2212.04493 §3.1 2212.04493 §3.2. The authors say that diffusion on a high-resolution 3D grid is too expensive, which is the reason for the latent stage 2212.04493 §3. The main text states neither the latent dimension nor the codebook size 2212.04493 §3.1, and the paper gives no weight for the VQ and commitment losses 2212.04493 §3.1. Reported results are a lower UHD than AutoSDF on ShapeNet completion, 0.0557 against 0.0567 2212.04493 tab. 1, and Pix3D chamfer distance 1.852 [arxiv:2212.04493 tab. 2]. Card: SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation.

Learned landmarks for hierarchical planning

QPHIL quantizes states, not generated outputs. A VQ-VAE maps each state to one of a small set of landmark tokens. A transformer plans a sequence of landmarks toward a goal, and two offline IQL policies follow it: one moves between landmarks, and a goal policy takes over for the exact goal once the plan is used up 2411.07760 §4 §4.3. The tokenizer adds a temporal triplet loss, so states close in time receive close latents 2411.07760 §4.1. Its planner is trained by teacher forcing on the points where the token changes, and the authors say conditioning on the full compressed history helps 2411.07760 §4.2. The paper's table lists codebook sizes from 24 to 96 by maze, but does not name the window or map its loss coefficients to the text's symbols 2411.07760 app. B.2. The method assumes the low-level policy can reach any state inside a landmark from any other state of it 2411.07760 §4.2. Stitching splices two trajectories that share a pair of landmarks, which adds plans the data does not contain 2411.07760 §4.2.

Author-reported on AntMaze-Ultra with diverse data, QPHIL without stitching reaches 70.3 ± 6.9, against 52.6 ± 8.7 for HIQL without representation 2411.07760 tab. 1. On the new AntMaze-Extreme map with diverse data, the stitched variant reaches 39.5 ± 13 and the unstitched one 12.5 ± 8.5, against 21.9 ± 6.6 for HIQL without representation 2411.07760 fig. 5. Card: Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning.

Action tokens for text-conditioned humanoid control

UH-1 uses a 1-D convolutional encoder-decoder with a vector-quantized bottleneck. Each token covers four frames, and the codebook has 2048 entries 2412.14172 §4 app. D.1. A causal transformer reads a CLIP embedding of the command and predicts the next token, with an end token to stop 2412.14172 §4 app. D.2. The paper does not state the tokenizer's loss weights or its optimiser 2412.14172 app. D.1.

Two control modes sit on top of the same tokens. In text-to-keypoint mode, the model emits keypoints and a goal-conditioned policy tracks them in closed loop. In text-to-action mode, the model emits actions for open-loop control, and the lower body then needs a separate locomotion policy 2412.14172 §4 §5.3. On HumanoidML3D, the author-reported FID is 0.445 against 0.582 for MDM and 0.667 for T2M-GPT 2412.14172 tab. 1. On a real Unitree H1-2, the authors report 70% to 100% success per command 2412.14172 tab. 3. Card: Learning from Massive Human Videos for Universal Humanoid Pose Control.

Post-training k-means codebooks for a stored representation

Not a generative tokenizer. Reduced 3D Gaussian Splatting trains the scene first, then clusters each attribute group, except position, into 256-entry k-means codebooks, and stores one byte per value 2406.17074 §4.3. The twenty groups are opacity, the three scale components (one shared codebook), the two rotation parts, base colour and 15 spherical-harmonic groups 2406.17074 §4.3. Positions and codebook entries are stored as 16-bit floats 2406.17074 §4.3. There is no prior over codes and no training through the quantizer, so the codebook only needs to be fitted once. Its effect is one part of a larger pipeline that also prunes Gaussians and drops spherical- harmonic bands. The paper reports a file of 29 MB against 744 MB for the 3DGS re-run on Mip-NeRF360, with PSNR 27.10 against 27.42 2406.17074 tab. 1. The paper's own ablation finds half-float storage without codebooks smaller than the k-means version, 29 MB against 38 MB on Mip-NeRF360, with PSNR 27.10 against 27.17 2406.17074 tab. 4. The reference code is under a non-commercial licence, so this page cites the paper only. Card: Reducing the Memory Footprint of 3D Gaussian Splatting.

Product quantization for search, not generation

Faiss compresses a vector set so that approximate nearest-neighbour search fits in memory. A single k-means codebook over the whole vector is the best fit for the error, but it is impractical beyond about 3-byte codes. So a product quantizer splits the vector into M sub-vectors, each with its own k-means codebook of K entries 2401.08281 §4.1. The notation PQ6x10 means six sub-vectors with 2^10 entries each 2401.08281 §4.1. Distances are estimated from the stored codes, usually by asymmetric distance computation, which compares the raw query with the decoded vectors 2401.08281 §4. Product quantization is paired with an inverted file: a coarse quantizer picks the nearest lists, and only those are scanned 2401.08281 §5.1. There is no prior over codes, so this variant does not generate anything. The paper's largest index, 1.5 trillion vectors, uses a different codec: PCA to 72 dimensions and 6-bit scalar quantization, with about 1 s per query, author- reported 2401.08281 §7.1. Card: The Faiss library.

When to use which

The choice depends on the decoding budget, the pretraining budget, and how much detail one codebook can carry.

Sizing the codes. More capacity helps reconstruction faster than it helps samples. In VideoGPT, the largest latent (16×32×32) gives the best reconstruction, but the 8×32×32 latent gives the best samples. The paper attributes this to limited compute for the prior 2104.10157 tab. 5 2104.10157 §4.7. MoMask shows the same pattern with residual layers 2312.00063 tab. 2. UH-1 shows the same direction for text-to-motion: FID falls from 0.539 with 512 codes to 0.463 with 2048 [arxiv:2412.14172 §5.4]. Pick the size by the generation metric, not the reconstruction metric.

Implementation sketch

The common core is the nearest-code lookup, the tokenizer loss, the codebook update, and a prior over the codes. Variants change only one of these. Lines are Python-flavoured pseudocode, not a port of any one repository.

# Nearest-code lookup with straight-through gradient  [arxiv:2104.10157 §2.1]
def quantize(latents, codebook):
    # Squared distance from each latent to every code  (N, K)
    dist = pairwise_sq_distance(latents, codebook)
    indices = argmin(dist, axis=-1)
    codes = codebook[indices]
    # Forward pass uses the code; gradient skips the snap
    # [arxiv:2312.00063 §3.1]
    straight = latents + stop_gradient(codes - latents)
    return straight, indices

# One tokenizer training step  [arxiv:2104.10157 §2.1] [arxiv:2312.00063 §3.1]
def tokenizer_step(x, encoder, decoder, codebook, beta):
    latents = encoder(x)
    straight, indices = quantize(latents, codebook)
    # Reconstruction term against the input  [arxiv:2104.10157 §2.1]
    loss = reconstruction_error(decoder(straight), x)
    # Commitment: pull encoder outputs toward their codes. The weight beta
    # differs across papers, so take it from the card, not from this sketch
    loss = loss + beta * mse(latents, stop_gradient(codebook[indices]))
    return loss

# Codebook update by moving average, with dead-code reset
# [arxiv:2104.10157 §2.1]
def ema_codebook_update(codebook, counts, sums, latents, indices, decay):
    counts = decay * counts + (1 - decay) * count_per_code(indices)
    sums = decay * sums + (1 - decay) * sum_per_code(latents, indices)
    codebook = sums / (counts[:, None] + eps)
    # Codes that stop being used are re-seeded from current latents  [§4.1]
    dead = counts < threshold
    codebook = where(dead[:, None], random_latents(latents), codebook)
    return codebook, counts, sums

# Prior over codes by masked decoding in rounds  [arxiv:2212.05199 alg. 1]
def masked_decode(prior, cond, n_codes, rounds):
    tokens = full(n_codes, MASK)
    for r in range(rounds):
        logits = prior(tokens, cond)
        sampled = sample(logits)
        free = tokens == MASK
        tokens = where(free, sampled, tokens)
        # Re-mask a cosine-shaped share of the least confident codes
        # [arxiv:2212.05199 §3.2]
        remask = cos(pi / 2 * (r + 1) / rounds)
        tokens = remask_lowest(tokens, logits, sampled, free, remask)
    return tokens

Where the variants differ:

• Residual stacks: run quantize on the residual left by the previous layer and sum the codes 2312.00063 §3.1.
• Bit-wise: replace the argmin with a sign per channel, and replace the codebook update with nothing 2412.04431 §3.2.
• Frozen codes: skip tokenizer_step and the codebook update, and use the frozen encoder's indices as targets 2209.00647 §3.1.
• Codes as words: give each code an id after the text vocabulary, and wrap each motion in start and end ids 2306.14795 §3.2. To interleave by time instead, place each word before the motion token whose window follows the word's end 2308.10897 §3.3.
• Autoregressive prior: replace masked_decode with a sampler that writes one code at a time, conditioned on the codes already written 2104.10157 §3.
• Contrastive targets: quantize the unmasked features with a Gumbel-softmax product quantiser and a diversity term, then score the masked positions with InfoNCE against negatives from other masked steps 2205.10643 §3.3.2.
• Diffusion: keep the tokenizer, but train a denoiser on the latent instead of a prior over codes 2212.04493 §3.2.
• Post-training k-means: skip tokenizer_step and the codebook update. After training, cluster each attribute group with k-means and store the indices and centres 2406.17074 §4.3.
• Multi-scale: pool the residual to each scale before quantize, and subtract the upsampled reconstruction before the next scale 2404.02905 alg. 1.
• Product quantization: split the vector into M sub-vectors and run a k-means per sub-vector. Store the M indices, and skip the training loss and the prior 2401.08281 §4.1.
• Landmarks: quantize states, not outputs. Add the temporal triplet term to the tokenizer loss, then train the planner on the compressed token sequence, collapsing repeated tokens 2411.07760 §4.1 §4.2.
• Action tokens: encode each clip to one token per four frames, and train a causal transformer to predict the tokens given a text embedding, with an end token 2412.14172 §4.

Contradictions

No direct contradiction on the core claims. The following are the disagreements the cards support. They are not resolved here.

• Codebook size and generation. VideoGPT's BAIR sweep shows FVD flat from 256 to 4096 codes 2104.10157 tab. 6. Infinity reports reconstruction rFID falling as the implicit vocabulary grows from 2^32 to 2^64 2412.04431 tab. 4. The two measure different things on different data, so they do not settle whether more codes help generation. MAGVIT uses 1024 codes where TATS used 16384, and reports better FVD, but it changes the decoder too, so the codebook size is not isolated 2212.05199 app. A.1.
• Codes against pixels for colour. Code targets win on colour LPIPS and segmentation, and pixel targets win on colour MSE 2209.00647 tab. 1. The metrics disagree about which target is better, so the question is open.
• Commitment weight. Of the papers that use a commitment term, VideoGPT states β = 0.25 2104.10157 tab. 8, and the listener model states 0.02 for its embedding term 2308.10897 app. B. MoMask and SDFusion leave the weight unstated 2312.00063 eq. 2 2212.04493 §3.1. MotionGPT gives no weights in the paper, while its code uses 0.02 2306.14795 §3.1. The listener paper writes its losses as sums, while its code averages them, so the same weight acts on a different scale 2308.10897 §3.2. The cards record code values that differ widely. Do not carry a value from one paper to another.
• Within-paper numbers. MoMask reports HumanML3D FID 0.045 in its main table and 0.051 in its ablation table, and KIT-ML FID 0.204 and 0.228 2312.00063 tab. 1 2312.00063 tab. 2. Cite the table you use. VAR's repository README lists 1.97 and 1.80 for its d30 rows, where Table 1 gives 1.92 and 1.73 2404.02905 tab. 1. The VAR card explains the sampling settings.
• Text against table, large maze. QPHIL's text says it outperforms all other methods on the larger maps 2411.07760 §5.2. In its Table 1, HIQL with representation scores 88.2 ± 5.3 on large diverse, above both QPHIL variants at 80.25 and 80.5 2411.07760 tab. 1. The claim does not hold for that row.

Papers

• Self-Supervised Speech Representation Learning: A Review: survey; quantized targets in wav2vec 2.0 and the VQ-VAE as a generative model.
• VideoGPT: Video Generation using VQ-VAE and Transformers: one codebook with an autoregressive prior over video codes.
• Visual Prompting via Image Inpainting: frozen VQGAN codes as masked-prediction targets for an MAE.
• Unified-IO: A unified model for vision, language, and multi-modal tasks: one large VQ-GAN codebook in a shared text-and-vision vocabulary.
• SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation: quantized 3D latent under latent diffusion.
• MAGVIT: Masked Generative Video Transformer: one 3D codebook with parallel masked decoding over video tokens.
• MoMask: Generative Masked Modeling of 3D Human Motions: residual vector quantization with masked and residual transformers.
• Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis: bit-wise quantization with a per-bit classifier and next-scale generation.
• Reducing the Memory Footprint of 3D Gaussian Splatting: post-training k-means codebooks for 3DGS attributes, with pruning and SH band reduction.
• Zero-Shot Text-to-Image Generation: discrete VAE with one 8192-entry codebook, feeding a joint text and image transformer.
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction: multi-scale codes from one shared codebook, predicted scale by scale.
• The Faiss library: product quantization with inverted files for approximate nearest-neighbour search.
• Muse: Text-To-Image Generation via Masked Generative Transformers: one VQGAN codebook with masked parallel decoding, conditioned on frozen T5-XXL text.
• MotionGPT: Human Motion as a Foreign Language: motion codes added to a Flan-T5 vocabulary, with instruction tuning across motion tasks.
• Can Language Models Learn to Listen?: motion codes from a VQ-VAE interleaved with timed words for a text-pretrained GPT-2.
• Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning: landmark codes from a VQ-VAE with a temporal triplet loss, planned by a transformer over offline policies.
• Learning from Massive Human Videos for Universal Humanoid Pose Control: 2048-entry action tokenizer under a text-conditioned transformer, for humanoid motion.