---
kind: concept
slug: discrete-speech-units
title: Discrete speech units
summary: >-
  Turns continuous speech into a short sequence of integer codes that a language model can read and write like text. Two routes are covered: k-means clusters over self-supervised speech features (the SpeechGPT tokenizer), and learned vector quantisers such as wav2vec 2.0's codebook. A survey of self-supervised speech describes both; no paper here compares them on the same task.
tags: [speech, tokenization, multimodal]
papers: ["2205.10643", "2305.11000"]
related: [vector-quantization, self-supervised-learning, k-means-clustering, multimodal-language-model]
status: draft
updated_at: 2026-11-01
---

## Overview

Language models consume and emit discrete tokens, while speech is a continuous
signal. Discrete speech units are the bridge: short stretches of audio are
mapped to one of a finite set of codes, so that speech can share a vocabulary
with text [arxiv:2305.11000 §1] [arxiv:2305.11000 §4.1]. The codes are usually
derived from a self-supervised speech encoder, whose families the survey
reviews [arxiv:2205.10643 §3].

The common core: an encoder produces frame features, each frame gets one index
from a finite set of size K, and the index sequence is what the model sees.
Two choices differ between variants. The first is how the index is picked: the
nearest k-means centre, or a learned quantiser. The second is whether repeated
neighbouring indices are merged. Once the codes are fixed, a language model is
trained on them with the ordinary next-token objective [arxiv:2305.11000 §4.2].
Changing the encoder, its layer or K changes the unit vocabulary, so every
later stage has to be redone [arxiv:2305.11000 §4.1].

## Variants

### k-means over self-supervised features (SpeechGPT)

A HuBERT-style encoder maps the waveform to frame features. A k-means model
assigns each frame to its nearest cluster centre, and the cluster index is the
unit. Adjacent duplicate indices are then removed, so the unit sequence is
shorter than the frame sequence [arxiv:2305.11000 §4.1]. The paper names mHuBERT
as the tokenizer [arxiv:2305.11000 §3.1]. The number of clusters K is not
stated in the paper. The released setup uses 1000 [code:
speechgpt/utils/speech2unit/README.md#L9], see Contradictions.
Card: [SpeechGPT](/papers/2305.11000).

### Offline k-means targets, re-clustered (HuBERT-style training targets)

Here the cluster index is not a vocabulary for a generative model. It is a
prediction target for masked frames. The survey describes a first pass with
k-means over MFCC features. A second pass clusters a hidden layer of the first
model to produce new targets. The survey reports that two such iterations are
enough to match or beat the previous state of the art for low-resource
recognition [arxiv:2205.10643 §3.4.2]. The survey does not describe these
targets being used as a language-model vocabulary. The HuBERT paper has no card
in this wiki, so this variant rests only on the survey's description.
Card: [Self-Supervised Speech Representation Learning: A Review](/papers/2205.10643).

### Learned codebook (vector quantisation)

The index comes from a quantiser that is trained together with the encoder,
not from a separate clustering step. The survey lists three examples.
VQ-VAE picks the nearest codebook vector for each encoder output, passes
gradients through the choice with a straight-through estimator, and adds a
codebook term and a commitment term [arxiv:2205.10643 §3.2.2]. wav2vec 2.0
quantises the unmasked features with a Gumbel-softmax quantiser, uses product
quantisation over several codebooks, and adds a term that rewards using every
entry [arxiv:2205.10643 §3.3.2]. Discrete BERT takes its units from a frozen
vq-wav2vec model and predicts masked units with cross-entropy
[arxiv:2205.10643 §3.4.2]. The survey notes that the wav2vec 2.0 quantiser
sits inside a pre-training objective, so its codes are a by-product of that
model [arxiv:2205.10643 §3.3.2].
Card: [Self-Supervised Speech Representation Learning: A Review](/papers/2205.10643).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You want one LLM to take speech in and produce speech out, without ASR and TTS in the loop | k-means units over HuBERT-style features, as in SpeechGPT | This is the paper's stated goal and its pipeline [arxiv:2305.11000 §1] [arxiv:2305.11000 §4.2] |
| You want discrete targets to pre-train a speech encoder by masked prediction | Offline k-means targets, re-clustered over iterations | The survey reports this family for low-resource recognition after two iterations [arxiv:2205.10643 §3.4.2] |
| You want the codes to come from a model you train end to end | Learned codebook | The quantiser is trained inside the pre-training objective [arxiv:2205.10643 §3.2.2] [arxiv:2205.10643 §3.3.2] |
| You have only unlabeled speech and a text-only base model | Stage 1 (unit language modelling) is possible from unlabeled audio | The paper pre-trains on unlabeled audiobook speech [arxiv:2305.11000 §4.2] [arxiv:2305.11000 §5.1] |
| You need paired speech-text data for the cross-modal stage | Plan for paired data | The cross-modal stage is built from ASR corpora with transcripts [arxiv:2305.11000 §3.1] |
| You need control over emotion, tone or timing | Not covered by these variants | The SpeechGPT paper says paralinguistic information is not modelled and emotion and tone are not controlled [arxiv:2305.11000 §6] |
| You cannot afford full fine-tuning of a 7B to 13B model | Not covered by the SpeechGPT variant | Stages 1 and 2 train all parameters, sharded across nodes [arxiv:2305.11000 app. D] |
| You need to change the unit vocabulary later | Plan a full retrain | Changing K or the encoder layer changes the unit vocabulary, so every later stage must be redone [arxiv:2305.11000 §4.1] |

## Implementation sketch

The common core is the unit extraction, then vocabulary expansion, then the
stage-1 loss. Units are integers in 0 to K-1. Variant-specific choices are the
encoder, K, the layer and the quantiser, none of which the SpeechGPT paper pins
down. The learned-codebook variant replaces the nearest-centre step.

```pseudocode
# Frame features from a self-supervised speech encoder  [arxiv:2305.11000 §4.1]
def speech_to_units(waveform, encoder, kmeans):
    frames = encoder(waveform)                 # (T, d)
    # k-means variant: nearest cluster centre per frame, index in 0..K-1
    # [arxiv:2305.11000 §4.1]. Learned-codebook variant: the index comes
    # from the trained quantiser instead  [arxiv:2205.10643 §3.3.2]
    indices = [kmeans.nearest(f) for f in frames]
    # Drop adjacent duplicates; durations are lost  [arxiv:2305.11000 §3.1]
    units = [indices[0]]
    for idx in indices[1:]:
        if idx != units[-1]:
            units.append(idx)
    return units                               # length <= T

# Add K unit tokens to the vocabulary  [arxiv:2305.11000 §4.2]
def expand_vocabulary(vocab, embeddings, num_units):
    new_vocab = vocab + [unit_token(k) for k in range(num_units)]
    # Original rows are kept; new rows are random  [arxiv:2305.11000 §4.2]
    new_emb = random_init(len(new_vocab), dim)
    new_emb[: len(vocab)] = embeddings
    return new_vocab, new_emb

# Stage 1: predict each unit from the units before it  [arxiv:2305.11000 §4.2]
def stage1_loss(model, units):
    nll = 0.0
    for i in range(len(units)):
        log_probs = model.log_next_token(units[:i])
        nll += -log_probs[units[i]]
    return nll
```

## Contradictions

- **Stage 2 length.** The paper gives 2100 steps in the text [arxiv:2305.11000 §5.1]
  and 4000 steps in its table [arxiv:2305.11000 app. D].
- **Backbone size.** The paper states LLaMA-13B [arxiv:2305.11000 §5.1]. The
  released stage-1 script loads a 7B model [code:
  speechgpt/scripts/ma_pretrain.sh#L3].
- **Learning rate.** The paper gives a peak of 2e-4 for all stages
  [arxiv:2305.11000 app. D]. The released stage-2 and stage-3 scripts use 2e-5
  [code: speechgpt/scripts/cm_sft.sh#L46] [code: speechgpt/scripts/com_sft.sh#L50].
- **Batch sizes.** The paper gives 96 A100 GPUs [arxiv:2305.11000 §5.1]. The
  per-node batch sizes in the scripts do not multiply out to the paper's
  stage batch sizes on that hardware [code: speechgpt/scripts/ma_pretrain.sh#L43-L44].

The two papers do not disagree about the discretisation itself. The survey
describes k-means targets for masked prediction, while SpeechGPT uses the
cluster index as a token vocabulary. The survey also reports numbers for
encoder pre-training on recognition and identification tasks, and SpeechGPT
reports no quantitative results for speech generation. So there are no
competing numbers for the same task.

## Papers

- [Self-Supervised Speech Representation Learning: A Review](/papers/2205.10643): a survey of self-supervised speech methods. It covers k-means targets (HuBERT-style), learned quantisers (VQ-VAE, wav2vec 2.0) and Discrete BERT on frozen vq-wav2vec units. It proposes no new method.
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](/papers/2305.11000): uses HuBERT-style units with k-means and duplicate removal, expands an LLM's vocabulary with them, and trains in three stages.