SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

In plain words

Most language models read and write only text. The usual way to make a model talk is a relay: one model turns audio into text, the language model answers in text, and a third model turns that text back into sound. Each hand-off drops things like tone and emphasis, and the language model never receives the audio itself.

SpeechGPT skips the relay. It turns speech into a short sequence of discrete codes, something like words, by using a self-supervised speech model and a fixed set of cluster labels. These codes are added as new entries in the language model's vocabulary, so to the model, speech is just more tokens. Its training has three steps. First it learns to continue sequences of speech codes from unlabeled audio. Then it is tuned on instructions whose requests and answers are speech or text. Last, a small add-on teaches it to write its reasoning in text and then answer in speech.

The point is one model that can listen and talk, starting from a language model that already knows a lot, instead of several speech components glued together.

TL;DR

• SpeechGPT extends a LLaMA model with discrete speech units, so it takes speech in and produces speech out without ASR or TTS in the loop §4.1 §4.2.
• Training has three stages: unit-level pre-training, cross-modal instruction tuning on the SpeechInstruct data, and LoRA tuning on chain-of-modality data §4.2.
• The evidence is qualitative: a few cross-modal examples and ten spoken dialogue examples, judged by people. No benchmark metrics or baselines are reported §5.2 tab. 1 tab. 2.
• Before you implement, read "Paper vs code": the released code differs from the paper's stated hyperparameters in several places, and some of the paper's numbers are internally inconsistent.

Problem

Cascaded speech assistants chain an ASR model, an LLM and a TTS model. The LLM only generates text, and the speech and text representations are not aligned, so the LLM's knowledge does not carry over to speech. The cascade also loses paralinguistic cues such as emotion and prosody §1. Spoken language models built on discrete speech codes can synthesize speech, but they do not understand its content or follow human instructions §1 §2.

A second obstacle: LLMs consume and emit discrete tokens, while speech is a continuous signal. The paper treats discretisation as the bridge §1.

Assumptions: the cross-modal stage needs paired speech and transcripts, which the paper gets from English ASR corpora §3.1. It also needs instruction data with speech on at least one side. The paper builds that data itself §3.2.

Method

SpeechGPT has three components: a discrete unit extractor, the LLM, and a unit vocoder that turns units back into a waveform §4.1 fig. 2.

Speech to units §4.1. A HuBERT-style encoder maps the waveform to frame features. A k-means model maps each frame to the index of its nearest cluster centre, giving a sequence of indices in 0 to K-1. Adjacent duplicate indices are then removed, so the unit sequence is shorter than the frame sequence §4.1. The paper does not state K. The released tokenizer uses mHuBERT with 1000 clusters README.md and the released model has 1000 unit tokens ma_pretrain.py. The paper's footnote names mHuBERT as the tokenizer §3.1.

Vocabulary expansion §4.2 eq. 1 eq. 2. The vocabulary V grows by K unit tokens. The original embedding rows are copied, and the new rows are randomly initialised. The paper does not state the initialisation distribution.

# Expand the vocabulary with K unit tokens  [§4.2] [eq. 1]
def expand_vocabulary(vocab, embeddings, num_units):
    # The new vocabulary is the union of V and K unit tokens  [eq. 1]
    unit_tokens = [unit_token(k) for k in range(num_units)]
    new_vocab = vocab + unit_tokens
    # Random rows for the expanded table  [§4.2]
    new_embeddings = random_init(len(new_vocab), dim)  # (|V''|, d)
    # Keep the original rows as they were  [eq. 2]
    new_embeddings[: len(vocab)] = embeddings          # (|V|, d)
    return new_vocab, new_embeddings

Stage 1: modality-adaptation pre-training §4.2 eq. 3. The model is trained on unlabeled speech, converted to units, with the ordinary language modelling objective: predict each unit from the units before it. The paper uses LibriLight, about 60K hours of English audiobook speech §5.1. The paper's loss is written per utterance. The code instead packs the token streams into 1024-token blocks (see Pitfalls).

# Stage 1: predict each unit from the units before it  [§4.2] [eq. 3]
def stage1_loss(model, unit_corpus):
    # Unlabeled speech, already converted to units  [§5.1]
    total_nll = 0.0
    for units in unit_corpus:
        # Score each unit given the earlier units of the same utterance
        for i in range(len(units)):
            log_probs = model.log_next_token(units[:i])
            total_nll += -log_probs[units[i]]
    return total_nll

Stage 2: cross-modal instruction tuning §4.2 eq. 4. The stage-1 model is fine-tuned on a mix of the cross-modal SpeechInstruct data and the moss-002-sft text instruction data §4.2. Each sample is a prefix followed by a text. The loss covers only the text part, not the prefix. The learning rate and step count are in the Hyperparameters table, and the released code differs from them.

# Stage 2: loss on the text part of each sample  [§4.2] [eq. 4]
def stage2_loss(model, sample):
    # Each sample is a prefix followed by a text  [§4.2]
    prefix_length = len(sample.prefix_tokens)
    total_nll = 0.0
    # Score tokens after the prefix; the prefix only conditions  [eq. 4]
    for i in range(prefix_length, len(sample.tokens)):
        log_probs = model.log_next_token(sample.tokens[:i])
        total_nll += -log_probs[sample.tokens[i]]
    return total_nll

Stage 3: chain-of-modality instruction tuning §4.2. A LoRA adapter is added to the attention layers of the stage-2 model. Only the adapter is trained, with the stage-2 loss. The Chain-of-Modality data is described under Data below.

# Stage 3: LoRA adapters on attention, stage-2 loss  [§4.2]
def add_stage3_adapters(model):
    # Rank 8 and alpha 16  [app. D]
    # Released config adapts only the query and value projections
    # [code: speechgpt/scripts/com_sft.sh#L32-L35]
    for block in model.blocks:
        block.attention.query = add_lora(block.attention.query)
        block.attention.value = add_lora(block.attention.value)
    return model

Unit vocoder §4.1. A multi-speaker HiFi-GAN takes unit embeddings through transposed convolutions and dilated residual blocks. A speaker embedding is concatenated to every up-sampled frame. The discriminators follow the architecture of the work cited as polyak2021speech §4.1.

Inference format. Chain-of-modality answers use the tags [tq] (text instruction), [ta] (text answer) and [ua] (unit answer), and end with <eoa> app. C. The cross-modal instruction template is [Human]: {D}. This is input: {U}<eoh>.[SpeechGPT]: {T}<eos>. §3.1.

Data: SpeechInstruct

SpeechInstruct has two parts §3. Both are built by the authors. No separate license for the dataset is recorded on this card.

Cross-modal instruction §3.1. English ASR corpora (GigaSpeech, Common Voice, LibriSpeech) are converted to units with mHuBERT, with adjacent duplicates removed. This yields 9 million unit-text pairs §3.1. Each pair is turned into an ASR or TTS task with a probability p (value not stated). The task wording comes from a pool of 100 instructions per task, generated by GPT-4 with a zero-shot prompt §3.1 app. A app. B. Samples are chained into multi-turn conversations up to the maximum input length §3.1.

Chain-of-modality instruction §3.2. A Transformer encoder-decoder text-to-unit generator is trained on LibriSpeech unit-text pairs §3.2. It converts 37,969 moss-002-sft samples with responses shorter than 35 words into speech. This gives quadruplets of speech instruction, text instruction, text response and speech response §3.2. Four input-output formats are built from them, each with a template in app. C.

Hyperparameters

Values as stated in the paper unless marked code. "Not stated" means neither the paper nor the released scripts give it.

Results

Author-reported. Setup: human judgement of model outputs on cross-modal instruction following and spoken dialogue. The paper does not state the number of raters, the protocol or the number of cases §5.1. Baselines: none reported.

The paper concludes that SpeechGPT follows multi-modal instructions and holds spoken dialogue §7. The examples in §5.2 are the only support given. No quantitative claim is made.

Paper vs code

Official code: https://github.com/0nutation/SpeechGPT (Apache-2.0), at commit b290011570431cc559546f841335170a9619fa72. The repository is linked from the paper's author block, which names it as the code link.

• Backbone size. Paper: LLaMA-13B §5.1, and the trainable-parameter row for stages 1 and 2 reads 13B app. D. Code: the stage-1 script loads llama/hf/7B ma_pretrain.sh, the README asks for LLaMA 7B README.md, and the released checkpoints are 7B README.md. The repo does not show which size produced the paper's results.
• Stage 1 length. Paper: 900 steps §5.1. Code: 3 epochs [code: speechgpt/scripts/ma_pretrain.sh#L45]. The step count is not recoverable from the script without the dataset size.
• Stage 2 length. Paper: 2100 steps in the text §5.1 and 4000 steps in the table app. D. Code: 3 epochs cm_sft.sh.
• Stage 3 length. Paper: 4200 steps §5.1 app. D. Code: 300 epochs com_sft.sh.
• Learning rate. Paper: 2e-4 peak for all three stages app. D. Code: 2e-5 for stages 2 and 3 cm_sft.sh [code: speechgpt/scripts/com_sft.sh#L50]. Stage 1 sets none, so the Hugging Face default applies ma_pretrain.sh.
• Batch size. Paper: 768, 1536 and 128 for stages 1, 2 and 3 §5.1. The scripts give a per-node batch of 3 × 8 × 8 = 192 [code: speechgpt/scripts/ma_pretrain.sh#L43-L44], 6 × 12 × 8 = 576 [code: speechgpt/scripts/cm_sft.sh#L39-L41] and 12 × 4 × 8 = 384 [code: speechgpt/scripts/com_sft.sh#L43-L45] on 8-GPU nodes. 768 is four such nodes, not the 96 GPUs stated. 1536 is not a multiple of 576. 128 is smaller than one node's 384.
• Stage 2 data. Paper: cross-modal data mixed with moss-002-sft §4.2. Code: the README downloads only the cross-modal file for stage 2 [code: speechgpt/README.md#L184], and the stage-2 script reads only that file [code: speechgpt/scripts/cm_sft.sh#L30].
• Stage 2 loss mask. Paper: the prefix is excluded and only the text part is trained §4.2 eq. 4. Code: only the prefix field is masked. In the released data this field is the system instruction [code: speechgpt/README.md#L51]. The human turn, including its unit tokens, stays in the loss cm_sft.py. The flag train_on_inputs is declared cm_sft.py and set in the scripts, but nothing reads it.
• Unit delimiters. Paper: the unit sequence has no delimiters in the template §3.1. Code: units are wrapped in <sosp> and <eosp> [code: speechgpt/utils/speech2unit/speech2unit.py#L169-L170], and both are added as tokens ma_pretrain.py.
• End-of-response marker. Paper: cross-modal answers end with <eos> §3.1. Released sample: units end with <eosp><eoa> [code: speechgpt/README.md#L52]. Only <eos> is added as a token in the stage-2 script cm_sft.py. <eoa> is not added as a token, so it is tokenized as ordinary text. Inference parses <eoa> from the decoded string cli_infer.py.
• Decoding. Paper: temperature 0.8, top-k 60, top-p 0.8, max length 2048 §5.1. Code: the defaults are these, except 1024 new tokens [code: speechgpt/src/infer/cli_infer.py#L28-L35]. But the generation call uses temperature 0.7, top-k 50 and 2048 new tokens [code: speechgpt/src/infer/cli_infer.py#L158-L165]. The defaults dictionary is assigned to an attribute cli_infer.py and never passed to generate.
• Vocoder. Paper: the authors train a multi-speaker unit HiFi-GAN §4.1. Code: the vocoder script downloads a fairseq checkpoint whose name ends in _lj vocoder.sh. No vocoder training code is in the repository. The generator uses speaker sampling only if the config lists several speakers [code: speechgpt/utils/vocoder/generate_waveform_from_code.py#L53-L55]. Which vocoder the paper's samples used cannot be settled from the repository.
• Stage 3 precision. Paper: not stated. Code: the base model is loaded in 8-bit and prepared for int8 training [code: speechgpt/src/train/com_sft.py#L195-L202], then LoRA is applied [code: speechgpt/src/train/com_sft.py#L204-L212].
• Stage 1 loss unit. Paper: a loss per utterance §4.2 eq. 3. Code: tokenizes each line, concatenates all tokens, cuts 1024-token blocks, and sets labels to the input ids ma_pretrain.py ma_pretrain.py. Loss is computed on every token in each block.

The stage order and the unit pipeline (mHuBERT units, duplicates removed, three training scripts in sequence) match the paper.

Pitfalls

• K is not stated. Changing the number of clusters, or the layer of the encoder, changes the unit vocabulary, so every later stage must be retrained. The released setup uses 1000 clusters from mHuBERT, layer 11 [code: speechgpt/utils/speech2unit/speech2unit.py#L125].
• Duplicates are removed, so durations are lost. Adjacent duplicates are dropped in the training data §3.1 and the speech output uses the vocoder's duration prediction vocoder.sh. Timing and emphasis are therefore not controlled by the LLM. The limitations section says paralinguistic information is not modelled §6.
• Long samples are truncated silently. Stage 2 caps sequences at 512 tokens app. D. The code truncates at model_max_length [code: speechgpt/src/train/cm_sft.py#L184-L185]. The paper concatenates instructions to the input limit §3.1. How truncation of a long speech turn should be handled is not stated.
• Chain-of-modality speech is synthetic. Speech responses come from a text-to-unit generator trained on LibriSpeech pairs, and the instructions come from moss-002-sft §3.2. The generator's quality is not reported.
• Task descriptions are generated. The 100 wordings per task come from GPT-4 §3.1. The mixing probability p between ASR and TTS is not stated.
• Full fine-tuning is expensive. Stages 1 and 2 train all parameters of a 13B (or 7B) model app. D ma_pretrain.py, sharded with FSDP across nodes ma_pretrain.sh.
• Evaluation is thin. The paper reports examples, not metrics, and it does not describe how the examples were chosen §5.2 tab. 1 tab. 2.
• The authors' own caveat. The README says the released performance is not optimal, and that the data and step counts were not increased for release README.md.
• Known limits from the paper. No control over emotion or tone §6; a text answer must be produced before the speech answer §6; no multi-turn support, because of the context limit §6.

Lineage

• Builds on: LLaMA as the backbone §4.1; HuBERT and mHuBERT for discrete speech units §4.1 §3.1; LoRA for stage 3 §4.2; a HiFi-GAN style vocoder with the architecture of the work cited as polyak2021speech §4.1.
• Compared against, in the related-work section: cascaded systems HuggingGPT and AudioGPT §2; generative spoken language models AudioLM and VALL-E §2; multimodal LLMs PaLM-E and LLaVA §2. No quantitative comparison is run §5.