Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

In plain words

Grad-TTS turns written text into a mel-spectrogram, a picture of the sound's pitch bands over time, and a separate vocoder turns that picture into audio. Older fast systems either need a separate tool to say how long each sound lasts, or are slow and sometimes skip or repeat words.

Grad-TTS instead starts from noise shaped by the text and removes the noise in small steps, the same basic idea behind image diffusion models. A text encoder predicts how long each sound lasts, and the noise is centred on the spectrogram those durations imply. The number of denoising steps is a dial: fewer steps are faster, more steps sound better, and the gain levels off.

In the authors' listening tests on the LJSpeech voice, 10 steps came close to Tacotron2 in perceived quality and ran faster on a GPU. These numbers are the authors' own.

TL;DR

• Grad-TTS is a text-to-mel feature generator. Its decoder is a diffusion model whose noise is centred on text-aligned encoder outputs §3.1.
• Durations come from a duration predictor trained on alignments found by Monotonic Alignment Search (MAS). The decoder is trained with a λ_t-weighted noise-prediction loss §3.2 eq. 17.
• Sampling solves the decoder ODE with a first-order Euler scheme. The number of steps N is the speed/quality dial; the paper says ten steps are enough for high fidelity §4.3.
• Author-reported LJSpeech results: MOS 4.38 at N=10 vs 4.32 for Tacotron2 and 4.53 for ground truth; real-time factor 0.033 vs 0.075 for Tacotron2 tab. 2.
• Watch the sampling temperature, the positional-embedding scale and loss normalisation, where the paper and the released code differ or stay silent (see Pitfalls).

Problem

• Autoregressive feature generators (Tacotron2, Transformer-TTS) give natural speech but are computationally slow and suffer attention failures such as skipped or repeated words §1.
• Non-autoregressive models such as FastSpeech and Parallel Tacotron are faster and more robust but need durations from a pre-computed teacher alignment §1.
• Glow-TTS learns alignments without an external aligner using MAS, but its decoder is a normalising flow §1 §3.
• The paper states that before this work no diffusion-based feature generator was known, though diffusion vocoders (WaveGrad, DiffWave) existed §1.
• Assumptions in the experiments: one English female voice (LJSpeech, 22.05 kHz), phonemized input, 80-bin mel-spectrograms §4. The paper notes that the decoder needs long training before its quality is good §4.

Method

The model has three parts: an encoder, a duration predictor and a diffusion decoder §3. The encoder maps text to a sequence of 80-dimensional means. The duration predictor estimates how many frames each token spans. Those durations expand the token means to frame-level means, which become the centre of the noise the decoder starts from. The decoder is a score network trained by denoising inside a generalised diffusion framework, and it is sampled by solving an ODE with a few Euler steps fig. 2.

Generalised diffusion. The forward process drifts each frame toward a mean μ with a time-varying noise schedule β_t. Its marginal given the data is Gaussian §2.1 eq. 6. With the covariance set to the identity §3.1 §3.2, the noise variance is λ_t eq. 16. The reverse-time ODE is the sampler the paper uses, rather than the stochastic reverse SDE §3.1.

1. Forward marginal §2.1 eq. 4 eq. 16. A frame at time t is a mix of the data and the mean μ, plus Gaussian noise with variance λ_t.
2. Encoder §3.3. Text ids go through an embedding, a pre-net of three convolution layers with a projection, six Transformer blocks with relative attention, and a linear projection to 80 dimensions μ̃.
3. Duration predictor §3.2 eq. 15. It takes encoder features with gradients stopped and predicts the log of each token's frame count. Its target is the log count from the MAS alignment.
4. Alignment §3.2 eq. 14. MAS finds the monotonic, surjective alignment A* that maximises the likelihood of the mels under N(μ̃, I). Training alternates alignment search and a parameter update, and the paper states that without the encoder loss the model failed to learn alignment.
5. Frame means §3.1. Each frame j takes the mean of its token, μ_j = μ̃_A(j). These frame means are the terminal mean and an extra input channel of the score network §3.3.
6. Diffusion loss §3.2 eq. 17 eq. 18. Sample a time t, sample noise ξ, form X_t = ρ + √λ_t ξ, and minimise the λ_t-weighted squared error of the score prediction. The score network is conditioned on μ and t.
7. Sampling §3.1 eq. 13. Draw z = μ + noise / τ, then take N backward Euler steps of the ODE. Temperature τ is 1.5 in the experiments §4. A HiFi-GAN vocoder turns the mel into audio §4.
8. Speed control §3.1. The number of steps N and the step size are chosen after training. Durations can be multiplied by a factor to change speaking rate.

Key equations, cited as in the source:

dX_t = \tfrac{1}{2}\big(\mu - X_t - s_\theta(X_t, \mu, t)\big)\beta_t\,dt eq. 13

\lambda_t = 1 - \exp\Big(-\int_0^t \beta_s\,ds\Big) eq. 16

\mathcal{L}_{diff} = \mathbb{E}_{X_0, t}\Big[\lambda_t\, \mathbb{E}_{\xi_t}\big[\|s_\theta(X_t, \mu, t) + \xi_t/\sqrt{\lambda_t}\|_2^2\big]\Big] eq. 17

The noise schedule is linear, β(t) = β0 + (β1 − β0) t, with β0 = 0.05, β1 = 20 and T = 1 §4. The integral in eq. 16 is then 0.05 t + ½(20 − 0.05) t².

Training step, one batch (Python-flavoured, paper-level; the code uses masked means and some extra steps listed under Paper vs code):

def training_step(text_ids, mels, text_lengths, mel_lengths):
    # Encode text into per-token means and log durations  [§3.3]
    token_means, log_durations = encoder(text_ids, text_lengths)
    # Most likely monotonic alignment, by MAS  [§3.2] [eq. 14]
    alignment = monotonic_align_search(token_means, mels, mel_lengths)
    # Duration target: log of the frame count per token  [§3.2] [eq. 15]
    duration_targets = log(frames_per_token(alignment))
    duration_loss = mean((log_durations - duration_targets) ** 2)
    # Copy each token's mean onto its frames, mu_j = mu_A(j)  [§3.1]
    frame_means = expand(token_means, alignment)            # (B, 80, F)
    # Use a random 2 s window of frames for the decoder  [§4]
    mels, frame_means = random_window(mels, frame_means, frames=172)
    # Encoder loss: squared error of mels around the frame means  [§3.2]
    prior_loss = mean(0.5 * ((mels - frame_means) ** 2 + log(2 * pi)))
    # Diffusion time per utterance, uniform on (0, 1)  [§4]
    times = uniform(0.0, 1.0)                               # (B,)
    # Cumulative noise: integral of beta from 0 to t  [§4] [eq. 16]
    cumulative = beta_min * times + 0.5 * (beta_max - beta_min) * times**2
    keep = exp(-0.5 * cumulative)                           # (B,)
    noise_variance = 1 - exp(-cumulative)                   # lambda_t
    # Noisy frames from the closed-form marginal  [§3.2] [eq. 18]
    noise = normal(0, 1, shape=mels.shape)
    noisy = (keep * mels + (1 - keep) * frame_means
             + sqrt(noise_variance) * noise)
    # Score network sees noisy frames, the means and the time  [§3.3]
    scores = decoder(noisy, frame_means, times)
    # lambda_t-weighted noise-prediction loss  [§3.2] [eq. 17]
    diffusion_loss = mean((sqrt(noise_variance) * scores + noise) ** 2)
    return duration_loss + prior_loss + diffusion_loss

Sampling, with N Euler steps:

def synthesize(token_means, durations, steps, temperature):
    # Stretch token means to frames by the predicted durations  [§3.1]
    frame_means = expand(token_means, durations)            # (80, F)
    # Start from noise centred on the means, scaled by temperature  [§4]
    state = frame_means + normal(0, 1, shape=frame_means.shape) / temperature
    # Backward Euler steps from t = 1 towards t = 0  [§3.1] [eq. 13]
    step = 1.0 / steps
    for index in range(steps):
        # Score at the step midpoint  [code: Grad-TTS/model/diffusion.py#L259]
        time = 1.0 - (index + 0.5) * step
        beta = beta_min + (beta_max - beta_min) * time
        scores = decoder(state, frame_means, time)
        # ODE drift from eq. 13, then an Euler update  [§3.1] [eq. 13]
        drift = 0.5 * (frame_means - state - scores) * beta
        state = state - drift * step
    return state

Hyperparameters

Results

Author-reported. Setup: LJSpeech, about 24 hours of English female speech at 22.05 kHz, with about 500 test utterances under 10 s §4. The mel-spectrograms feed a HiFi-GAN vocoder for every model compared §4. Baselines are the official Glow-TTS implementation, an unofficial FastSpeech implementation, and Tacotron2 §4. MOS: 40 test sentences per model, each rated by 10 Amazon Mechanical Turk "Master" assessors on a 9-point scale from 1 to 5 in steps of 0.5, with 95% confidence intervals §4.1.

Main comparison tab. 2:

Notes on the table: the log-likelihood is given once for all Grad-TTS rows, and the step count used for it is not stated §4.2. The paper reports Grad-TTS as having about 15 M parameters in total §4.3.

Step-count ablation tab. 1, subjective preference of Grad-TTS-10 against a model trained to denoise from N(0, I). The table's Worse and Better columns refer to that model, read against baseline Grad-TTS-10 from the caption and §4.1. The number of raters and sentences for this test is not stated.

The authors report Grad-TTS-10 significantly better than this model at 10, 20 and 50 steps (p < 0.005, sign test) §4.1.

Other reported results:

• Going from 10 to 1000 steps gives almost no MOS gain, while going from 4 to 10 steps gives a clear gain. Grad-TTS-1000 falls 0.1 short of ground truth §4.1.
• Errors that listeners heard fig. 4: Glow-TTS was judged to stress words wrongly and to sound robotic in about a quarter of cases. The paper says Tacotron2 and Grad-TTS have similar drawbacks.
• Efficiency §4.3: Grad-TTS is real-time on GPU with fewer than 100 decoder steps (RTF < 0.37). With 10 steps it is about twice as fast as Tacotron2. Its speed versus text length is shown in fig. 5.
• Objective likelihood §4.2: average log-likelihood on 50 test sentences, computed with the instantaneous change-of-variables formula and Hutchinson's trace estimator, so the value carries a 95% interval. The paper says Grad-TTS scores above Glow-TTS despite Glow-TTS's larger decoder.
• End-to-end variant §4.4: a preliminary WaveGrad-style decoder conditioned on μ. Quality was "fair enough" and it was not in the listening test. Demo samples are linked from the paper.

Paper vs code

Code: Grad-TTS/ in the repository huawei-noah/Speech-Backbones, at commit 7782c7a223dd25df47b4538698ee9756774e2924 (MIT). The repository also contains other projects (DiffVC/, SPIRAL/), which this card does not use.

• Sampling temperature. The paper writes the terminal distribution as N(μ, τ⁻¹ I) §4, which means variance 1/τ. The code computes μ + ε/τ with ε standard normal tts.py, so the standard deviation is 1/τ and the variance 1/τ². At τ = 1.5 the two give standard deviations of about 0.82 and 0.67.
• Euler time points. The paper uses first-order Euler without fixing which time the score is evaluated at §3.1. The code evaluates it at the midpoint of each step, t = 1 − (i + ½)/N diffusion.py.
• Positional-embedding scale. The paper does not mention it. The code's default is pe_scale = 1000 params.py. The README says the ICML 2021 submission model corresponds to pe_scale = 1 README.md.
• Encoder loss normalisation. Eq. 14 is an unnormalised sum over frames and mel bins eq. 14. The code averages the same expression over valid frames and bins tts.py, so the encoder loss is scaled down by the number of frames times 80 relative to the paper's equation. The relative weight against the other losses therefore differs from the paper's text.
• Diffusion loss form. The code multiplies the score by √λ_t and compares it to −ξ, averaged over valid frames and bins diffusion.py. This is the λ_t-weighted loss of eq. 17 written in another form.
• Training schedule. The paper's list describes repeated passes of alignment search and then parameter updates §3.2. The code runs MAS per batch inside the loss with no gradient, then takes one Adam step on the sum of the three losses tts.py train.py.
• Gradient clipping. Not in the paper. The code clips the encoder and decoder gradient norms to 1 separately before each step train.py.
• Duration target. Eq. 15 uses the log of the count eq. 15. The code adds 1e-8 inside the log tts.py.
• Inference durations. The code takes the ceiling of the predicted durations and multiplies by a length scale, which defaults to 0.91 in the inference script tts.py inference.py. The paper gives only the general idea of scaling durations §3.1.
• Diffusion time sampling. The paper samples t uniformly on [0, T] §3.2. The code clamps the sample to [1e-5, 1 − 1e-5] diffusion.py.
• Real-time factor timing. The code's RTF timer wraps only the acoustic model, not the HiFi-GAN vocoder inference.py. The paper does not say whether its RTF includes the vocoder tab. 2.
• Likelihood code. The repository has no likelihood or Hutchinson-estimator code. A search of the Grad-TTS Python files and notebooks for "likelihood" and "Hutchinson" found nothing. The Table 2 log-likelihood cannot be reproduced from this release.
• Vocoder. The inference script loads a HiFi-GAN checkpoint from checkpts/ inference.py. The paper does not describe the vocoder's training; it is an external dependency.

Pitfalls

• Temperature has two readings. Paper notation N(μ, τ⁻¹ I) and the code's μ + ε/τ give different noise levels. An implementation that copies the code matches the released samples; one that copies the paper's variance does not. Decide, and state which.
• The positional-embedding scale changes the time embedding, so the same weights give different outputs at scale 1 and 1000. Match the scale of the checkpoint you load, as the README describes. The paper leaves the value unstated.
• Which time the score is evaluated at in each Euler step matters when N is small. The paper does not say. The code uses midpoints. With N = 4 the step is large, and the paper's MOS for N = 4 is much lower than for N = 10 tab. 2.
• The encoder loss has two scales, the paper's unnormalised sum and the code's per-frame mean. Changing the normalisation changes how strongly the alignment and encoder terms pull against the diffusion term, and so the learned durations and means.
• The paper says the encoder loss is required: without it the model did not learn alignment §3.2. Treat that as the authors' observation, not a tested claim in this card.
• Diffusion training converges slowly, and quality depends on how well the score is learned at every t, not just the average loss. The paper reports two models with similar loss but different quality §4. Training length is therefore a result variable, not a fixed setting.
• Durations are computed with a ceiling, and a length scale of 0.91 is the code's default. This changes speaking rate and total length compared with the paper's raw durations. Set it explicitly.
• Text frontend. The paper says phonemized input but not how. The code uses CMUdict with ARPAbet phonemes, falls back to character symbols for unknown words, and interleaves a blank token [code: Grad-TTS/text/init.py#L14-L19] data.py. A different frontend or symbol set produces different sequence ids and will not match the released checkpoints.
• Mel frontend. The paper gives only "80-dimensional". The code uses reflect-padded STFT with magnitudes of the form sqrt(|X|² + 1e-9), a mel filterbank, and natural-log compression with a 1e-5 floor meldataset.py. Using a different mel definition breaks compatibility with the HiFi-GAN vocoder.
• The MOS differences are small. Grad-TTS-10 (4.38 ± 0.06) and Tacotron2 (4.32 ± 0.07) overlap within their 95% intervals. The paper's claim is "competitive", not "better". The ten-step preference gains are on a different test than MOS.
• Speed numbers are author-measured, on a GPU the paper does not name in the efficiency section, and with timing details not stated. Use them as relative guidance, not as absolute benchmarks.
• Only one speaker is evaluated (LJSpeech). The code also has a multi-speaker branch, but the paper does not evaluate it.

Lineage

• Builds on: Glow-TTS (Kim et al., NeurIPS 2020) for the encoder, duration predictor and MAS §3.2 §3.3. The paper's bibliography gives no arXiv id for it. The score-based SDE framework of Song et al. §2. The U-Net from DDPM (Ho et al.) §3.3.
• Compared against: Glow-TTS, FastSpeech (unofficial implementation), Tacotron2 §4. Vocoder for all systems: HiFi-GAN §4.
• Related: WaveGrad and DiffWave as diffusion vocoders §1. The end-to-end variant builds on WaveGrad §4.4.
• Superseded by: not stated in this wiki.