Monotonic alignment search

Overview

Text-to-speech models must know how many output frames each input token covers. Monotonic alignment search (MAS) is a way to learn that alignment during training. Given the encoder's per-token means and the target mel-spectrogram, it returns the monotonic, surjective mapping from frames to tokens that makes the mel frames most likely under Gaussians centred on those means 2105.06337 §3.2 2105.06337 eq. 14. The frame counts per token then serve as duration targets for a duration predictor 2105.06337 §3.2.

The only source card here is Grad-TTS, which uses MAS in this way. The card attributes the use of MAS for this purpose to Glow-TTS (Kim et al., NeurIPS 2020) 2105.06337 §1. That paper has no card on this wiki, so it is named but not described.

Variants

Only one variant is documented here.

MAS with a Gaussian likelihood, as in Grad-TTS

The objective is the log-likelihood of each mel frame under a unit-variance Gaussian centred on the mean of the token assigned to that frame. The search keeps the alignment that maximises the total. Training alternates between searching the alignment and updating the parameters. The encoder loss is included, and the authors report that without it the model did not learn an alignment 2105.06337 §3.2. Card: Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech.

When to use which

The source card supports only one comparison: a model that predicts durations without an alignment search needs durations from a pre-computed teacher alignment, whereas MAS learns the alignment jointly with the model 2105.06337 §1. Beyond that, no constraint-based guidance can be drawn from the cards on this wiki.

Implementation sketch

The card does not describe the search procedure itself, so the function monotonic_align_search below is a placeholder. Only its inputs, outputs and the way the alignment is used are taken from the source.

def train_step_alignment(token_means, mels, mel_lengths):
    # Likelihood of each frame under each token's mean, unit variance  [§3.2]
    # token_means: (tokens, 80); mels: (80, frames)
    # Search the monotonic, surjective alignment with the highest total
    # log-likelihood  [§3.2] [eq. 14]
    alignment = monotonic_align_search(token_means, mels, mel_lengths)
    # Frames per token: duration targets for the duration predictor  [§3.2]
    frames_per_token = count_frames_per_token(alignment)
    return alignment, frames_per_token

Points where variants differ, none of which are established by a second card: the normalisation of the alignment term in the loss, and whether the encoder loss is included. See Contradictions.

Contradictions

The Grad-TTS card records a difference between the paper and the released code for the encoder loss that the alignment search uses. The paper's equation 14 is an unnormalised sum over frames and mel bins 2105.06337 eq. 14. The released code averages the same expression over valid frames and bins, which scales the term down relative to the paper's equation tts.py. The two readings weight the alignment term differently against the other losses. The card does not resolve which is intended.

Papers

• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech: uses MAS to learn alignments for a diffusion-based text-to-speech model and sets the encoder loss that the alignment depends on 2105.06337 §3.2.