Gaussian mixture model

Overview

A GMM models a set of feature vectors as drawn from a mixture of several Gaussian components, each with its own mean and covariance. Five cards use a mixture, in different ways. WaveFake uses a GMM as a class-conditional density: one model is fit to the real speech features and one to the features of each generator. A clip is scored by the difference of the two log-likelihoods, which is a likelihood ratio 2111.02813 §4.1 2111.02813 eq. 5. The sign of this score separates the two classes, and the decision clips are then assessed by equal error rate 2111.02813 §4.1. The Kuzushiji paper uses a mixture of Gaussians as a conditional density, P(z_new | z_old), that maps a latent code from one domain to another 1812.01718 §3.2 1812.01718 alg. 1. The hypertime paper fits a standard Gaussian mixture, but over vectors that include the cosine and sine of time for each periodicity it finds, and uses the fitted model to predict p(a | x, t) 1810.04285 §3.5. The four uses share the Gaussian form but not the fitting or the scoring. Two cards use other mixture forms. Toto puts a mixture of Student-T components on a forecaster's output 2505.14766 §3.1. The LaserNet detector outputs a mixture of box distributions per point, trained with a Laplace-style loss 1903.08701 §3.5.

Variants

Diagonal-covariance GMM trained by gradient descent

The paper's stated method. Each component uses a diagonal covariance matrix. The paper uses 128 components, and 256 for the leave-one-out models. Training uses Adam with a learning rate of 1e-3, batch size 128 and 10 epochs 2111.02813 app. C. Card: WaveFake.

EM training (code only)

The released code also contains an expectation-maximisation training option. The paper does not describe it, so no numbers from the paper are tied to it 2111.02813 app. C. For the algorithm itself, see the expectation maximisation page. The released script's defaults differ from the paper's stated training, and the repository records no command that reproduces the paper's runs 2111.02813 app. C.

EM over time-projected vectors (hypertime)

Each measurement has a value a, a location x and a time t. For each periodicity T the vector gains a cosine and a sine of 2πt/T 1810.04285 eq. 1. A weighted sum of n multivariate Gaussians is then fit over these vectors 1810.04285 eq. 2 1810.04285 §3.5. Periodicities are added one at a time, each chosen by a spectral scan of the model error [arxiv:1810.04285 §3.7]. The loop stops when the training error no longer falls 1810.04285 §3.6. The paper gives two clustering routes. HyT-EM runs EM with a fixed number of clusters, through an OpenCV implementation. HyT-KM starts from k-means centres, uses cosine distance on the time coordinates, and chooses the number of clusters itself 1810.04285 §3.8. The text states no numbers. Figures show lower error than the baselines for the door and velocity experiments 1810.04285 §4.1 1810.04285 §4.3. As printed, eq. 7 has the absolute value inside the sum, so the score does not depend on the candidate period as the text describes 1810.04285 eq. 7. The card records no released hypertime code. Card: Warped Hypertime Representations for Long-term Autonomy of Mobile Robots.

Conditional mixture: mixture density network

A mixture density network (MDN) makes the mixture depend on an input. The Kuzushiji paper trains one with two hidden layers to model P(z_new | z_old) as a mixture of Gaussians, where both z vectors are 64-dimensional VAE latents 1812.01718 §3.2 1812.01718 alg. 1. The number of components, the hidden widths, the learning rate and the choice of training pairs are not stated 1812.01718 §3.2. The paper uses the MDN to draw a sampled latent for the stroke decoder, and reports no metric for this step; the evidence is qualitative examples 1812.01718 §3.2. Card: Deep Learning for Classical Japanese Literature.

Feature front end: LFCC or MFCC

The GMM is fed frame-level features, either linear-frequency cepstral coefficients (LFCC) or mel-frequency cepstral coefficients (MFCC). Both use 20 coefficients plus deltas and double deltas 2111.02813 §2.1 2111.02813 app. C. LFCC replaces the mel filterbank with a linear one and keeps the other steps 2111.02813 §2.1. The choice changes results in both directions (see the results table).

Student-T mixture head (not Gaussian)

Toto is a decoder-only forecaster for observability metrics. Its output head produces, for each token, K = 24 Student-T components and a weight for each 2505.14766 §3.1 2505.14766 app. A.6. The degrees of freedom are 2 plus a positive value, so the variance is finite [arxiv:2505.14766 app. A.3]. The mixture density is 2505.14766 eq. 3 and each component's density is 2505.14766 eq. 4. Training mixes the negative log-likelihood under the mixture 2505.14766 eq. 10 with a Cauchy-type robust loss on the mixture mean 2505.14766 eq. 11 2505.14766 §3.1. The components are Student-T, so their tails are heavier than a Gaussian's. The authors' ablation on held-out observability NLL replaces the mixture with a single Student-T output; the NLL rises by 27.2% 2505.14766 app. E. Card: This Time is Different.

Mixture of box distributions in a detector (not fitted as a GMM)

LaserNet works on the LiDAR range image. For each point, the network outputs class probabilities and, for each of K mixture components per class, a box with a log standard deviation, plus a mixture weight 1903.08701 §3.3. During training each point is matched to the one component whose mean is closest to its ground-truth box, and the box term is a Laplace-style loss rather than a Gaussian likelihood 1903.08701 §3.5 [arxiv:1903.08701 eq. 9]. At inference, predictions are merged by mean-shift clustering, and each cluster is fused as a product of Gaussians with precision weights 1/σ² 1903.08701 §3.4 1903.08701 eq. 6. The mixture width is K = 3 for vehicles and K = 1 for pedestrians and bikes on ATG4D [arxiv:1903.08701 §4.1]. The author-reported vehicle AP0.7 on ATG4D is 85.34, but on KITTI the method trails BEV detectors 1903.08701 tab. 1 1903.08701 tab. 4. Card: LaserNet.

When to use which

The authors describe the GMM as more robust and the neural RawNet2 as better on average 2111.02813 §6. The tables do not clearly support both claims at once; see Contradictions.

Implementation sketch

Common core for one GMM: extract frame features, fit one model per class, and score each clip by the log-likelihood ratio. Variants differ in the training loop (gradient descent or EM) and in the feature front end. The conditional variant (mixture density network) does not fit a model per class. It maps an input to the parameters of a mixture, then samples from it. The hypertime variant refits the mixture each time it adds a periodicity to the vectors; its loop is on the card 1810.04285 §3.4 1810.04285 §3.7.

# Frame features for one file  [arxiv:2111.02813 §2.1] [app. C]
# Input:  wav, mono waveform  (samples,)
# Output: features  (T, 60)  # 20 coefficients + delta + double delta

def extract_features(wav):
    # Resample to 16 kHz and remove long silences  [app. C]
    wav = remove_silence(resample(wav, 16_000))
    # Power spectrum, then 20 linear (LFCC) or mel (MFCC) bands  [§2.1]
    bands = filterbank(power_spectrum(wav), n_bands=20)  # (20, T)
    # Log compression, then DCT keeping 20 coefficients  [eq. 3]
    coeffs = dct(log(bands + 1e-6), n_coeffs=20)  # (20, T)
    # Central-difference deltas and double deltas  [eq. 4]
    deltas = delta(coeffs)
    double_deltas = delta(deltas)
    return concat([coeffs, deltas, double_deltas]).T

# Fit one diagonal-covariance GMM per class  [§4] [app. C]
# Input:  frames  (N, 60), components (128, or 256 for leave-one-out)
# Output: gmm with weights, means and variances

def fit_gmm(frames, components):
    # Paper: gradient descent with Adam, lr 1e-3, batch 128, 10 epochs
    # [app. C]. Alternative: EM, code only [app. C]
    ...

# Score one clip with a real and a fake GMM  [eq. 5] [§4.1]
# Input:  wav, real_gmm, fake_gmm
# Output: score (higher means more likely real)

def score_clip(wav, real_gmm, fake_gmm):
    frames = extract_features(wav)  # (T, 60)
    # The paper writes a sum over frames; the released code averages
    # per-frame log-likelihoods [eq. 5] [app. C]
    return log_likelihood(real_gmm, frames) - log_likelihood(fake_gmm, frames)

The window, hop and FFT size are not stated in the paper. They are not needed to implement the GMM itself, but they matter for reproducing the features.

# Sample from a mixture density network  [§3.2] [alg. 1]
# Input:  z_old (64,)  # latent from the old-domain VAE
# Output: z_new (64,)  # latent in the new domain

def sample_mdn(z_old, mdn):
    # Two hidden layers; widths not stated  [§3.2]
    h = mdn.hidden(z_old)
    # Head form (weights, means, variances) is not described in the paper
    weights, means, variances = mdn.heads(h)
    # Choose a component, then sample from its Gaussian  [§3.2]
    k = categorical(weights)
    return normal(means[k], sqrt(variances[k]))

Contradictions

• Average versus robustness. The text says RawNet2 does better on average and the GMM is more robust 2111.02813 §6. The one-set training tables show the best GMM at aEER 0.062 2111.02813 tab. 2 and the best RawNet2 at 0.258 2111.02813 tab. 3. The averaging rule is not stated, so the two sides are not reconciled here.
• Reported versus recomputed aEER. The paper defines aEER as an average over all test sets 2111.02813 §4.1. Averages recomputed from the rows of 2111.02813 tab. 2 give about 0.058 for FB-MelGAN, not the reported 0.062. The averaging rule is not stated.
• Score definition. The paper writes the log-likelihood ratio over the whole sequence 2111.02813 eq. 5. The released code averages per-frame log-likelihoods before subtracting 2111.02813 app. C. The two values differ by the frame count, so they rank clips of different length differently.

Papers

• Deep Learning for Classical Japanese Literature: Kuzushiji character datasets; a two-hidden-layer mixture density network maps Kuzushiji-Kanji latents to Modern Kanji latents for a stroke decoder, with qualitative results only.
• WaveFake: a dataset of generated speech with per-generator GMM baselines on LFCC and MFCC features, scored by log-likelihood ratio; the GMMs are reported as more robust than RawNet2 under telephone-channel simulation.
• This Time is Different: An Observability Perspective on Time Series Foundation Models: Toto, a forecaster whose output head is a Student-T mixture with a robust loss; a heavy-tailed relative of the GMM, not a Gaussian one.
• Warped Hypertime Representations for Long-term Autonomy of Mobile Robots: time projected onto periodic circles, then a Gaussian mixture over the extended vectors to predict p(a | x, t); results are shown only in figures.
• LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving: a range-image detector whose per-point outputs form a mixture of box distributions, fused with Gaussian products; not a GMM fit.