---
kind: concept
slug: gaussian-mixture-model
title: Gaussian mixture model
summary: >-
  A Gaussian mixture model (GMM) fits a density as a weighted sum of Gaussians. Five cards: WaveFake fits per-generator GMMs and scores clips by log-likelihood ratio; hypertime fits a GMM over time-projected vectors; Kuzushiji uses a mixture density network; Toto (Student-T) and LaserNet (box distributions) use other mixture forms. Author-reported.
tags: [audio, deepfake-detection, mixture-models, conditional-density, baseline, time-series, robotics, object-detection]
papers: ["1812.01718", "2111.02813", "2505.14766", "1810.04285", "1903.08701"]
related: [expectation-maximization, k-means-clustering, spectral-methods, variational-autoencoder, audio-deepfake-detection, generative-adversarial-network, time-series-forecasting, uncertainty-estimation, object-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

A GMM models a set of feature vectors as drawn from a mixture of several
Gaussian components, each with its own mean and covariance. Five cards use a
mixture, in different ways. WaveFake uses a GMM as a class-conditional density:
one model is fit to the real speech features and one to the features of each
generator. A clip is scored by the difference of the two log-likelihoods, which
is a likelihood ratio [arxiv:2111.02813 §4.1] [arxiv:2111.02813 eq. 5]. The
sign of this score separates the two classes, and the decision clips are then
assessed by equal error rate [arxiv:2111.02813 §4.1]. The Kuzushiji paper uses
a mixture of Gaussians as a conditional density, P(z_new | z_old), that maps a
latent code from one domain to another [arxiv:1812.01718 §3.2]
[arxiv:1812.01718 alg. 1]. The hypertime paper fits a standard Gaussian
mixture, but over vectors that include the cosine and sine of time for each
periodicity it finds, and uses the fitted model to predict p(a | x, t)
[arxiv:1810.04285 §3.5]. The four uses share the Gaussian form but not the
fitting or the scoring. Two cards use other mixture forms. Toto puts a mixture
of Student-T components on a forecaster's output [arxiv:2505.14766 §3.1]. The
LaserNet detector outputs a mixture of box distributions per point, trained
with a Laplace-style loss [arxiv:1903.08701 §3.5].

## Variants

### Diagonal-covariance GMM trained by gradient descent

The paper's stated method. Each component uses a diagonal covariance matrix.
The paper uses 128 components, and 256 for the leave-one-out models. Training
uses Adam with a learning rate of 1e-3, batch size 128 and 10 epochs
[arxiv:2111.02813 app. C]. Card: [WaveFake](/papers/2111.02813).

### EM training (code only)

The released code also contains an expectation-maximisation training option.
The paper does not describe it, so no numbers from the paper are tied to it
[arxiv:2111.02813 app. C]. For the algorithm itself, see the
[expectation maximisation](/concepts/expectation-maximization) page. The
released script's defaults differ from the paper's stated training, and the
repository records no command that reproduces the paper's runs
[arxiv:2111.02813 app. C].

### EM over time-projected vectors (hypertime)

Each measurement has a value a, a location x and a time t. For each periodicity
T the vector gains a cosine and a sine of 2πt/T [arxiv:1810.04285 eq. 1]. A
weighted sum of n multivariate Gaussians is then fit over these vectors
[arxiv:1810.04285 eq. 2] [arxiv:1810.04285 §3.5]. Periodicities are added one at
a time, each chosen by a spectral scan of the model error [arxiv:1810.04285
§3.7]. The loop stops when the training error no longer falls
[arxiv:1810.04285 §3.6]. The paper gives two clustering routes. HyT-EM runs EM
with a fixed number of clusters, through an OpenCV implementation. HyT-KM starts
from k-means centres, uses cosine distance on the time coordinates, and chooses
the number of clusters itself [arxiv:1810.04285 §3.8]. The text states no
numbers. Figures show lower error than the baselines for the door and velocity
experiments [arxiv:1810.04285 §4.1] [arxiv:1810.04285 §4.3]. As printed, eq. 7
has the absolute value inside the sum, so the score does not depend on the
candidate period as the text describes [arxiv:1810.04285 eq. 7]. The card
records no released hypertime code. Card:
[Warped Hypertime Representations for Long-term Autonomy of Mobile Robots](/papers/1810.04285).

### Conditional mixture: mixture density network

A mixture density network (MDN) makes the mixture depend on an input. The
Kuzushiji paper trains one with two hidden layers to model P(z_new | z_old) as a
mixture of Gaussians, where both z vectors are 64-dimensional VAE latents
[arxiv:1812.01718 §3.2] [arxiv:1812.01718 alg. 1]. The number of components,
the hidden widths, the learning rate and the choice of training pairs are not
stated [arxiv:1812.01718 §3.2]. The paper uses the MDN to draw a sampled latent
for the stroke decoder, and reports no metric for this step; the evidence is
qualitative examples [arxiv:1812.01718 §3.2]. Card:
[Deep Learning for Classical Japanese Literature](/papers/1812.01718).

### Feature front end: LFCC or MFCC

The GMM is fed frame-level features, either linear-frequency cepstral
coefficients (LFCC) or mel-frequency cepstral coefficients (MFCC). Both use 20
coefficients plus deltas and double deltas [arxiv:2111.02813 §2.1]
[arxiv:2111.02813 app. C]. LFCC replaces the mel filterbank with a linear one
and keeps the other steps [arxiv:2111.02813 §2.1]. The choice changes results
in both directions (see the results table).

### Student-T mixture head (not Gaussian)

Toto is a decoder-only forecaster for observability metrics. Its output head
produces, for each token, K = 24 Student-T components and a weight for each
[arxiv:2505.14766 §3.1] [arxiv:2505.14766 app. A.6]. The degrees of freedom
are 2 plus a positive value, so the variance is finite [arxiv:2505.14766 app.
A.3]. The mixture density is [arxiv:2505.14766 eq. 3] and each component's
density is [arxiv:2505.14766 eq. 4]. Training mixes the negative
log-likelihood under the mixture [arxiv:2505.14766 eq. 10] with a Cauchy-type
robust loss on the mixture mean [arxiv:2505.14766 eq. 11]
[arxiv:2505.14766 §3.1]. The components are Student-T, so their tails are
heavier than a Gaussian's. The authors' ablation on held-out observability NLL
replaces the mixture with a single Student-T output; the NLL rises by 27.2%
[arxiv:2505.14766 app. E]. Card:
[This Time is Different](/papers/2505.14766).

### Mixture of box distributions in a detector (not fitted as a GMM)

LaserNet works on the LiDAR range image. For each point, the network outputs
class probabilities and, for each of K mixture components per class, a box
with a log standard deviation, plus a mixture weight [arxiv:1903.08701 §3.3].
During training each point is matched to the one component whose mean is
closest to its ground-truth box, and the box term is a Laplace-style loss
rather than a Gaussian likelihood [arxiv:1903.08701 §3.5] [arxiv:1903.08701
eq. 9]. At inference, predictions are merged by mean-shift clustering, and each
cluster is fused as a product of Gaussians with precision weights 1/σ²
[arxiv:1903.08701 §3.4] [arxiv:1903.08701 eq. 6]. The mixture width is K = 3
for vehicles and K = 1 for pedestrians and bikes on ATG4D [arxiv:1903.08701
§4.1]. The author-reported vehicle AP0.7 on ATG4D is 85.34, but on KITTI the
method trails BEV detectors [arxiv:1903.08701 tab. 1] [arxiv:1903.08701 tab. 4].
Card: [LaserNet](/papers/1903.08701).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Train on one generator's output, test on the others, LFCC features | GMM on LFCC | Best single-source LFCC GMM reaches aEER 0.062 when trained on FB-MelGAN data [arxiv:2111.02813 tab. 2] |
| Telephone-channel degradation (simulated) | GMM | Highest GMM error is 0.003 across tested generators, against errors up to 0.937 for RawNet2 [arxiv:2111.02813 tab. 7] [arxiv:2111.02813 tab. 8] |
| Need a detector that holds up when one generator is held out | Compare both; results vary | Leave-one-out GMM range 0.058–0.086 [arxiv:2111.02813 tab. 4]; RawNet2 range 0.040–0.241 [arxiv:2111.02813 tab. 5] |
| Compute and memory budget | Not stated | No runtime or memory figures for the GMM are recorded in the card [arxiv:2111.02813 §4.1] |
| Need the claimed best average accuracy | Authors favour the neural model | The authors report RawNet2 as better on average and GMM as more robust [arxiv:2111.02813 §6] |
| Need a density over a target that depends on an input, such as mapping a latent from one domain to another | Mixture density network | The Kuzushiji paper uses one for this step, but gives no metric for it, so its quality is not shown [arxiv:1812.01718 §3.2] |
| Need a density over a robot's state that repeats daily or weekly, with the time of day used as a feature | Hypertime GMM (HyT-EM or HyT-KM) | Figures show lower error than the baselines in two experiments, but the text gives no numbers and the cluster count is not stated for HyT-EM [arxiv:1810.04285 §4.1] [arxiv:1810.04285 §3.8] |
| Need a forecast distribution with heavy tails over spiky series | Student-T mixture head | Toto's ablation shows the mixture beats a single Student-T output by 27.2% NLL [arxiv:2505.14766 app. E] |
| Single-stage LiDAR detector that also reports per-box uncertainty | Mixture head (LaserNet) | Author-reported 30 ms total runtime on KITTI [arxiv:1903.08701 tab. 3]; the authors say the distribution head needs more data than KITTI provides [arxiv:1903.08701 §4.5] |

The authors describe the GMM as more robust and the neural RawNet2 as better on
average [arxiv:2111.02813 §6]. The tables do not clearly support both claims at
once; see Contradictions.

## Implementation sketch

Common core for one GMM: extract frame features, fit one model per class, and
score each clip by the log-likelihood ratio. Variants differ in the training
loop (gradient descent or EM) and in the feature front end. The conditional
variant (mixture density network) does not fit a model per class. It maps an
input to the parameters of a mixture, then samples from it. The hypertime
variant refits the mixture each time it adds a periodicity to the vectors; its
loop is on the card [arxiv:1810.04285 §3.4] [arxiv:1810.04285 §3.7].

```pseudocode
# Frame features for one file  [arxiv:2111.02813 §2.1] [app. C]
# Input:  wav, mono waveform  (samples,)
# Output: features  (T, 60)  # 20 coefficients + delta + double delta

def extract_features(wav):
    # Resample to 16 kHz and remove long silences  [app. C]
    wav = remove_silence(resample(wav, 16_000))
    # Power spectrum, then 20 linear (LFCC) or mel (MFCC) bands  [§2.1]
    bands = filterbank(power_spectrum(wav), n_bands=20)  # (20, T)
    # Log compression, then DCT keeping 20 coefficients  [eq. 3]
    coeffs = dct(log(bands + 1e-6), n_coeffs=20)  # (20, T)
    # Central-difference deltas and double deltas  [eq. 4]
    deltas = delta(coeffs)
    double_deltas = delta(deltas)
    return concat([coeffs, deltas, double_deltas]).T
```

```pseudocode
# Fit one diagonal-covariance GMM per class  [§4] [app. C]
# Input:  frames  (N, 60), components (128, or 256 for leave-one-out)
# Output: gmm with weights, means and variances

def fit_gmm(frames, components):
    # Paper: gradient descent with Adam, lr 1e-3, batch 128, 10 epochs
    # [app. C]. Alternative: EM, code only [app. C]
    ...
```

```pseudocode
# Score one clip with a real and a fake GMM  [eq. 5] [§4.1]
# Input:  wav, real_gmm, fake_gmm
# Output: score (higher means more likely real)

def score_clip(wav, real_gmm, fake_gmm):
    frames = extract_features(wav)  # (T, 60)
    # The paper writes a sum over frames; the released code averages
    # per-frame log-likelihoods [eq. 5] [app. C]
    return log_likelihood(real_gmm, frames) - log_likelihood(fake_gmm, frames)
```

The window, hop and FFT size are not stated in the paper. They are not needed
to implement the GMM itself, but they matter for reproducing the features.

```pseudocode
# Sample from a mixture density network  [§3.2] [alg. 1]
# Input:  z_old (64,)  # latent from the old-domain VAE
# Output: z_new (64,)  # latent in the new domain

def sample_mdn(z_old, mdn):
    # Two hidden layers; widths not stated  [§3.2]
    h = mdn.hidden(z_old)
    # Head form (weights, means, variances) is not described in the paper
    weights, means, variances = mdn.heads(h)
    # Choose a component, then sample from its Gaussian  [§3.2]
    k = categorical(weights)
    return normal(means[k], sqrt(variances[k]))
```

## Contradictions

- **Average versus robustness.** The text says RawNet2 does better on average
  and the GMM is more robust [arxiv:2111.02813 §6]. The one-set training tables
  show the best GMM at aEER 0.062 [arxiv:2111.02813 tab. 2] and the best
  RawNet2 at 0.258 [arxiv:2111.02813 tab. 3]. The averaging rule is not
  stated, so the two sides are not reconciled here.
- **Reported versus recomputed aEER.** The paper defines aEER as an average
  over all test sets [arxiv:2111.02813 §4.1]. Averages recomputed from the
  rows of [arxiv:2111.02813 tab. 2] give about 0.058 for FB-MelGAN, not the
  reported 0.062. The averaging rule is not stated.
- **Score definition.** The paper writes the log-likelihood ratio over the
  whole sequence [arxiv:2111.02813 eq. 5]. The released code averages
  per-frame log-likelihoods before subtracting [arxiv:2111.02813 app. C]. The
  two values differ by the frame count, so they rank clips of different length
  differently.

## Papers

- [Deep Learning for Classical Japanese Literature](/papers/1812.01718): Kuzushiji character datasets; a two-hidden-layer mixture density network maps Kuzushiji-Kanji latents to Modern Kanji latents for a stroke decoder, with qualitative results only.
- [WaveFake](/papers/2111.02813): a dataset of generated speech with per-generator GMM baselines on LFCC and MFCC features, scored by log-likelihood ratio; the GMMs are reported as more robust than RawNet2 under telephone-channel simulation.
- [This Time is Different: An Observability Perspective on Time Series Foundation Models](/papers/2505.14766): Toto, a forecaster whose output head is a Student-T mixture with a robust loss; a heavy-tailed relative of the GMM, not a Gaussian one.
- [Warped Hypertime Representations for Long-term Autonomy of Mobile Robots](/papers/1810.04285): time projected onto periodic circles, then a Gaussian mixture over the extended vectors to predict p(a | x, t); results are shown only in figures.
- [LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving](/papers/1903.08701): a range-image detector whose per-point outputs form a mixture of box distributions, fused with Gaussian products; not a GMM fit.