--- kind: concept slug: audio-deepfake-detection title: Audio deepfake detection summary: >- Decide whether a speech clip came from a speech generator. One card so far, WaveFake: a dataset of generated speech with two baselines, a per-generator Gaussian mixture model (GMM) on LFCC or MFCC features and RawNet2 on raw audio. The main choice is the GMM, which the authors find more robust, or RawNet2, which does better on average. Single paper, author-reported results. tags: [audio, deepfake-detection, speech-synthesis, dataset] papers: ["2111.02813"] related: [generative-adversarial-network, image-manipulation-detection, adversarial-robustness] status: draft updated_at: 2026-10-18 --- ## Overview The task is to decide whether a speech clip is real or was produced by a speech generator [arxiv:2111.02813 §1]. Detection work on generated media has mostly covered images, and generated audio had no overview and no shared dataset [arxiv:2111.02813 §1] [§2.3]. The paper's answer is a dataset that pairs each generated clip with the real recording it was made from, so detectors can be compared on the same material [arxiv:2111.02813 §1] [§3]. The generated clips are resynthesised from real mel spectrograms, so they are a proxy for malicious audio. The authors argue that results may transfer, and they list the limits: one speaker in the main reference set, and no adversarial robustness evaluation [arxiv:2111.02813 §5]. ## Variants ### Per-generator GMM on LFCC or MFCC features For each generator, fit one diagonal-covariance GMM to real frame features and one to generated frame features. Each frame has 20 coefficients with their deltas and double deltas [arxiv:2111.02813 app. C]. A clip is scored by the log-likelihood ratio between the two models, and a low score means fake [arxiv:2111.02813 §4.1] [eq. 5]. Results are EER per test set, averaged into aEER [arxiv:2111.02813 §4.1]. Card: [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](/papers/2111.02813). Reported, author-reported: - LFCC, trained on one generator's set: FB-MelGAN has the lowest aEER, 0.062; MelGAN 0.215 and MelGAN (L) 0.222 [arxiv:2111.02813 tab. 2]. - LFCC, leave-one-out: with HiFi-GAN held out, aEER 0.058, and a range of 0.058 to 0.086 across held-out generators [arxiv:2111.02813 tab. 4]. - MFCC, trained on one generator's set: WaveGlow has the lowest aEER, 0.214, and the range is 0.214 to 0.360 [arxiv:2111.02813 tab. 6] [app. D]. The MFCC and LFCC numbers come from different tables with different best generators, so the card does not support a clear winner between them. ### RawNet2, a CNN-GRU on raw waveform RawNet2 takes raw audio, padded or trimmed to 4 s, and outputs one binary score [arxiv:2111.02813 §4.1] [app. C]. It has no hand-made features. Card: [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](/papers/2111.02813). Reported, author-reported: - Trained on one generator's set: MelGAN (L) has the lowest aEER, 0.258; the others are 0.292 to 0.363 [arxiv:2111.02813 tab. 3]. - Leave-one-out: with MelGAN held out, aEER 0.040, the lowest value in the paper, but the range is 0.040 to 0.241 and several models fail on the JSUT sets [arxiv:2111.02813 tab. 5]. - Under the phone-call simulation, errors rise to as high as 0.937 [arxiv:2111.02813 tab. 8]. ## When to use which | Situation | Use | Why | |---|---|---| | Audio may pass through a phone channel and you need stable results | Per-generator GMM on LFCC | The authors call the GMMs more robust [arxiv:2111.02813 §6]. Under their phone-call simulation, the worst LFCC GMM error is 0.003, against errors up to 0.937 for RawNet2 [arxiv:2111.02813 tab. 7] [tab. 8]. The simulation parameters are the code's, not the paper's [arxiv:2111.02813 §4.1]. | | You want the best average score and accept a less stable result under phone-call audio | RawNet2 | The authors say it does better on average [arxiv:2111.02813 §6]. The best held-out result is RawNet2's 0.040 [tab. 5]. | | The generator at test time is unknown, several training generators are available | Either, judged on spread | Leave-one-out spread: GMM 0.058 to 0.086 [tab. 4]; RawNet2 0.040 to 0.241 [tab. 5]. The GMM's spread is much narrower. | Both variants were evaluated on the paper's own data. The card's "Paper vs code" section lists differences between the paper and the released code, such as silence trimming and the RawNet2 source. Check it before you reproduce any of these numbers. ## Implementation sketch The shared core is: extract frame features, train real and generated models per generator, score each clip, then report EER per test set. The two variants differ only in the model and the feature step. ```pseudocode # Shared core for both variants [arxiv:2111.02813 §4.1] [app. C] # Input: real clips, generated clips from one generator, test clips # Output: EER per test set, averaged into aEER def train_gmm_pair(real_clips, fake_clips): # Frame features from each clip, pooled per class [app. C] real_frames = stack(features(c) for c in real_clips) fake_frames = stack(features(c) for c in fake_clips) # One diagonal GMM per class: 128 components, 256 for leave-one-out [app. C] # Fit with Adam, learning rate 1e-3, 10 epochs, batch 128 [app. C] return fit_gmm(real_frames, 128), fit_gmm(fake_frames, 128) def score(clip, real_gmm, fake_gmm): # Log-likelihood ratio; a higher score means more likely real [eq. 5] # The code averages per-frame values; the paper sums them # [code: dfadetect/models/gaussian_mixture_model.py#L383-L396] frames = features(clip) return mean(log_lik(real_gmm, frames)) - mean(log_lik(fake_gmm, frames)) def aEER(scores_by_test_set): # EER per test set, then the mean over sets; the averaging rule is not stated [§4.1] return mean(eer(real, fake) for real, fake in scores_by_test_set) ``` - For RawNet2, replace `train_gmm_pair` and `score` with a binary classifier on 4 s inputs [arxiv:2111.02813 app. C]. - `features` is the LFCC or MFCC recipe. The FFT size is set in the code, but the window and hop are not stated in the paper [arxiv:2111.02813 app. C]. - The paper gives a 20 % hold-out [arxiv:2111.02813 §4.1]. The code takes the last 20 % of a sorted file list, a fixed split, not a random one [code: dfadetect/datasets.py#L280-L283]. ## Contradictions - Averaging: the paper gives FB-MelGAN's LFCC aEER as 0.062 [arxiv:2111.02813 §4.1]. Recomputing the mean of its tab. 2 row gives about 0.058, and the averaging rule is not stated [arxiv:2111.02813 tab. 2]. Recompute from per-set EERs before comparing numbers. - Robustness versus average: the authors rank the GMM as more robust and RawNet2 as better on average [arxiv:2111.02813 §6]. This is a trade-off the paper reports, not a disagreement, but the two claims point to different choices. ## Papers - [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](/papers/2111.02813): the dataset, plus the GMM and RawNet2 baselines and their results.