Audio deepfake detection

Overview

The task is to decide whether a speech clip is real or was produced by a speech generator 2111.02813 §1. Detection work on generated media has mostly covered images, and generated audio had no overview and no shared dataset 2111.02813 §1 §2.3. The paper's answer is a dataset that pairs each generated clip with the real recording it was made from, so detectors can be compared on the same material 2111.02813 §1 §3.

The generated clips are resynthesised from real mel spectrograms, so they are a proxy for malicious audio. The authors argue that results may transfer, and they list the limits: one speaker in the main reference set, and no adversarial robustness evaluation 2111.02813 §5.

Variants

Per-generator GMM on LFCC or MFCC features

For each generator, fit one diagonal-covariance GMM to real frame features and one to generated frame features. Each frame has 20 coefficients with their deltas and double deltas 2111.02813 app. C. A clip is scored by the log-likelihood ratio between the two models, and a low score means fake 2111.02813 §4.1 eq. 5. Results are EER per test set, averaged into aEER 2111.02813 §4.1. Card: WaveFake: A Data Set to Facilitate Audio Deepfake Detection.

Reported, author-reported:

• LFCC, trained on one generator's set: FB-MelGAN has the lowest aEER, 0.062; MelGAN 0.215 and MelGAN (L) 0.222 2111.02813 tab. 2.
• LFCC, leave-one-out: with HiFi-GAN held out, aEER 0.058, and a range of 0.058 to 0.086 across held-out generators 2111.02813 tab. 4.
• MFCC, trained on one generator's set: WaveGlow has the lowest aEER, 0.214, and the range is 0.214 to 0.360 2111.02813 tab. 6 app. D.

The MFCC and LFCC numbers come from different tables with different best generators, so the card does not support a clear winner between them.

RawNet2, a CNN-GRU on raw waveform

RawNet2 takes raw audio, padded or trimmed to 4 s, and outputs one binary score 2111.02813 §4.1 app. C. It has no hand-made features. Card: WaveFake: A Data Set to Facilitate Audio Deepfake Detection.

Reported, author-reported:

• Trained on one generator's set: MelGAN (L) has the lowest aEER, 0.258; the others are 0.292 to 0.363 2111.02813 tab. 3.
• Leave-one-out: with MelGAN held out, aEER 0.040, the lowest value in the paper, but the range is 0.040 to 0.241 and several models fail on the JSUT sets 2111.02813 tab. 5.
• Under the phone-call simulation, errors rise to as high as 0.937 2111.02813 tab. 8.

When to use which

Both variants were evaluated on the paper's own data. The card's "Paper vs code" section lists differences between the paper and the released code, such as silence trimming and the RawNet2 source. Check it before you reproduce any of these numbers.

Implementation sketch

The shared core is: extract frame features, train real and generated models per generator, score each clip, then report EER per test set. The two variants differ only in the model and the feature step.

# Shared core for both variants  [arxiv:2111.02813 §4.1] [app. C]
# Input:  real clips, generated clips from one generator, test clips
# Output: EER per test set, averaged into aEER

def train_gmm_pair(real_clips, fake_clips):
    # Frame features from each clip, pooled per class  [app. C]
    real_frames = stack(features(c) for c in real_clips)
    fake_frames = stack(features(c) for c in fake_clips)
    # One diagonal GMM per class: 128 components, 256 for leave-one-out  [app. C]
    # Fit with Adam, learning rate 1e-3, 10 epochs, batch 128  [app. C]
    return fit_gmm(real_frames, 128), fit_gmm(fake_frames, 128)

def score(clip, real_gmm, fake_gmm):
    # Log-likelihood ratio; a higher score means more likely real  [eq. 5]
    # The code averages per-frame values; the paper sums them
    # [code: dfadetect/models/gaussian_mixture_model.py#L383-L396]
    frames = features(clip)
    return mean(log_lik(real_gmm, frames)) - mean(log_lik(fake_gmm, frames))

def aEER(scores_by_test_set):
    # EER per test set, then the mean over sets; the averaging rule is not stated  [§4.1]
    return mean(eer(real, fake) for real, fake in scores_by_test_set)

• For RawNet2, replace train_gmm_pair and score with a binary classifier on 4 s inputs 2111.02813 app. C.
• features is the LFCC or MFCC recipe. The FFT size is set in the code, but the window and hop are not stated in the paper 2111.02813 app. C.
• The paper gives a 20 % hold-out 2111.02813 §4.1. The code takes the last 20 % of a sorted file list, a fixed split, not a random one datasets.py.

Contradictions

• Averaging: the paper gives FB-MelGAN's LFCC aEER as 0.062 [arxiv:2111.02813 §4.1]. Recomputing the mean of its tab. 2 row gives about 0.058, and the averaging rule is not stated 2111.02813 tab. 2. Recompute from per-set EERs before comparing numbers.
• Robustness versus average: the authors rank the GMM as more robust and RawNet2 as better on average 2111.02813 §6. This is a trade-off the paper reports, not a disagreement, but the two claims point to different choices.

Papers

• WaveFake: A Data Set to Facilitate Audio Deepfake Detection: the dataset, plus the GMM and RawNet2 baselines and their results.