WaveFake: A Data Set to Facilitate Audio Deepfake Detection

In plain words

Computer-generated speech can now sound convincingly human, which makes it a possible tool for malicious use. Most research on spotting fake media has looked at pictures, while fake audio has been studied much less.

This paper tackles that gap with a dataset. The authors took real speech recordings and ran them through several different speech-generating programs, which produced imitation copies of the same recordings. Anyone can then train a detector on the real and fake versions and compare how well it works. The authors also add a few simple baseline detectors and a way to see which parts of the sound a detector relies on.

The main finding is that detectors do well on the generator they were trained on, and some transfer to other generators, but the results are uneven. A neural network did better on average, while the simpler statistical models held up more reliably. The authors frame the work as a first step, with more evaluation still to do.

TL;DR

• WaveFake is a dataset of audio generated by six architectures from LJSpeech and JSUT recordings, plus a text-to-speech pipeline, released under CC BY-SA 4.0 §3 app. B.
• Baselines: one diagonal-covariance Gaussian mixture model (GMM) per data set, scored by log-likelihood ratio against a real-speech GMM, on LFCC or MFCC features, and a RawNet2 CNN-GRU §4.1 eq. 5.
• Author-reported: the best single-source LFCC GMM (FB-MelGAN) reaches aEER 0.062 tab. 2; the authors call the GMM classifiers more robust than the neural RawNet2 model, which does better on average §6.
• The code and pretrained-model links are in the paper; the RawNet2 source must be fetched separately README.md.

Problem

Detection research for generated media has concentrated on images, while generated audio had no overview and no shared dataset §1 §2.3. Detectors need samples from several realistic generators, and a fair comparison needs real and generated clips that pair up one-to-one §1 §3.

The authors argue that results on this proxy data may transfer to malicious audio, and they list the limits of that argument: one speaker in the main reference set, and no adversarial robustness evaluation §5.

Method

Dataset construction §3

• Reference data: LJSpeech, 13,100 short clips read by one female speaker, about 24 hours §3. JSUT basic5000, 5,000 sentences by one female Japanese speaker, about 6.7 hours §3.
• Each generator is fed the mel spectrograms extracted from the reference audio, and its output is the generated copy §3 (paragraph "Sampling procedure").
• Generators: MelGAN and a larger MelGAN (L) variant, Full-band MelGAN (FB-MelGAN), Multi-band MelGAN (MB-MelGAN), HiFi-GAN, WaveGlow and Parallel WaveGAN (PWG) §3 (paragraph "Architectures"). WaveGlow uses the official code at commit 8afb643 with its PyTorch Hub checkpoint; PWG uses a popular GitHub implementation at commit 12c677e; HiFi-GAN uses the pretrained models from its repository §3.
• TTS pipeline: a Conformer maps novel phrases to mel spectrograms, then a PWG vocoder (fine-tuned on LJSpeech) produces audio. Its phrases come from 16,283 Common Voice sentences, so they do not overlap the training data §3 fig. 1.
• Output: 117,985 16-bit PCM WAV clips, about 196 hours, on Zenodo (DOI 10.5281/zenodo.5642694) §3.

Features §2.1 app. C

Mel-scale mapping eq. 1, mel spectrogram eq. 2, and MFCC eq. 3 are defined as follows. LFCC replaces the mel filterbank with a linear one and keeps all other steps §2.1. Deltas and double deltas use a central-difference filter eq. 4. The classifiers use 20 coefficients plus delta and double-delta app. C; the code stacks the three blocks, giving 60 values per frame datasets.py.

f_{\text{mel}} = 2595 \cdot \log_{10}\!\left(1 + \frac{f}{700}\right) eq. 1

X_{\mathrm{mel}}(t,s) = \sum_{k=0}^{K-1} |X(t,k)|\, H_{\mathrm{mel}}(s,k) eq. 2

c(t,r) = \sum_{s=0}^{S-1} \log\!\big[X_{\mathrm{mel}}(t,s)\big]\, \cos\!\Big[\frac{\pi r (s+0.5)}{S}\Big] eq. 3

d(t) = \frac{\sum_{n=1}^{N} n\,\big[c(t+n) - c(t-n)\big]}{2 \sum_{n=1}^{N} n^2} eq. 4

The window length of the delta filter N is user-defined §2.1 and is not given for the paper's runs app. C.

# Frame features for one file  [§2.1] [app. C]
# Input:  wav, mono waveform  (samples,)
# Output: features  (T, 60)  # 20 LFCC + delta + double delta

def extract_features(wav):
    # Resample to 16 kHz and remove long silences  [app. C]
    wav = remove_silence(resample(wav, 16_000))
    # Power spectrogram from the short-time Fourier transform
    # [code: experiment_config.py#L9]
    power = stft_power(wav, n_fft=512)  # (F, T)
    # Linear filterbank of 20 triangular bands, 0 to Nyquist
    # [code: dfadetect/lfcc.py#L66-L70]
    bands = linear_filterbank(power, n_bands=20)  # (20, T)
    # Log compression, then DCT keeping 20 coefficients  [eq. 3]
    coeffs = dct(log(bands + 1e-6), n_coeffs=20)  # (20, T)
    # Central-difference deltas and double deltas  [eq. 4]
    deltas = delta(coeffs)  # (20, T)
    double_deltas = delta(deltas)  # (20, T)
    return concat([coeffs, deltas, double_deltas]).T

The pseudocode follows the code where the paper is silent. The FFT size is set only in the code; the window and hop are not set anywhere in the repository, so library defaults apply. The log(bands + 1e-6) offset is code-only lfcc.py.

Detectors §4 §4.1 app. C

• Baseline 1, GMM: for each generator, one GMM is fit to real features and one to generated features, both with diagonal covariance. The paper uses 128 components, and 256 for the leave-one-out models app. C.
• Score: the log-likelihood ratio between the real and the generated model eq. 5. The file is classed as fake when the score is low.
• Metric: equal error rate (EER) per test set, averaged over test sets into aEER. Chance is 0.5 §4.1.
• Baseline 2, RawNet2: a CNN-GRU taking raw audio, input length 4 s, with a binary output and a trained fully connected layer §4.1 app. C.

\Lambda(\mathbf{X}) = \log p(\mathbf{X}\mid\theta_n) - \log p(\mathbf{X}\mid\theta_s) eq. 5

Here \theta_n is the real model and \theta_s the generated one §4.1. The paper writes this as a log-likelihood of the whole feature sequence \mathbf{X}.

# Score one file with a real and a fake GMM  [§4.1] [eq. 5]
# Input:  wav, real_gmm, fake_gmm
# Output: score (a higher value means more likely real)

def score_file(wav, real_gmm, fake_gmm):
    # Frame features for this file  [app. C]
    frames = extract_features(wav)  # (T, 60)
    # Per-frame log-likelihood under each model  [§4.1]
    real_ll = log_likelihood(real_gmm, frames)  # (T,)
    fake_ll = log_likelihood(fake_gmm, frames)  # (T,)
    # Log-likelihood ratio, averaged over frames in code  [eq. 5]
    # [code: dfadetect/models/gaussian_mixture_model.py#L383-L396]
    return mean(real_ll) - mean(fake_ll)

GMM training, as described by the paper app. C:

# Fit one diagonal GMM to frame features  [app. C]
# Input:  frames  (N, 60), components (128 or 256)
# Output: gmm with learned weights, means and variances

def fit_gmm(frames, components):
    # Gradient descent with Adam, initial learning rate 1e-3  [app. C]
    # Loss: mean negative log-likelihood of frames  [app. C]
    # [code: dfadetect/trainer.py#L105-L106]
    # Trained for 10 epochs with batch size 128  [app. C]
    ...

The code also has an EM training option that the paper does not describe train_models.py; the paper's stated training is gradient descent app. C.

Training and preprocessing app. C

• Resample every file to 16 kHz and remove silent parts longer than 2 s app. C.
• For RawNet2, pad or trim each file to 4 s app. C.
• Splits: a hold-out set of 20% of the data §4.1.
• GMM: Adam, learning rate 1e-3, batch size 128, 10 epochs app. C.
• RawNet2: binary cross-entropy, Adam, learning rate 1e-4, weight decay 1e-4, batch size 128, 10 epochs; the best model by validation accuracy is restored app. C.

Attribution §4.2 fig. 4 app. G

The paper uses BlurIG, an integrated-gradients variant that blurs the input across increasing sigma levels, to show which LFCC bands drive a classifier's output §4.2. Step count and blur settings are not stated in the paper.

Hyperparameters

Results

Author-reported. Setup: each row is a training set, each column a test set (LJSpeech-based or JSUT-based) §4.1 tab. 2. Chance is 0.5.

The text names FB-MelGAN as the best single training set for the GMM (average EER 0.062) §4.1. It reports the neural model as better on average and the GMM as more robust §6. The leave-one-out gain is described as 0.062 to 0.058 §4.1.

Paper vs code

• Silence removal: the paper says silences longer than 2 s are removed app. C; the code trims silences longer than 0.2 s at a 1% volume threshold datasets.py (applied in datasets.py).
• Score: the paper writes the likelihood ratio over the whole sequence eq. 5; the code averages per-frame log-likelihoods before subtracting gaussian_mixture_model.py.
• Training schedule: the paper states 10 epochs and batch size 128 app. C. The script defaults are 5 epochs, batch size 8 and 10 EM retraining runs train_models.py. The repository records no command that reproduces the paper's runs.
• Leave-one-out epochs: the code divides the epoch count by the number of other sets train_models.py. The paper gives one value for all models.
• Validation split for GMM: the paper reports a 20% hold-out only §4.1. The GMM trainer additionally holds out 5% of the training files for a test loss, and that split is not used for model selection train_models.py trainer.py.
• RawNet2 model selection: the best checkpoint is chosen on a random 20% of the training files, not the reporting split trainer.py trainer.py. The paper says "validation accuracy" app. C.
• Test split: the code takes the last 20% of the sorted file list, which is a fixed, not random, split datasets.py.
• Padding: the code pads short files by repeating the waveform, not with zeros datasets.py. The paper says only "pad or trim" app. C.
• RawNet2 source: the code imports dfadetect/models/raw_net2.py train_models.py, which is not in the repository; the README says to download it from the ASVspoof 2021 baseline README.md. The paper says the authors' open-source version was used app. C.
• Phone simulation: the paper gives no parameters §4.1. The code applies a 4 kHz low-pass, a compander, resampling to 8 kHz and a GSM codec datasets.py.
• Pitch statistics: the paper names normalised cross-correlation with median smoothing, 50–500 Hz §3. The statistics script uses torchaudio's detect_pitch_frequency at 22,050 Hz with 50–500 Hz statistics.py statistics.py.
• Attribution: the paper names BlurIG without settings §4.2. The code uses a maximum sigma of 100 attribute.py and 50 steps by default attribute.py.

Source code: https://github.com/RUB-SysSec/WaveFake at commit d52d51b9ccdb0cec3f484e84b228791f06b955be, MIT licence (copyright Ruhr University Bochum) LICENSE.

Pitfalls

• Silence and padding details differ from the paper (see Paper vs code). Use the code's 0.2 s silence trim and tiled padding if you want to reproduce the released results, and state which one you use.
• Unstated hyperparameters: FFT size, window, hop, delta window N, GMM initialisation, and the attribution blur settings are not in the paper. The code fixes FFT size 512 and uses library defaults for window and hop.
• Score scale: the paper's sum over frames eq. 5 and the code's per-frame mean differ by the frame count. Their values are not interchangeable, and they rank files differently when files have different lengths.
• EER pairing: the code truncates the real and fake score lists to the shorter length before computing EER evaluate_models.py. The code's comment says the JSUT fake samples are fewer, so the two lists are not equal in size there.
• aEER definition: the paper says aEER averages over all test sets §4.1. Computed by hand from the rows of tab. 2 (10 test sets), the FB-MelGAN row averages to about 0.058, not 0.062, and the MelGAN row to about 0.196, not 0.215. The averaging rule is not stated; recompute from per-set EERs.
• Set count in the text: §3 says ten sample sets, "seven based on LJSpeech" and "two based on JSUT", but the list of LJSpeech sets omits MB-MelGAN, which appears in tab. 2 and tab. 1. The tables show eight LJSpeech-based and two JSUT-based columns.
• Appendix references: the body cites supplementary material by number (for example "Section 3" for training details and "Section 8" for the filterbank). The appendix uses letters, so these are cited here as app. A–app. H. The attribution subsection is §4.2 in the LaTeX numbering, but the appendix refers to "Section 4.3".
• Dataset licence: the authors release the data under CC BY-SA 4.0, but say the legal status of generated copies of CC-licensed audio is unresolved §3 app. B. Treat the licence as the authors' choice, not as settled law.
• Generalisation is uneven: RawNet2 errors rise sharply under phone simulation tab. 8, and several RawNet2 leave-one-out models fail on JSUT tab. 5. The authors call the results encouraging but say much room for improvement remains §5 §6.
• Adversarial robustness and realistic attacks are not evaluated; the authors flag this as future work §5.

Lineage

• Builds on: the MFCC and LFCC recipes of Sahidullah et al. §4.1 app. C, the RawNet2 model used by the ASVspoof challenge §4 app. C, and the GAN-based vocoders and WaveGlow whose samples form the dataset §2.2 §3. These papers have no cards in this wiki, so nothing from them is used here.
• Compared against: the ASVspoof 2021 baseline GMM and RawNet2 models §4.
• Superseded by: none recorded in this wiki.