Mel-frequency cepstral coefficients

Overview

MFCCs summarise the spectral shape of a short audio frame in a few numbers. The recipe is: take the short-time spectrum, pool it through a filterbank whose bands are spaced on the mel scale, take logarithms, then apply a discrete cosine transform (DCT) 2111.02813 §2.1 eq. 1 eq. 2 eq. 3. The DCT step is where the "cepstral" part comes from. Deltas and double deltas (finite differences over neighbouring frames) are often appended to capture how the spectrum changes over time 2111.02813 §2.1 eq. 4.

The WaveFake paper defines this recipe for audio deepfake detection. It calls the same pipeline with a linear filterbank LFCC, and it reports both as inputs to a Gaussian mixture model 2111.02813 §2.1 §4.1. The numeric results on this page therefore still rest on one paper. A second card, an emotion detector for faces and speech, computes MFCCs as one of six speech features 2605.30099 §III.E, and a third, an emotional speech database, feeds MFCCs to its classifier inside a larger openSMILE feature vector 2105.14762 §4.4. Neither isolates the effect of MFCCs (see Variants).

Variants

MFCC: mel-spaced filterbank

The filterbank H_mel pools the spectrum into bands whose spacing follows the mel mapping in eq. 1. Each band output is a weighted sum of the magnitude spectrum eq. 2. The log of the band energies is then passed through a DCT eq. 3. The paper's classifiers keep 20 coefficients and add their deltas and double deltas, which gives 60 values per frame 2111.02813 app. C datasets.py. The paper does not state the number of mel bands 2111.02813 app. C. Card: WaveFake: A Data Set to Facilitate Audio Deepfake Detection.

Reported, author-reported:

• Trained on one generator's set, with a GMM per generator: WaveGlow has the lowest aEER, 0.214, and the range is 0.214 to 0.360 2111.02813 tab. 6 app. D.

LFCC: linear-spaced filterbank

LFCC keeps every other step of the MFCC recipe and replaces the mel filterbank with a linear one 2111.02813 §2.1. The released code uses 20 linear triangular bands lfcc.py and adds a small offset inside the logarithm lfcc.py. The paper does not state that offset. Card: WaveFake: A Data Set to Facilitate Audio Deepfake Detection.

Reported, author-reported:

• Trained on one generator's set, with a GMM per generator: FB-MelGAN has the lowest aEER, 0.062; MelGAN 0.215 and MelGAN (L) 0.222 2111.02813 tab. 2.
• Leave-one-out, with HiFi-GAN held out: aEER 0.058, and a range of 0.058 to 0.086 2111.02813 tab. 4.
• Under the paper's phone-call simulation, the highest error across the tested generators is 0.003 2111.02813 tab. 7 app. E.

MFCC as one feature among several

An emotion detector for faces and speech computes, per audio clip, the zero-crossing rate, a 12-bin chroma vector, chroma deviation, MFCCs, root-mean-square energy and a mel spectrogram 2605.30099 §III.E. Its speech CNN is fed these features together. The card gives no MFCC count, filterbank size or delta settings, and it reports no result for MFCCs alone. Its headline accuracy is for the full face-plus-speech system 2605.30099 tab. 3. Card: Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection.

MFCC inside an openSMILE frame vector

The ESD paper validates its emotional-speech database with a speech emotion recognizer. Its input is openSMILE frame-level features, computed with a 25 ms window and a 10 ms shift, giving 312 values per frame. The vector includes zero-crossing rate, voicing probability, MFCCs and the mel spectrum 2105.14762 §4.4. The card does not give the MFCC count or the filterbank size inside openSMILE. The classifier is one LSTM layer, then a fully connected layer of 256 ReLU units and a 5-way softmax [arxiv:2105.14762 §4.4]. Its reported accuracy, 92.0% for Chinese and 89.0% for English, is for the whole vector, not for MFCCs alone 2105.14762 fig. 9. Card: Emotional Voice Conversion: Theory, Databases and ESD.

The same card names a different spectral feature for its voice conversion benchmark: 24-dimensional mel-cepstral coefficients (MCEP) [arxiv:2105.14762 §5.2]. The card does not describe how MCEP is computed, so this page does not treat it as the MFCC recipe above.

When to use which

The card does not explain why the authors chose LFCC for their main tables, and it does not test the two features on a single controlled comparison. Check the paper's appendix D and the code before you treat the numbers as a ranking.

Implementation sketch

The MFCC frame pipeline is shown below. For LFCC, replace the mel filterbank with a linear one. The steps are in the order given by the paper.

# MFCC frame features  [§2.1] [app. C]
# Input:  wav, mono waveform  (samples,)
# Output: features  (T, 60)  # 20 MFCC + delta + double delta

def mfcc_features(wav):
    # Resample to 16 kHz and remove long silences  [app. C]
    wav = remove_silence(resample(wav, 16_000))
    # Magnitude of the short-time Fourier transform  [eq. 2]
    # The paper gives no FFT size, window or hop  [app. C]
    magnitude = abs(stft(wav))  # (K, T)
    # Triangular filterbank on the mel scale  [eq. 1] [eq. 2]
    # The paper does not state the band count  [app. C]
    bands = mel_filterbank(magnitude)  # (S, T)
    # Log of the band energies, then a DCT keeping 20 coefficients  [eq. 3]
    coeffs = dct(log(bands), n_coeffs=20)  # (20, T)
    # Central-difference deltas and double deltas  [eq. 4]
    # The paper leaves the window N open; the code uses 5  [code: dfadetect/datasets.py#L192]
    deltas = delta(coeffs)  # (20, T)
    double_deltas = delta(deltas)  # (20, T)
    return concat([coeffs, deltas, double_deltas]).T

• The LFCC variant replaces mel_filterbank with 20 linear bands and adds a small offset inside the logarithm; both details are from the code, not the paper lfcc.py lfcc.py.
• The downstream GMM and the score are the same for both features 2111.02813 §4.1 eq. 5. See audio-deepfake-detection for those steps.

Contradictions

No paper reports a conflicting MFCC result. Two points to keep in mind:

• The MFCC and LFCC numbers from WaveFake are reported in separate tables and are not a like-for-like ranking; see When to use which.
• WaveFake's aEER averages do not match a recomputation from its per-set table. Averaging the FB-MelGAN row of tab. 2 gives about 0.058, where the paper states 0.062 2111.02813 §4.1 tab. 2. The averaging rule is not stated, so the 0.062 figure is used here as the paper reports it.

Papers

• WaveFake: A Data Set to Facilitate Audio Deepfake Detection: defines the MFCC and LFCC recipes used for its GMM baselines and reports their results.
• Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection: uses MFCCs as one of several speech features in a face-plus-speech emotion detector; reports no MFCC-only result.
• Emotional Voice Conversion: Theory, Databases and ESD: includes MFCCs in the openSMILE vector of its speech emotion recognizer; reports accuracy for the whole vector only. Its MCEP voice-conversion feature is not treated as MFCC here.