Emotional Voice Conversion: Theory, Databases and ESD

In plain words

Computers that change the emotion in a recorded voice, for example making a neutral sentence sound angry, need examples of the same sentence spoken in different moods. Most public emotional speech collections do not offer this: the sentences differ between moods, the speakers are few or all of one sex, or the recordings are noisy.

This paper introduces ESD, a studio-recorded collection in which 20 actors, half speaking English and half Mandarin Chinese, read the same set of short sentences in five moods: neutral, happy, angry, sad and surprised. Because the sentences match across moods, a model can learn what changes when the mood changes. The authors also show two existing voice-conversion methods trained on ESD and a simple emotion classifier that reaches high accuracy on it.

TL;DR

• Survey of emotional voice conversion (EVC) methods and 19 emotional speech databases, with a comparison table tab. 1.
• ESD: 20 speakers (10 English, 10 Chinese), 5 emotions, parallel acted speech, about 29 hours §4.2 tab. 3.
• Reference results on ESD: speech emotion recognition (SER) 92.0% (Chinese) and 89.0% (English) fig. 9; CycleGAN-EVC has lower MCD than VAWGAN-EVC for all four neutral-to-emotion pairs tab. 5 and higher listener ratings fig. 11 fig. 12.
• Use it when you need parallel, multi-speaker, bilingual emotional speech; it is small (about 350 sentences per speaker per emotion) and the paper's own numbers are author-reported.

Problem

Emotional voice conversion needs training data that is parallel (same content, different emotion) and multi-speaker for speaker-independent models §1 §3.3. Existing databases lack one or more of: large lexical variety, many speakers, both genders, a second language, and clean recording conditions. Many are also improvised or contain laughter, sighs and external noise, which act as confounds for conversion §3.1 §3.4 §3.5 tab. 1.

The paper frames the problem as a gap in data rather than in methods: the authors state that the lack of a large, multi-speaker, open emotional speech database limits EVC research, and that few studies train speaker-independent EVC §2 §3.

Method

Dataset design (ESD)

• Speakers. 10 native English and 10 native Chinese speakers, all in the age group 25–35, with a balanced gender setting; Chinese speakers use standard Mandarin, English speakers use North American English §4.2 [§4.2 items 3–4]. Speaker IDs 0001–0010 are Chinese and 0011–0020 English fig. 6.

• Acting, parallel text. Speakers act each sentence in each of five emotions (neutral, happy, angry, sad, surprise) with the same linguistic content §4.1 §4.2. The paper calls this acted, parallel data and contrasts it with improvised databases §4.1.

• Recording. Typical indoor environment, SNR above 20 dB, sampled at 16 kHz §4.2 item 5.

• Text. Chinese: 350 sentences with 939 unique characters across a speaker's 1750 utterances; English: 997 unique words §4.3.1 tab. 3. Average 11.5 characters per Chinese utterance and 6.31 words per English utterance §4.3.1 fig. 7.

• Size. 350 utterances per speaker per emotion, so 1750 per speaker and 35,000 in total across 20 speakers tab. 2 tab. 3. Total duration is 56,385 s for Chinese and 48,370 s for English, about 29 hours tab. 3 [abstract].

• Partition per speaker and emotion tab. 2 §4.2.1:

  Set	Utterances per speaker per emotion	Selection
  Evaluation	20	first 20 utterances
  Test	30	next 30 utterances
  Training	300	remaining utterances

• Directory layout. Data/<speaker>/ with one sub-folder per emotion, each holding Evaluation Set, Test Set and Training Set folders of wav files, plus a transcription text file per speaker and a ReadMe at the root fig. 6 §4.2.2.

Speech emotion recognition used for validation

• Trained on the training partition, evaluated on the test partition §4.4 tab. 2.
• Features: openSMILE frame-level features with a 25 ms sliding window and 10 ms shift, giving 312-dimensional vectors that include zero-crossing rate, voicing probability, MFCCs and the mel-spectrum §4.4.
• Model: one LSTM layer, followed by a fully connected layer with 256 ReLU units, dropout with keep probability 0.5 on the LSTM output, and a softmax over 5 emotions §4.4.
• Output: a confusion matrix per language fig. 9 and t-SNE plots of the deep features for 20 utterances per emotion fig. 10 §4.4.

Emotional voice conversion benchmark

• Speaker used: 0013, one male speaker §5.1.
• Conversion pairs: neutral to angry, happy, sad and surprise. Each pair has its own model; no frame alignment is used, so no parallel training is needed §5.1.
• Baselines, both from the authors' earlier work §5.1:
  • CycleGAN-EVC §2.3.1 fig. 3: a generator pair with adversarial, cycle-consistency and identity-mapping losses, trained without parallel data.
  • VAWGAN-EVC §5.1 fig. 4: a variational auto-encoder with a conditional generator, a discriminator and an emotion code as condition.
• Features: 24-dimensional mel-cepstral coefficients (MCEP) for spectrum §5.2.
• Evaluation: objective mel-cepstral distortion (MCD, dB) against the target §5.2 tab. 5; subjective 5-point mean opinion score (MOS) for quality and an XAB preference test for emotion similarity §5.3 fig. 11 fig. 12.
• A "zero effort" row in tab. 5 compares source and target with no conversion.

CycleGAN-EVC loss as implemented in the authors' repository

The paper names the losses in words §2.3.1 fig. 3 but does not write them out. The loss below is taken from the code and is not stated in the paper.

# CycleGAN-EVC generator objective  [fig. 3] [code: model.py#L56-L72]
# Inputs: real features A (neutral) and B (emotion), both (24, frames)
# Output: generator loss, discriminator loss

def generator_objective(real_a, real_b):
    # Translate each domain to the other  [code: model.py#L44-L48]
    fake_b = G_a2b(real_a)
    fake_a = G_b2a(real_b)
    # Translate back to reconstruct the input  [code: model.py#L45-L48]
    cycle_a = G_b2a(fake_b)
    cycle_b = G_a2b(fake_a)
    # Adversarial terms, least-squares form  [code: model.py#L68-L69]
    adv = l2_loss(ones, D_b(fake_b)) + l2_loss(ones, D_a(fake_a))
    # L1 cycle loss and L1 identity loss  [code: model.py#L57] [code: model.py#L60]
    cyc = l1(real_a, cycle_a) + l1(real_b, cycle_b)
    idt = l1(real_a, G_b2a(real_a)) + l1(real_b, G_a2b(real_b))
    # Weighted sum; identity weight is set to 0 after 10,000 iterations
    # [code: model.py#L72] [code: train.py#L98-L99]
    return adv + lambda_cycle * cyc + lambda_identity * idt

Hyperparameters

Values stated in the ESD paper (dataset and benchmark):

Values for the CycleGAN-EVC baseline, taken from the authors' repository and not stated in the ESD paper. They come from the code of Zhou et al. 2020 (Odyssey), which the ESD paper cites as its CycleGAN-EVC §5.1 [ref. list]:

Not stated anywhere in the paper: SER optimizer, learning rate, number of epochs, and how frame-level outputs are pooled to an utterance decision; the VAWGAN-EVC hyperparameters; the number of training iterations used in the ESD experiments.

Results

All numbers are author-reported on ESD, with the setup stated.

Speech emotion recognition (training partition, test partition) fig. 9:

Per-class recalls are read from the confusion-matrix cells fig. 9.

Objective: MCD in dB, lower is better tab. 5, speaker 0013, 350 utterances per emotion pair:

Subjective: MOS, 5-point, 11 listeners fig. 11. The reference recording scores about 4.6–4.7 on all pairs. CycleGAN-EVC scores higher than VAWGAN-EVC on all four pairs; values read from the bar chart are roughly 3.6–3.9 for CycleGAN-EVC and 2.5–3.1 for VAWGAN-EVC. The paper gives no table for these values.

XAB emotion similarity (share of listener choices) fig. 12:

Training cost, same hardware §5.4: CycleGAN-EVC took nearly 72 h on one GeForce GTX 1080 Ti; VAWGAN-EVC took about 7 h in the same setup.

The paper also reports that deep emotional features cluster by emotion in a 2-D t-SNE plot for the 20 utterances per emotion fig. 10 §4.4. It does not give a number for this.

Paper vs code

• Dataset release. The paper says ESD "is made available for research purpose upon request" and links a GitHub page in footnote 2 §4.2. The paper does not state a license for the audio. The linked repository HLTSingapore/Emotional-Speech-Data reports the MIT license through the GitHub API. That repository was not cloned or read: this card uses only the paper, so no dataset content is used here. Check the dataset's own terms before any use of the audio.
• CycleGAN-EVC code. Searched: footnote 3 of the paper links KunZhou9646/emotional-voice-conversion-with-CycleGAN-and-CWT-for-Spectrum-and-F0. It is cloned at fc8f22a79654d80ecb0cacb39142da1856242b1b. The repository has no LICENSE file (GitHub API license: null), so it is cited by path and line only; nothing is quoted. The README describes it as the implementation of the Odyssey 2020 paper (arXiv:2002.00198), which the ESD paper cites as the CycleGAN-EVC baseline §5.1 [ref. list].
• VAWGAN-EVC code. Footnote 4 links KunZhou9646/Speaker-independent-emotional-voice-conversion-based-on-conditional-VAW-GAN-and-CWT. The GitHub API shows no license. It was not cloned or read, so no code claim is made for VAWGAN-EVC.
• Hyperparameters. The paper states none for its EVC runs. The values in the table above come from train.py and model.py in the repository and apply to that repository's defaults. They are not confirmed as the settings used for the ESD benchmark. Cite them as "code defaults", not paper values.
• Default data paths. The repository's defaults train NEUTRAL to SURPRISE with evaluation_all folders and name the output neutral_to_suprise train.py. These paths are the only example pair in the code. The paper's benchmark covers four pairs with speaker 0013.
• F0 handling. The README says the code converts spectrum and CWT-encoded F0 README.md. In train.py, F0 is converted by mapping log-F0 mean and standard deviation between domains, not by CWT train.py train.py. CWT appears only in the separate train_f0.py, which was not read. The ESD paper does not say which F0 path its benchmark used.
• SER. No code is cited for the SER model. The paper gives its feature set and layer sizes §4.4 but not optimizer or training schedule.

Pitfalls

• "350 utterances per speaker" is ambiguous. Section 4.2 and the abstract read as if each speaker contributes 350 utterances in total §4.2. Table 2 and Table 3 show 350 per speaker per emotion, so 1750 per speaker. Use the tables.
• Text counts differ slightly from the table totals. The text gives 6.31 words per English utterance §4.3.1. Dividing the 11,015 words in the text by 1750 utterances gives about 6.29. Expect small rounding differences in any derived statistic.
• Parallel text is within a speaker. The parallel pairs are the same sentence spoken in the five emotions by one speaker §4.2. The paper does not state whether speakers share sentences, so do not assume cross-speaker alignment.
• Small benchmark. The EVC benchmark uses one male speaker §5.1. The paper reports no speaker-independent conversion results, although the database is described as built for speaker-independent work §4.1 §6.1.
• Listener versus intent. The authors note that listener judgements may not match the speaker's intended emotion, and leave this for future work §5.4. Do not treat the XAB result as ground truth for emotion.
• Training time. CycleGAN-EVC needed about 72 h on one GPU for four pairs, and VAWGAN-EVC about 7 h §5.4. Budget accordingly.
• Figures and tables cited from the PDF only. The source was a PDF, so values read from figures (MOS bars, per-class recalls) are approximate or read from labels. Use the table values where a table exists.
• Duration statistics. Average utterance duration is 2.76 s for English and 3.22 s for Chinese, and per-emotion averages are in tab. 3. Sad utterances are the longest on average for both languages tab. 3 fig. 8.
• F0 differences. Female speakers have higher mean F0 and standard deviation than male speakers within every emotion category tab. 4. Do not pool genders when normalising F0 §4.3.3.

Lineage

• Builds on: CycleGAN-EVC and VAWGAN-EVC, the two conversion methods used as benchmarks §5.1 fig. 3 fig. 4; both are the authors' earlier work.
• Compared against: 19 existing emotional speech databases on lexical, language, speaker, confounding-factor and recording criteria tab. 1 §3.
• Superseded by: not stated in this card's sources.