---
kind: concept
slug: voice-conversion
title: Voice conversion
summary: >-
  Changing the emotion (or voice) of a recording while keeping the words. Emotional voice conversion needs parallel, multi-speaker emotional speech, which the ESD set provides. On ESD, CycleGAN-EVC beats VAWGAN-EVC on MCD and listening tests, at about ten times the training time. Author-reported, one speaker.
tags: [speech, audio, emotion, dataset]
papers: ["2105.14762"]
related: [generative-adversarial-network, mel-frequency-cepstral-coefficients, spectrogram, audio-deepfake-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

Voice conversion changes properties of a speech signal, here the emotion
expressed, while the spoken content stays the same. Emotional voice conversion
(EVC) is the variant in which the target property is the emotion, for example
turning a neutral sentence into an angry one [arxiv:2105.14762 §1].

The paper frames the main obstacle as data rather than model design. EVC
training data should be parallel (the same content in different emotions) and
multi-speaker if the model is to be speaker-independent [arxiv:2105.14762 §1]
[arxiv:2105.14762 §3.3]. Most public emotional speech sets lack one or more of:
large lexical variety, many speakers, both genders, a second language, or clean
recordings [arxiv:2105.14762 §3.1] [arxiv:2105.14762 tab. 1]. Card:
[Emotional Voice Conversion: Theory, Databases and ESD](/papers/2105.14762).

The invariant the methods aim for is that the converted features keep the
source content while moving toward the target emotion. Conversion quality is
checked against the target by mel-cepstral distortion (MCD, dB; lower is
better) and by listening tests [arxiv:2105.14762 §5.2] [arxiv:2105.14762 §5.3].

## Variants

Only one paper in the corpus lists this concept. The two conversion methods
below are the ones that paper benchmarks. Each is described from that paper
only; their original papers have no card here.

### CycleGAN-EVC

A pair of generators with adversarial, cycle-consistency and identity-mapping
losses, trained without parallel data [arxiv:2105.14762 §2.3.1]
[arxiv:2105.14762 fig. 3]. In the benchmark it is trained once per
neutral-to-emotion pair, with no frame alignment [arxiv:2105.14762 §5.1].
Card: [Emotional Voice Conversion: Theory, Databases and ESD](/papers/2105.14762).

Reported on ESD for speaker 0013: lower MCD than VAWGAN-EVC in all four pairs
(4.32 to 4.68 dB), higher listener quality scores in all four pairs, and more
listener votes for emotion similarity in three of four pairs [arxiv:2105.14762 tab. 5]
[arxiv:2105.14762 fig. 11] [arxiv:2105.14762 fig. 12]. Training took nearly
72 h on one GTX 1080 Ti [arxiv:2105.14762 §5.4].

### VAWGAN-EVC

A variational auto-encoder with a conditional generator and a discriminator,
where the emotion is given to the model as a condition code [arxiv:2105.14762
§5.1] [arxiv:2105.14762 fig. 4]. It is the authors' earlier method and the
second benchmark baseline. Card: [Emotional Voice Conversion: Theory, Databases and ESD](/papers/2105.14762).

Reported on ESD for the same speaker: higher MCD than CycleGAN-EVC in all four
pairs (5.31 to 5.89 dB), lower listener quality scores in all four pairs, and
fewer emotion-similarity votes in three of four pairs
[arxiv:2105.14762 tab. 5] [arxiv:2105.14762 fig. 11] [arxiv:2105.14762
fig. 12]. Training took about 7 h in the same setup [arxiv:2105.14762 §5.4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Quality and emotion similarity matter most, and days of GPU time are acceptable | CycleGAN-EVC | Lower MCD and higher listener scores on ESD [arxiv:2105.14762 tab. 5] [arxiv:2105.14762 fig. 11] |
| Training budget is a few hours on one GPU | VAWGAN-EVC | About 7 h against about 72 h [arxiv:2105.14762 §5.4] |
| Neutral-to-sad conversion | Either, with caution | Emotion similarity did not favour either method: 29% vs 29%, with 42% "no difference" [arxiv:2105.14762 fig. 12] |
| Need a model that works across speakers | No result in this paper | The benchmark uses one male speaker and reports no speaker-independent conversion [arxiv:2105.14762 §5.1] [arxiv:2105.14762 §6.1] |
| No emotional data yet, need a starting set | ESD as a reference set | Acted, parallel, 20 speakers, 5 emotions, about 29 h [arxiv:2105.14762 §4.2] [arxiv:2105.14762 tab. 3] |

Caveats from the paper. Listener judgements may not match the emotion the
speaker intended, which the authors leave to future work [arxiv:2105.14762
§5.4]. The MOS values are read from a bar chart and are approximate [arxiv:2105.14762 fig. 11].
The objective and subjective numbers come from one speaker and four
neutral-to-emotion pairs.

## Implementation sketch

The common core of both variants: one model per emotion pair, operating on
per-frame spectral features. Where the paper gives no value, the sketch says so.

```pseudocode
# Build one model per neutral -> target-emotion pair  [arxiv:2105.14762 §5.1]
# Features: 24-dimensional mel-cepstral coefficients per frame
# [arxiv:2105.14762 §5.2]
pairs = [("neutral", e) for e in ["angry", "happy", "sad", "surprise"]]

# Data: source and target utterances need not be frame-aligned
# [arxiv:2105.14762 §5.1]
src_frames, tgt_frames = load_mcep(pair)

# Train: adversarial, cycle-consistency and identity-mapping losses
# (CycleGAN-EVC) [arxiv:2105.14762 §2.3.1]
# Weights and optimizer are not stated in the paper
# [arxiv:2105.14762 §5.1]
model = train_cyclegan_evc(src_frames, tgt_frames)

# Convert: map source spectral frames to the target emotion
# [arxiv:2105.14762 §5.1]
converted = model.convert(src_frames)

# Evaluate: mel-cepstral distortion in dB against the target
# [arxiv:2105.14762 §5.2]
mcd = mel_cepstral_distortion(converted, tgt_frames)
```

The paper does not state how F0 is converted in the benchmark. Its F0
normalisation (log-F0 statistics per gender) is described in
[arxiv:2105.14762 §4.3.3], so do not pool genders when normalising F0.

## Contradictions

None known. Only one card lists this concept, so there is nothing to compare
across papers.

## Papers

- [Emotional Voice Conversion: Theory, Databases and ESD](/papers/2105.14762):
  introduces the ESD emotional speech set and benchmarks CycleGAN-EVC against
  VAWGAN-EVC on it.