Voice conversion

Overview

Voice conversion changes properties of a speech signal, here the emotion expressed, while the spoken content stays the same. Emotional voice conversion (EVC) is the variant in which the target property is the emotion, for example turning a neutral sentence into an angry one 2105.14762 §1.

The paper frames the main obstacle as data rather than model design. EVC training data should be parallel (the same content in different emotions) and multi-speaker if the model is to be speaker-independent 2105.14762 §1 2105.14762 §3.3. Most public emotional speech sets lack one or more of: large lexical variety, many speakers, both genders, a second language, or clean recordings 2105.14762 §3.1 2105.14762 tab. 1. Card: Emotional Voice Conversion: Theory, Databases and ESD.

The invariant the methods aim for is that the converted features keep the source content while moving toward the target emotion. Conversion quality is checked against the target by mel-cepstral distortion (MCD, dB; lower is better) and by listening tests 2105.14762 §5.2 2105.14762 §5.3.

Variants

Only one paper in the corpus lists this concept. The two conversion methods below are the ones that paper benchmarks. Each is described from that paper only; their original papers have no card here.

CycleGAN-EVC

A pair of generators with adversarial, cycle-consistency and identity-mapping losses, trained without parallel data 2105.14762 §2.3.1 2105.14762 fig. 3. In the benchmark it is trained once per neutral-to-emotion pair, with no frame alignment 2105.14762 §5.1. Card: Emotional Voice Conversion: Theory, Databases and ESD.

Reported on ESD for speaker 0013: lower MCD than VAWGAN-EVC in all four pairs (4.32 to 4.68 dB), higher listener quality scores in all four pairs, and more listener votes for emotion similarity in three of four pairs 2105.14762 tab. 5 2105.14762 fig. 11 2105.14762 fig. 12. Training took nearly 72 h on one GTX 1080 Ti 2105.14762 §5.4.

VAWGAN-EVC

A variational auto-encoder with a conditional generator and a discriminator, where the emotion is given to the model as a condition code [arxiv:2105.14762 §5.1] 2105.14762 fig. 4. It is the authors' earlier method and the second benchmark baseline. Card: Emotional Voice Conversion: Theory, Databases and ESD.

Reported on ESD for the same speaker: higher MCD than CycleGAN-EVC in all four pairs (5.31 to 5.89 dB), lower listener quality scores in all four pairs, and fewer emotion-similarity votes in three of four pairs 2105.14762 tab. 5 2105.14762 fig. 11 [arxiv:2105.14762 fig. 12]. Training took about 7 h in the same setup 2105.14762 §5.4.

When to use which

Caveats from the paper. Listener judgements may not match the emotion the speaker intended, which the authors leave to future work [arxiv:2105.14762 §5.4]. The MOS values are read from a bar chart and are approximate 2105.14762 fig. 11. The objective and subjective numbers come from one speaker and four neutral-to-emotion pairs.

Implementation sketch

The common core of both variants: one model per emotion pair, operating on per-frame spectral features. Where the paper gives no value, the sketch says so.

# Build one model per neutral -> target-emotion pair  [arxiv:2105.14762 §5.1]
# Features: 24-dimensional mel-cepstral coefficients per frame
# [arxiv:2105.14762 §5.2]
pairs = [("neutral", e) for e in ["angry", "happy", "sad", "surprise"]]

# Data: source and target utterances need not be frame-aligned
# [arxiv:2105.14762 §5.1]
src_frames, tgt_frames = load_mcep(pair)

# Train: adversarial, cycle-consistency and identity-mapping losses
# (CycleGAN-EVC) [arxiv:2105.14762 §2.3.1]
# Weights and optimizer are not stated in the paper
# [arxiv:2105.14762 §5.1]
model = train_cyclegan_evc(src_frames, tgt_frames)

# Convert: map source spectral frames to the target emotion
# [arxiv:2105.14762 §5.1]
converted = model.convert(src_frames)

# Evaluate: mel-cepstral distortion in dB against the target
# [arxiv:2105.14762 §5.2]
mcd = mel_cepstral_distortion(converted, tgt_frames)

The paper does not state how F0 is converted in the benchmark. Its F0 normalisation (log-F0 statistics per gender) is described in 2105.14762 §4.3.3, so do not pool genders when normalising F0.

Contradictions

None known. Only one card lists this concept, so there is nothing to compare across papers.

Papers

• Emotional Voice Conversion: Theory, Databases and ESD: introduces the ESD emotional speech set and benchmarks CycleGAN-EVC against VAWGAN-EVC on it.