Talking Face Generation by Conditional Recurrent Adversarial Network

In plain words

Give the system one photo of a person and a recording of someone speaking, and it produces a video of that face moving its mouth in time with the sound. Two things make this hard: the face has to stay steady from one frame to the next, and the lips have to form the right shapes for the sounds.

The authors feed the sound and the face into a recurrent network, which passes information from one time step to the next, so each frame is produced with the earlier frames in mind. Three critics, called discriminators, judge the output. Two check whether frames and short clips look like real video. The third is a lip-reading model that checks whether the mouth shapes would be read as the right word. Training pushes the output to look real and to read correctly.

TL;DR

• Input: one identity face image and an audio clip. Output: a sequence of 128 × 128 frames §4.2.1.
• Core idea: image and audio features are concatenated at each step, a recurrent unit runs over the steps, and a decoder with U-Net-style skip connections produces each frame §3.2.
• Training: an L1 reconstruction loss plus three adversarial terms (frame, 3D video, lip-reading) §3.3 eq. 5.
• Author-reported gain: on LRW, lip-reading accuracy rises from 18.5% / 38.0% to 36.2% / 63.0% (top-1 / top-5) when the lip-reading discriminator is added tab. 1.
• The released code differs from the paper in several loss weights, the training schedule and the video discriminator's training path. See Paper vs code.

Problem

Generating video frame by frame gives jitter, because the frames are not tied together over time. Sequential generation, which conditions each frame on the previous one, keeps only short-term dependence and loses the identity over long clips §1 §3.1 §4.3.2. Talking-face generation also needs accurate lip shapes, and a pixel reconstruction loss alone is said to score lip errors only as average pixel distance, not as wrong mouth movements §3.3.1. The paper frames the task as sampling a frame sequence conditioned on an audio sequence and one identity image §3.1.

Method

The paper describes the following pipeline §3.2 fig. 2.

1. Audio features. The audio is cut with a sliding window. Each window is an MFCC feature that an audio encoder maps to a vector §3.2.1. The window that gave the best result among those tried is 350 ms §4.2.1. The paper does not state the number of MFCC coefficients or the hop size.
2. Image features. For each audio window, the frame at the middle of the window's time span is taken as the image with that lip shape §3.2.2. The identity image I* can be any single frame of the sequence §3.1 §3.2.2. An image encoder maps it to a vector §3.2.2.
3. Hybrid recurrent generation. The audio and image vectors are concatenated and fed to a recurrent unit that makes the features temporally correlated. A decoder then produces the frames §3.2. The paper does not name the recurrent unit. The code uses a GRU model_G.py.
4. Decoder. The decoder adds lower-level encoder features back in, following the U-Net idea, to keep the identity for faces unseen in training §4.2.2.
5. Lip-reading discriminator. A pretrained lip-reading network reads mouth regions and is updated adversarially. It is applied only to LRW samples, because VoxCeleb and TCD-TIMIT lack word labels §3.3.1 §4.2.3.
6. Spatial-temporal discriminators. A frame discriminator improves per-image quality, and a 3D-convolution video discriminator improves smoothness across frames §3.3.2 §4.2.2.
7. No post-processing. The authors state that no separate deblurring or video stabilisation step is needed §1.
8. Pose and expression extension (single person). The previously generated frame is also fed into the recurrent input so that pose and expression can change over time. It is trained on the Obama dataset §5 fig. 6.

Objectives. Reconstruction is an L1 distance between generated and ground-truth frames eq. 1:

\mathcal{L}_{rec} = \left\| I_t - \tilde{I}_t \right\|_1 eq. 1

The lip-reading discriminator's objective is written as eq. 2:

\mathcal{L}_{l} = \sum y \log D_{l}(I_i) - \sum y \log D_{l}(\tilde{I}_i) eq. 2

where y is the real word label. The image and video discriminators use the standard GAN objective, written as eq. 3 and eq. 4:

\mathcal{L}_{I} = \mathbb{E}[\log D_I(I_t)] + \mathbb{E}[\log(1 - D_I(\tilde{I}_t))] eq. 3

\mathcal{L}_{V} = \mathbb{E}[\log D_V(I_i)] + \mathbb{E}[\log(1 - D_V(\tilde{I}_i))] eq. 4

The generator's total loss is eq. 5:

\mathcal{L} = \mathcal{L}_{rec} + \lambda_I \mathcal{L}_I + \lambda_V \mathcal{L}_V + \lambda_{l} \mathcal{L}_{l} eq. 5

The paper does not spell out the generator's form of the lip term or the generator's adversarial term. The code's forms are given in the pseudocode below and in Paper vs code.

# Conditional recurrent generation  [§3.1] [§3.2]
# Inputs:  identity_image (3, 128, 128); audio_segments (K, 12, 35)
# Output:  frames (K, 3, 128, 128)

def generate(identity_image, audio_segments):
    # Encode the identity face  [§3.2.2]
    image_feature = image_encoder(identity_image)         # (512,)
    # Encode each audio window  [§3.2.1]
    audio_features = [audio_encoder(a) for a in audio_segments]
    # Concatenate image and audio features per step  [§3.2]
    hybrid = [concat(image_feature, z) for z in audio_features]
    # Run the recurrent unit over the steps  [§3.2]
    states = recurrent_unit(hybrid)                        # (K, hidden)
    # Decode each state with U-Net skips  [§3.2] [§4.2.2]
    frames = [decoder(s, image_skips) for s in states]
    return frames

# Lip-reading discriminator loss  [§3.3.1] [eq. 2]
# Real clips use the word label; fake clips are pushed to a constant

def lip_discriminator_loss(real_logits, fake_logits, word_label):
    # Real clips: cross-entropy with the word label; label -1 is ignored
    real_term = cross_entropy(real_logits, word_label, ignore_index=-1)
    # Fake clips: masked squared error of the logits towards 0.002
    fake_term = masked_mse(fake_logits, 0.002, word_label >= 0)
    return real_term + fake_term

Hyperparameters

Results

Author-reported. Setup: LRW test set, quantitative comparison with Chung et al. (2017) and Zhou et al. (2019) §4.4 tab. 1. Chen et al. (2018) is excluded from the quantitative study because it generates only the lip region §4.4.

tab. 1. The paper reports that real videos reach about 60% / 80% top-1 / top-5 with the same lip-reader §4.4. It notes that PSNR and SSIM fall when the discriminators add detail, and that LMD may not reflect correct lip movement §4.4.

User study (Amazon Mechanical Turk; 60 samples per worker from VoxCeleb and LRW; 10 workers per sample; workers in the UK or USA) §4.5 tab. 2. Each cell is "votes to ours / votes to the other method", as the table header reads:

tab. 2

Qualitative results: the proposed method's frames show sharper skin texture and mouth shapes than the two baselines on a "high education" example §4.3.1 fig. 3. The loss ablation shows that adding each discriminator sharpens details and reduces jitter between frames §4.3.2 fig. 4. The generation-scheme comparison shows that sequential generation loses the identity and frame-to-frame generation has large variation between adjacent frames §4.3.2 fig. 5. The Obama extension is shown as a figure only §5 fig. 6.

Paper vs code

Code: https://github.com/susanqq/Talking_Face_Generation at commit 6ea80c49797d87ef3ca4b49e01d00c09fbe36bb5. The paper links this repository in its footnote §4. The repository has no LICENSE file and no license statement in the README README.md. Its PyTorch code is cite-only for this wiki.

Discrepancies found after reading the training, loss, model and run-script code:

• Image adversarial weight. Paper λ_I = 1e-3 §4.2.3. Code hard-codes 0.002 in both trainers train.py train.py.
• Video and lip weights match. λ_V = 1e-2 and λ_l = 1e-3 match the code defaults train.py.
• Lip-reading loss. Paper eq. 2 is a log-likelihood form §3.3.1. The code computes cross-entropy with the word label for real clips, ignoring label -1 loss.py loss.py. For fake clips it applies a masked squared error on the logits towards the constant 0.002 [code: loss.py#L62-L67]. For the generator it uses cross-entropy of generated clips with the true label train.py. The model returns logits and the softmax is commented out model_LipNet.py.
• Adversarial form. Paper eqs. 3–4 are written in the min-max form §3.3.2. The code's generator term uses a binary cross-entropy with target 1 on discriminator logits, which is the non-saturating form train.py train.py loss.py.
• Optimiser. Paper gives one Adam learning rate α = 0.0002 and β = 0.5 §4.2.3. The code uses betas (0.5, 0.999) and learning rate 0.001 for all discriminators train.py.
• Schedule. Paper: 30 epochs without discriminators, then 15 epochs with them §4.2.3. The released scripts run 50 epochs for the CNN variants [code: run_train_lrw.sh#L4], or 20 epochs starting from an earlier checkpoint for the lip-adversarial variants run_train_lrw.sh. No script reproduces the 30 + 15 split.
• Video discriminator in the CNN path is never updated. Its optimiser is built train.py but the CNN training step only steps the generator and the image and lip discriminators train.py. The RNN path does step it train.py.
• Recurrent unit. Paper does not name it §3.2. Code: a single-layer GRU with 1024 hidden units over concatenated image and audio features [code: model_G.py#L373] model_G.py.
• Previous-frame feedback (§5). The code's recurrent generator takes only the encoder features and the recurrent output, with no path that feeds the previous frame back in model_G.py. No Obama-specific script is in the repository.
• Image encoding per step. Paper encodes the identity image once §3.2.2. The recurrent generator encodes an image for every step model_G.py, and the test loader repeats the identity image for each audio window [code: train.py#L474-L476].
• Teacher forcing. The recurrent generator accepts teacher_forcing_ratio but never uses it in the forward pass model_G.py. The paper does not mention teacher forcing.
• Reconstruction. L1 as in the paper loss.py. For sequences the code averages only over valid, unpadded frames loss.py.
• Lip-reader architecture. Paper does not state it. Code: two-layer LSTM, 512 hidden units, 500 output classes model_D.py.
• Metric code. The repository has no PSNR, SSIM, landmark-distance or top-k accuracy code. The only landmark code is a preprocessing script get_LRW_landmarks.py.
• Audio input. The code drops the first MFCC coefficient and feeds a 12 × 35 map data.py model_G.py. The paper does not mention dropping a coefficient.

No further discrepancies were found in the L1 reconstruction, the 128 × 128 input size, the U-Net-style skip connections or the Adam β1 = 0.5 setting.

Pitfalls

• Author list. The arXiv metadata lists "Xiaolong Wang" as the fourth author. The LaTeX source of the IJCAI version lists "Andy Wang" in the same position with IBM as affiliation. This card follows the arXiv metadata.
• Old stack. The README pins Python 3.6 and PyTorch 0.4.0 and says the code is still being cleaned README.md README.md. The data loader imports scipy.misc functions dataloader.py, which current SciPy releases may not provide. Not tested here.
• Missing pieces. The lip-reader checkpoint is loaded from a path outside the repository run_train_lrw.sh. Face cropping, MFCC extraction and the list files are not included in full. The list files use absolute paths from the authors' machines TCD_training_list_128.txt.
• Identity frame. The first row of the TCD list names the same file in the input-image and ground-truth columns TCD_training_list_128.txt. The repository does not document how the identity frame is chosen per sample.
• Short clips for the video discriminator. A clip shorter than the required length is padded by repeating its last frame model_D.py.
• Lip-reader cost. The mouth crop loops over batch and frames in Python model_LipNet.py, which may limit speed on long clips. Not measured here.
• Unexplained constant. The value 0.002 in the fake-clip lip-reader loss is not explained in the paper or the code.
• Evaluation caveats. Quantitative results are reported only on LRW. TCD-TIMIT and VoxCeleb appear only in the qualitative figures and the user study §4.4 §4.5. PSNR, SSIM and LMD are weak proxies for lip accuracy, as the authors note §4.4.
• Lip-reader scope. The lip-reading discriminator works on word-level clips only. Sentence-level lip reading is listed as future work §6.
• Windowing choice. The 350 ms window was picked from the windows tried; the other windows and their results are not reported §4.2.1.

Lineage

• Builds on: GAN training §3.1; sequential video generation in Vid2Vid, which this paper contrasts with recurrent generation §2.2 §3.1.
• Compared against: Chung et al. (2017), "You said that?", a frame-independent generator §1 tab. 1; Zhou et al. (2019), an audio-visual disentangled model §2.3 tab. 1.
• Concurrent: Chen et al. (2018), lip-region generation §2.3.
• Future work: end-to-end raw-audio input, sentence-level lip-reading discriminator, and a super-resolution stage §6.