Age Progression/Regression by Conditional Adversarial Autoencoder

In plain words

Most face-aging systems need photos of the same person at different ages, and they need to know the age of the face you give them. This paper asks whether a model can learn the general look of faces without such pairs and then change only the age of a given face.

The model squeezes a face into a short list of numbers that stands for who the person is, and a second network draws a face from those numbers plus a chosen age. Because the age is a separate input, you can keep the numbers fixed and turn the age up or down, and the face keeps its identity. Two adversarial critics push the numbers toward an evenly spread space and push the drawn faces toward looking real for their age. The paper's evidence is a human study that asks whether the aged faces look like the same person.

TL;DR

• A conditional adversarial autoencoder (CAAE): an encoder E maps a face to a personality vector z, and a generator G renders it given an age label §4.1.
• Training needs no paired images of one person at different ages, and no age label on the test face §1 §5.2.
• Two discriminators do the work: one keeps z close to a uniform prior, the other judges realism conditioned on age §4.3 §4.4.
• Results are a user study, not a face-recognition score: 52.77% of votes prefer CAAE over prior work §5.3.
• Most architecture and optimiser values are stated. Reproducing the results needs a training set the paper only partly describes, and several weights are missing (see Pitfalls).

Problem

Face aging was mostly done with group-based or physical models. Prototype-based methods split training data into age groups and learn a transformation between adjacent groups. This needs paired images across short time spans, and it needs the true age of the query face §1 §2.1. Physical models need long, dense age spans and are costly to compute §2.1. Rejuvenation to baby faces was especially weak in prior work §1. The paper assumes that faces lie on a manifold, and that moving along an age direction on that manifold gives smooth aging while keeping identity §3.

Method

The paper gives no single algorithm box. It describes one network with four sub-networks, and the training steps are written as equations.

1. Encode. A convolutional encoder E maps a 128 x 128 x 3 face x to a vector z = E(x) that carries the personal features §4.1. Strided convolution replaces pooling §4.1. The output is bounded by tanh to [-1, 1] §5.2. The figure labels the vector as 50 long fig. 3.
2. Attach age. The age label l is one-hot over 10 bins §5.1. It is concatenated to z, and the result [z, l] goes to G §4.1 fig. 3.
3. Decode. A deconvolutional generator G maps [z, l] to a face x̂ = G(z, l) with the same size §4.1.
4. Reconstruct. The encoder and generator are trained so that the output matches the input under the L2 norm §4.2 eq. 2. The full objective adds a total-variation term and two adversarial terms §4.2 eq. 5.
5. Regularise z. Discriminator D_z tries to tell samples from a uniform prior apart from codes E(x). The encoder is trained to fool it, so the codes fill the latent space with no holes §4.3 eq. 3 fig. 5.
6. Judge images. Discriminator D_img sees real faces with their age label and generated faces with the same label, and learns to tell them apart. The generator is trained to fool it. The age label is concatenated after the first convolution of D_img §4.4 eq. 4 §4.2.
7. Update. The paper says the networks are updated one after another, in the order of eq. 2, eq. 3, then eq. 4 §4.2.
8. Age at test time. Only E and G run. E maps an input face to z, with no true age needed. Any target age label is appended, and G renders the face §5.2. Keeping z fixed and changing only the age label traverses the age axis §3 fig. 2.

The objective, as the paper writes it, is a min-max over the four networks:

\min_{E,G}\; \mathcal{L}\left(x,\, G(E(x),l)\right) eq. 2

\min_{E}\;\max_{D_z}\; \mathbb{E}_{z^*\sim p(\mathbf{z})}\left[\log D_z(z^*)\right] + \mathbb{E}_{x\sim p_{data}}\left[\log\left(1-D_z(E(x))\right)\right] eq. 3

\min_{G}\;\max_{D_{img}}\; \mathbb{E}_{x,l}\left[\log D_{img}(x,l)\right] + \mathbb{E}_{x,l}\left[\log\left(1-D_{img}(G(E(x),l))\right)\right] eq. 4

\min_{E,G}\;\max_{D_z,D_{img}}\; \lambda\,\mathcal{L}\left(x,G(E(x),l)\right) + \gamma\, TV\left(G(E(x),l)\right)

+ \mathbb{E}_{z^*\sim p(\mathbf{z})}\left[\log D_z(z^*)\right] + \mathbb{E}_{x\sim p_{data}}\left[\log\left(1-D_z(E(x))\right)\right]

+ \mathbb{E}_{x,l}\left[\log D_{img}(x,l)\right] + \mathbb{E}_{x,l}\left[\log\left(1-D_{img}(G(E(x),l))\right)\right] eq. 5

The paper defines TV as the total variation that "is effective in removing the ghosting artifacts" and gives no formula §4.2. Equation 1 in the paper is the original GAN objective, quoted only as background §2.2.

Training step, condensed from the equations. The paper has no algorithm box, so this is our arrangement, not the paper's:

# Condensed training step, following eq. 2 to eq. 5  [§4.2]
# images: (B, 128, 128, 3) in [-1, 1]
# ages: (B, 10) one-hot labels, values in {-1, 1}  [§5.2]
def training_step(images, ages, z_prior):
    # Encode each face to its personality vector  [§4.1]
    codes = encoder(images)                    # (B, n); n = 50 in [fig. 3]
    # Decode the codes with the age label appended  [§4.1]
    recons = generator(concat(codes, ages))    # (B, 128, 128, 3)

    # Encoder and generator losses, as in eq. 5  [§4.2]
    # Reconstruction is L2 in the text  [§4.2]
    recon_loss = 100 * l2(images - recons)     # lambda = 100  [§5.2]
    smooth_loss = 10 * total_variation(recons) # gamma = 10  [§5.2]
    fool_z = log(1 - D_z(codes))               # encoder side of [eq. 3]
    fool_img = log(1 - D_img(recons, ages))    # generator side of [eq. 4]
    eg_loss = recon_loss + smooth_loss + fool_z + fool_img

    # Discriminator on codes: the uniform prior counts as real  [§4.3]
    d_z_loss = -log(D_z(z_prior)) - log(1 - D_z(codes))
    # Discriminator on images, conditioned on the age label  [§4.4]
    d_img_loss = -log(D_img(images, ages)) - log(1 - D_img(recons, ages))

    # Update in the order the paper gives  [§4.2]
    update(encoder_and_generator, eg_loss)
    update(discriminator_z, d_z_loss)
    update(discriminator_img, d_img_loss)
    return eg_loss, d_z_loss, d_img_loss

Test-time use, from the description in §5.2:

# Age a face at test time  [§5.2]
def age_face(image, target_age):
    # Personality code from the face; no true age is needed  [§5.2]
    code = encoder(image)
    # Append the desired age bin and decode  [§5.2]
    return generator(concat(code, target_age))

Hyperparameters

Values are from the paper unless the "Where" column says code. "Not stated" means the paper does not give it in the text, figures or tables we checked.

Results

Author-reported. The paper has no table. Its numbers are vote shares from user studies. We found no face-recognition or image-quality metric in the text.

Setup for all rows: training data are about 3,000 images from Morph and CACD plus 7,670 crawled images, 10,670 in total, with ages and genders estimated from captions or an age estimator §5.1. Evaluation uses FGNET: 1,002 images of 82 subjects aged 0 to 69 §5.3.

Paper vs code

The official repository is https://github.com/ZZUTK/Face-Aging-CAAE, linked from the paper's footnote §1 and from the authors' project page, at commit cb5a5b21768543d6e13c9af123346eeeccb1ff66. It has no LICENSE file, so code_license says so. We quote no code from it; the citations below are for facts only. The paper's Bitbucket link returned 404 when we checked. The project page names the same GitHub repository as the code.

Differences, each with both citations:

• Reconstruction loss. Paper: L2 norm §4.2 fig. 3. Code: mean absolute error (L1) FaceAging.py. The L2 form is commented out in the code FaceAging.py.
• Loss weights. Paper: lambda = 100 and gamma = 10 on the reconstruction and TV terms §5.2. Code default: the adversarial term on G gets weight 0.0001, the encoder adversarial term and the TV term get 0, and the reconstruction term is unweighted FaceAging.py FaceAging.py. A comment in the code says the weights may differ from the paper because of the dataset FaceAging.py.
• Gender label. Paper: the input label is age only §4.1 §5.1. Code: a 2-value gender label is also given to G and D_img FaceAging.py FaceAging.py FaceAging.py.
• Encoder and generator depth. Figure 3 shows four encoder convolutions ending at 8 x 8 x 512 and a generator that starts at 8 x 8 x 1024 fig. 3. Code uses five stride-2 encoder convolutions, ending at 4 x 4 x 1024, and a generator that starts at 4 x 4 x 1024 FaceAging.py FaceAging.py FaceAging.py. The two disagree and we cannot say which the released weights follow.
• Activation in D_img. Paper: ReLU in all intermediate layers of each block §5.2. Code: leaky ReLU with slope 0.2 for the hidden fully connected layer of D_img FaceAging.py ops.py.
• Batch norm in D_z. Paper: silent on D_z §5.2. Code: batch norm on the three hidden layers of D_z FaceAging.py.
• Update order. Paper: "sequentially updating" the networks in the order of eq. 2, 3, 4 §4.2. Code: the three optimisers run in one session.run call, so the discriminator steps are computed from the same forward pass as the encoder and generator step FaceAging.py.
• Adversarial loss form. Paper: the minimax form of eq. 3 and eq. 4 §4.2. Code: the encoder and generator adversarial terms use the non-saturating form (target label 1) FaceAging.py.
• Prior bounds. Paper: uniform, bounds not stated in the text §4.3. Code: U[-1, 1] FaceAging.py.
• Initial weights. The repository ships an initial model to start training from README.md. The paper does not mention it.
• Training data. Paper: Morph, CACD and crawled images, 10,670 in total §5.1. Code README: UTKFace, 23,708 images README.md README.md. The repository does not include the paper's training set.
• Test protocol. Code: each test face is processed twice, once as male and once as female README.md. The paper does not describe this.
• Compute. Not stated in the paper. Code README: about 2.5 hours for 50 epochs on UTKFace on one NVIDIA TITAN X (12 GB) README.md.

Pitfalls

• Latent size is only in the figure. The text never gives n. Figure 3 gives 50 fig. 3, and the code default is 50 FaceAging.py.
• Reconstruction norm. L2 in the text, L1 in the code. Implement one and state which.
• Total variation is undefined. The paper gives no formula. The code's TV is a normalised squared difference FaceAging.py. Write TV from an explicit formula, and say which one you chose.
• Adversarial weights are not stated. Eq. 5 has no weights on the adversarial terms, while lambda and gamma apply to the other two terms. The code's effective weights differ (see Paper vs code). Expect to tune.
• Saturating encoder loss. The minimax form of eq. 3 and eq. 4 saturates when the discriminator is confident. The code uses the non-saturating form FaceAging.py. The paper does not discuss this choice.
• Gender is absent from the paper. A reader following the paper would not add it. The code uses it FaceAging.py FaceAging.py.
• Label noise. Ages and genders for crawled images come from captions or an age estimator §5.1. The paper does not report label accuracy.
• Vote counts do not add up. Volunteers answered 45 questions each, with 63 volunteers, which allows at most 2,835 votes. The paper reports 3,208 §5.3. The paper does not explain this.
• The evidence is a user study on identity. Volunteers judged whether the output and the ground truth look like the same person §5.3. The paper reports no measure of whether outputs show the requested age.
• Ghosting and blur. The paper says TV removes ghosting artifacts §4.2 and that D_img improves texture, especially for older faces §4.4. Both are shown qualitatively fig. 6 only.
• Scale. The paper uses about 10,670 images at 128 x 128 §5.1. Robustness to pose, expression and occlusion appears in one figure with no numbers §5.4 fig. 10.
• Preprocessing matters. Faces are aligned with a 68-landmark detector §5.1. The paper does not say what happens to unaligned faces.

Lineage

• Builds on: the original GAN [goodfellow2014generative] for adversarial training §2.2; the adversarial autoencoder of Makhzani et al. (ICLR 2016) [Alireza in the bibliography], which CAAE adds two discriminators to §4.5; DCGAN for stride convolutions and batch norm §4.1 §2.2; CGAN for conditioning on labels §2.2.
• Compared against: Wang et al. (2016) recurrent face aging; Kemelmacher-Shlizerman et al. (2014) illumination-aware age progression; Shu et al. (2015) personalised age progression with a dictionary; Suo et al. (2010) compositional modelling; Face Transformer (FT) for age regression §5.3.
• Superseded by: not known from the content of this wiki.