Generative Dual Adversarial Network for Generalized Zero-shot Learning

In plain words

Zero-shot learning asks a model to recognise object classes it never saw in training. It is given a description of each class instead, such as a list of attributes. The generalized version is harder: a test image can come from a seen class or an unseen one, and the model must not favour the seen ones.

GDAN handles this with one model that combines three ways of linking pictures to descriptions. A generator invents image features for a class from its description. A regressor reads an image's features and guesses its description. A judge scores whether an image and a description belong together. The generator and the regressor check each other: a generated feature should map back to its description, and a described image should come back as its features. The judge learns to tell real pairs from fakes, and the other two learn to fool it. After training, the model invents features for the unseen classes and classifies with a simple nearest-neighbour rule over real and invented examples.

TL;DR

• GDAN trains three networks together: a conditional VAE that generates visual features from class embeddings, a regressor from visual features back to embeddings, and a discriminator that scores feature-embedding pairs §3.2.
• The generator and regressor are linked by a cyclic-consistency loss, and both learn from the discriminator through adversarial losses §3.4 §3.5 eq. 3 eq. 6 eq. 5.
• For each unseen class, the model generates features; they join the real seen features, and a 1-NN classifier labels test images §3.7.
• Author-reported: highest unseen-class accuracy on SUN, CUB, AwA2 and aPY among baselines not marked with a star §4.3 tab. 2.
• No variance across seeds is reported. The source code differs from the paper in several hyperparameters and in the noise distribution (see Paper vs code).

Problem

Zero-shot classification needs a shared space in which images and class descriptions can be compared. Prior work has two common routes. Mapping visual features into the attribute space suffers from the hubness problem, where projected features cluster around a few points §1. Mapping each class's description to a single visual point ignores that one class has many visual appearances §1.

Generative methods can produce many features per class. Some lack a mapping in both directions, and some lack an adversarial metric that judges feature-description similarity §1 §2. Metric-learning methods such as RelationNet learn a similarity but do not generate features or learn latent features §1 §2.

The generalized setting is harder than conventional zero-shot learning. The test label space contains both seen and unseen classes, and the model does not know in advance which kind an image belongs to §1. The score is therefore the harmonic mean of seen and unseen accuracy §4.1 eq. 9.

Method

Notation: a seen-class training set of image features v, labels y and class embeddings s; unseen test classes are disjoint from seen ones, and the classifier predicts over the union §3.1.

Feature generation network (CVAE) §3.3. The generator is the decoder of a conditional VAE. It maps a latent noise vector z and a class embedding s to a visual feature v. The encoder is conditioned on v alone, not on s, which the authors adopt to reduce posterior collapse §3.3. The CVAE is trained on the image features (not raw images), because blurry generated images were judged poor for attribute detail §3.3. The CVAE objective is eq. 1. An adversarial term is added to the CVAE through the discriminator (see below) §3.3 eq. 6.

Regressor §3.4. The regressor R maps a visual feature to a class embedding. It is trained with a supervised squared-error loss on real pairs eq. 2 and with a cyclic-consistency loss against the CVAE eq. 3. The cycle has two parts: a feature generated from a regressed embedding should match the original feature, and an embedding regressed from a generated feature should match the original embedding eq. 3.

Discriminator §3.5. The discriminator D takes a feature-embedding pair and outputs a compatibility score. It is trained on four kinds of pairs §3.5 eq. 4:

• real pairs (v, s), pushed toward 1;
• generated pairs (G(s, z), s), where z comes from the encoder applied to v, pushed toward 0;
• regressor pairs (v, R(v)), pushed toward 0 (this term lets the regressor learn from D);
• control pairs (v, s⁻), where s⁻ is the embedding of a different seen class, pushed toward 0.

The adversarial losses use least-squares GAN targets (1 and 0) §3.5 eq. 4.

Full objective §3.6 eq. 7. The CVAE and regressor are trained together with a weighted sum of the CVAE loss, the generator adversarial term, the cyclic loss, the supervised regression loss and the regressor adversarial term. The weights λ1, λ2 and λ3 multiply the cyclic, supervised and regressor adversarial terms eq. 7 §4.2. The discriminator is trained separately §3.6.

Training procedure §3.6. Pretrain the CVAE alone with eq. 1. Then train the whole model: each iteration does one discriminator update with eq. 4, then one update of the generator, encoder and regressor with eq. 7. The paper's training algorithm block is commented out in the source, so the card has no algorithm number §3.6.

Evaluation protocol §3.7. For each unseen class, generate synthetic features from its embedding. Add them to the real seen-class features, and classify test images with 1-NN. The paper uses 1-NN "as most baselines do" for a fair comparison §3.7.

Key equations:

\mathcal{L}_{\mathrm{CVAE}}(\theta_E, \theta_G) = \mathbb{E}_{P_{data}(\mathbf{v},\mathbf{s}),\,P_E(\mathbf{z}\mid\mathbf{v})}\big[\log P_G(\mathbf{v}\mid\mathbf{z},\mathbf{s})\big] - D_{KL}\big(P_E(\mathbf{z}\mid\mathbf{v})\,\|\,P(\mathbf{z})\big) eq. 1

\mathcal{L}_{\mathrm{sup}}(\theta_R) = \mathbb{E}_{P_{data}(\mathbf{v},\mathbf{s})}\,\big\|\mathbf{s} - R(\mathbf{v})\big\|_2^2 eq. 2

\mathcal{L}_{\mathrm{cyc}}(\theta_G,\theta_E,\theta_R) = \mathbb{E}\Big[\big\|\mathbf{v} - G(R(\mathbf{v}),\mathbf{z})\big\|_2^2 + \big\|\mathbf{s} - R(G(\mathbf{s},\mathbf{z}))\big\|_2^2\Big], \quad \mathbf{z}\sim P_E(\mathbf{z}\mid\mathbf{v}) eq. 3

\begin{aligned} \mathcal{L}_{\mathrm{adv}}(\theta_D) ={} & \mathbb{E}_{P_{data}(\mathbf{v},\mathbf{s})}\big[(D(\mathbf{v},\mathbf{s})-1)^2\big] + \mathbb{E}\big[D(G(\mathbf{s},\mathbf{z}),\mathbf{s})^2\big] \\ & + \mathbb{E}_{P_{data}(\mathbf{v})}\big[D(\mathbf{v},R(\mathbf{v}))^2\big] + \mathbb{E}_{P_{data}(\mathbf{v},\mathbf{s}),\,P_{data}(\mathbf{s}^-\mid\mathbf{s})}\big[D(\mathbf{v},\mathbf{s}^-)^2\big] \end{aligned} eq. 4

\mathcal{L}_{\mathrm{adv}}(\theta_R) = \mathbb{E}_{P_{data}(\mathbf{v})}\big[(D(\mathbf{v},R(\mathbf{v}))-1)^2\big] eq. 5

\mathcal{L}_{\mathrm{adv}}(\theta_E,\theta_G) = \mathbb{E}\big[(D(G(\mathbf{s},\mathbf{z}))-1)^2\big] eq. 6

\mathcal{L} = \mathcal{L}_{\mathrm{CVAE}} + \mathcal{L}_{\mathrm{adv}}(\theta_G,\theta_E) + \lambda_1\mathcal{L}_{\mathrm{cyc}} + \lambda_2\mathcal{L}_{\mathrm{sup}} + \lambda_3\mathcal{L}_{\mathrm{adv}}(\theta_R) eq. 7

Harmonic mean and per-class accuracy:

\mathrm{Acc}_{\mathcal{Y}} = \frac{1}{|\mathcal{Y}|}\sum_{c} \frac{\#\,\text{correct predictions in } c}{\#\,\text{samples in } c} eq. 8

H = \frac{2\,\mathrm{Acc}_{\mathcal{Y}^u}\,\mathrm{Acc}_{\mathcal{Y}^s}}{\mathrm{Acc}_{\mathcal{Y}^u} + \mathrm{Acc}_{\mathcal{Y}^s}} eq. 9

Pseudocode. Stage 1 pretrains the CVAE; stage 2 is the adversarial training; the last block is evaluation. The reconstruction and KL forms are not given in the paper; see Pitfalls.

# Stage 1: pretrain the CVAE on seen-class pairs  [§3.3] [§3.6] [eq. 1]
# Input:  image features v (B, 2048); class embeddings s (B, A)
# Output: encoder E and generator G

def pretrain_cvae(pairs, num_epochs):
    # Fixed epoch count, not stated in paper  [code: pretrain_gdan.py#L66]
    for epoch in range(num_epochs):
        # Draw a batch of seen-class pairs  [code: pretrain_gdan.py#L70-L72]
        v, s = sample_batch(pairs)
        # Posterior mean and log-variance, from v only  [§3.3]
        mu, log_var = E(v)
        # Reparameterised sample; noise distribution: see Pitfalls  [§3.3]
        z = mu + exp(log_var / 2) * noise(mu)
        # Reconstruction term plus KL to the prior  [eq. 1]
        loss = reconstruction(G(s, z), v) + kl(mu, log_var)
        update(E, G, loss)
    return E, G

# Stage 2: adversarial training of D, then E, G and R  [§3.6] [eq. 4] [eq. 7]
# Input:  pretrained E and G; regressor R; discriminator D; seen pairs
# Output: trained E, G, R and D

LAMBDA_1 = LAMBDA_2 = LAMBDA_3 = 0.1  # [§4.2] [eq. 7]

def train_gdan(E, G, R, D, pairs, num_iterations):
    # One discriminator step, then one generator step, per iteration  [§3.6]
    for iteration in range(num_iterations):
        # Fresh batch for the discriminator step  [code: train_gdan.py#L115]
        v, s = sample_batch(pairs)
        # Embedding of a different seen class, for control pairs  [§3.5]
        s_other = sample_other_class_embedding(pairs, s)
        # Generated features, held fixed for this step  [§3.5] [eq. 4]
        v_fake = G(s, sample_posterior(E, v)).detach()
        # Regressor embeddings of real features, held fixed  [§3.5] [eq. 4]
        s_reg = R(v).detach()
        # Real, generated, regressor and control terms  [eq. 4]
        loss_d = (D(v, s) - 1) ** 2 + D(v_fake, s) ** 2
        loss_d = loss_d + D(v, s_reg) ** 2 + D(v, s_other) ** 2
        update(D, loss_d)
        # Fresh batch for the generator, encoder and regressor step  [§3.6]
        v, s = sample_batch(pairs)
        mu, log_var = E(v)
        z = mu + exp(log_var / 2) * noise(mu)
        # Features generated from the posterior sample  [eq. 6]
        v_recon = G(s, z)
        # Regressor embedding of the real image  [§3.4]
        s_hat = R(v)
        # Cycle through G and through R  [eq. 3]
        cycle = sq_err(G(s_hat, z), v) + sq_err(R(v_recon), s)
        # Supervised regression loss  [eq. 2]
        supervised = sq_err(s_hat, s)
        # Generator adversarial term  [eq. 6]
        adv_gen = (D(v_recon, s) - 1) ** 2
        # Regressor adversarial term  [eq. 5]
        adv_reg = (D(v, s_hat) - 1) ** 2
        # Full objective, weighted as in eq. 7  [eq. 7]
        loss = cvae_loss(v, v_recon, mu, log_var) + adv_gen
        loss = loss + LAMBDA_1 * cycle + LAMBDA_2 * supervised
        loss = loss + LAMBDA_3 * adv_reg
        update(E, G, R, loss)
    return E, G, R, D

# Evaluation: synthesise unseen features, then 1-NN  [§3.7] [§4.2]
# Input:  generator G; unseen embeddings; seen pairs; test features
# Output: predicted label per test image

def predict(G, unseen_embeddings, seen_pairs, num_samples):
    # Generate features for each unseen class  [§3.7]
    synthetic = []
    for label, s_u in unseen_embeddings:
        for _ in range(num_samples):
            # Noise drawn from the prior; see Pitfalls  [§3.3]
            synthetic.append((G(s_u, noise_prior()), label))
    # Real seen features plus synthetic unseen features  [§3.7]
    bank = seen_pairs + synthetic
    # 1-NN over the bank; the metric is not stated in the paper  [§3.7]
    return nearest_neighbour(bank, test_features)

Hyperparameters

Values stated in the paper, with the code's values where they differ. Batch size, seeds and hardware are not stated in the paper.

Results

Author-reported. Setup: four GBU benchmarks with the split and features of [xian2018gbu] §4.1. Metrics: average per-class top-1 accuracy on unseen (U) and seen (S) classes, and their harmonic mean H eq. 8 eq. 9. Classification is 1-NN over real seen features plus generated unseen features §3.7. The two starred rows are copied from their original papers, so their settings are not identical to this paper's §4.3. The paper reports no seeds or error bars §4.3.

Table 1: dataset statistics tab. 1.

The CUB unseen-test count does not add up with the row's totals (see Pitfalls).

Table 2: generalized zero-shot results, unseen (U), seen (S) and harmonic mean (H) tab. 2. Rows marked with a star train an extra classifier for testing (SE-GZSL trains a LinearSVC; f-CLSWGAN trains extra embedding models), so the paper says they are not directly comparable §4.3.

Headline claims §4.3: GDAN has the highest unseen accuracy among the unstarred baselines on all four datasets. On SUN its seen accuracy is far above the rest (89.9). On CUB it is slightly below SP-AEN on seen classes. On aPY it gives the highest H among unstarred rows. On AwA2 the text says DEM and RelationNet are "less than 1%" higher on H; the table is not consistent with that (see Pitfalls).

Table 3: component analysis, unseen accuracy only tab. 3 §4.4. Single-component rows are trained alone and tested as described in §4.4. "w/o" rows drop one component from the full model.

Synthetic-sample count §4.5 fig. 4: accuracy on CUB, SUN and aPY rises with the number of generated samples and levels off at about 400. AwA2 gains little. The paper suggests the larger CUB and SUN gains may come from their larger numbers of unseen classes (50 and 72, against 12 and 10 for aPY and AwA2) §4.5.

Figures, described in words only:

• fig. 1 (§1): four ways of linking features and semantics: (a) visual-to-semantic embedding; (b) semantic-to-visual mapping; (c) metric learning; (d) the proposed unification.
• fig. 2 (§3.2): the overall model with its three parts and the data flow.
• fig. 3 (§3.2): the detailed model. The encoder and generator form the CVAE, the regressor is R, and the discriminator is D. The CVAE and regressor interact through the cyclic loss, and both learn from D through the adversarial loss. The regressor also has the supervised loss. The CVAE loss is not drawn.
• fig. 4 (§4.5): unseen accuracy against the number of synthetic samples, as described above.
• fig. 5 (§4.6): t-SNE of 200 synthetic and 200 real features per unseen class, for AwA2 (a) and aPY (b). The paper names a failure case in each: the rat class in AwA2 and the statue class in aPY §4.6.

Paper vs code

Official repository: stevehuanghe/GDAN, linked from the paper's experiment section §4.2. Commit 14e5bc555f5e4f78ce3398e38cef40b91fce0d68. License: MIT, copyright He Huang, 2018 LICENSE. I read the model, the three training and evaluation scripts, the data loader and all four dataset configs. I did not run the code: it needs the GBU data and a GPU (CUDA is hard-coded).

Differences:

1. Regressor width. Paper: 600 hidden units §4.2. Code: 512 in all four configs cub.yaml sun.yaml apy.yaml awa2.yaml and in the parser default config_gdan.py.
2. Generator width. Paper: 800 hidden units for the CVAE generator and the discriminator §4.2. Code: 600 for CUB, SUN and aPY cub.yaml, 512 for AWA2 awa2.yaml. The discriminator is 800 in all configs cub.yaml.
3. Encoder width. Paper: 1200, 600 for all datasets §4.2. Code: 1200, 600 except AWA2, which uses 1024, 512 awa2.yaml.
4. Learning rates. Paper: 1e-5 for the discriminator, 1e-4 for the CVAE and regressor §4.2. Code: one learning rate, 1e-5 by default, passed to all three optimizers train_gdan.py config_gdan.py. All four configs set 0.00001 cub.yaml. Pretraining also uses the config learning rate pretrain_gdan.py.
5. Adam betas and weight decay. Paper: betas (0.9, 0.999) §4.2. Code: (0.8, 0.999) in the GDAN stage train_gdan.py; pretraining uses the PyTorch default betas pretrain_gdan.py. Weight decay 0.01 is applied to every optimizer; the paper does not mention it train_gdan.py pretrain_gdan.py.
6. Weight of the regressor term in the discriminator loss. Paper: weight 1 on the D(v, R(v)) term eq. 4. Code: multiplied by θ3 = 0.1 train_gdan.py.
7. Noise distribution. Paper: the prior P(z) is a unit Gaussian §3.3. Code: the reparameterisation noise and the noise used for generation are drawn from a uniform distribution on [0, 1) (torch.rand), not a Gaussian gdan.py gdan.py gdan.py. The KL term in the code still assumes a standard normal prior gdan.py.
8. Loss reduction. Paper: squared L2 norms summed over dimensions for L_sup and L_cyc eq. 2 eq. 3. Code: mean squared error, averaged over the batch and over dimensions, so each supervised or cyclic term is divided by the embedding or feature width (312 for CUB, 2048 for features) train_gdan.py train_gdan.py train_gdan.py. The reconstruction term in the CVAE loss is a sum of squared errors, times 0.5 gdan.py. The paper gives no likelihood form.
9. Output activations. The paper does not mention output activations. Code: the decoder ends in a ReLU, so generated features are non-negative gdan.py; the regressor also ends in a ReLU gdan.py.
10. Encoder dropout. Not in the paper. Code: dropout with p = 0.3 between encoder layers gdan.py.
11. Which seen classes are used for training. Paper: "training data (including validation)" §3.1. Code: GDAN training uses the train split only train_gdan.py utils.py. The final classifier at test time also uses the train split only valtest_gdan.py. The validation split is used only for choosing a checkpoint, by unseen accuracy on the validation classes valtest_gdan.py valtest_gdan.py.
12. Checkpoint criterion. Paper: "the best" checkpoint on validation §4.2, no criterion given. Code: the highest validation unseen accuracy, not H valtest_gdan.py.
13. Pretraining length. Paper: not stated §3.6. Code: 1000 epochs, with the --epoch argument ignored pretrain_gdan.py. The checkpoint loaded for training is named per dataset: CUB 670, SUN 320, aPY 590, AWA2 150 cub.yaml sun.yaml apy.yaml awa2.yaml.
14. Per-dataset settings. Paper: one setting for all datasets, except that the paper gives none for batch size §4.2. Code: batch 128 for CUB, SUN and aPY, 256 for AWA2 awa2.yaml cub.yaml; encoder and generator widths differ (items 2 and 3).
15. Evaluation metric for 1-NN. Paper: 1-NN, metric not stated §3.7. Code: scikit-learn's default Euclidean distance valtest_gdan.py. A cosine-distance helper exists but is not called utils.py.
16. Synthetic sample count. Paper: no number for the main results §4.5. Code: 500 per unseen class by default valtest_gdan.py cub.yaml.
17. Features are not normalised in code. The loader reads the stored feature matrix and labels directly utils.py utils.py. The paper does not describe any normalisation.

Search steps: the paper's own footnote link was enough, so no GitHub search was run. The repository is the authors' own (first author He Huang). No other repository was considered.

Pitfalls

• Equation numbers. The numbers in this card follow the order of the numbered equations in the LaTeX source in raw/src. No compiled PDF is in the source bundle, so the numbers are not checked against one. The generic CVAE objective in §3.3 is unnumbered in the source and is not cited here. If a number matters, cite the section.
• Equation 6 is written with one argument. The source writes D(G(s, z)), but D takes two arguments everywhere else. Read it as D(G(s, z), s), which the code does train_gdan.py.
• The CVAE likelihood is not stated. The paper writes log P_G(v | z, s) with no form. The code uses a fixed-variance Gaussian form (0.5 times the sum of squared errors) gdan.py. Its scale against the KL term and the λ weights changes the balance of the objective.
• The noise distribution is unclear. The paper says the prior is a unit Gaussian §3.3, but the code samples uniform noise on [0, 1) for both training and generation gdan.py gdan.py. The paper does not say which was used for its reported numbers. Pick one deliberately; the card does not resolve this.
• λ weights are not comparable to the paper's scale. Because the code uses mean squared errors (Paper vs code, item 8), λ = 0.1 means a smaller effective weight than in the squared-norm formulas. The paper says only that these values "generally work well" §4.2.
• Two independent batches per iteration. The code draws a new batch for the discriminator step and another for the generator step train_gdan.py train_gdan.py. Batches are drawn at random, with replacement across batches, so an epoch is not a pass over all data data_factory.py.
• A computed term is unused. The code computes a score from prior-sampled features (fake_scores2) but does not add it to the discriminator loss train_gdan.py train_gdan.py. Do not add it unless you have a reason to.
• Table 1 counts disagree. For CUB, the row says 11,788 images in total, 7,057 in train+val and 2,679 unseen in test. 11,788 − 7,057 = 4,731, which equals 2,967 + 1,764, so the test-unseen count should be 2,967. A commented-out paragraph in the source gives 2,967 §4.1 (3_experiment.tex, commented block). For AwA2, the total of 37,332 does not match 23,527 + 7,913 + 5,882 = 37,322; the commented text gives 37,322.
• Text claims that the table does not support.
  • CUB: the text says GDAN has the highest harmonic mean (49.5) §4.3. The starred f-CLSWGAN row has 49.7 tab. 2.
  • AwA2: the text says DEM and RelationNet beat GDAN on H "by less than 1%" §4.3. The table gives 45.1 and 45.3 against 43.5, a gap of 1.6 and 1.8 points.
  • AwA2: the text says GDAN's unseen accuracy is 2.7 points higher §4.3. The table gives 1.6 over DEM (30.5) and 2.1 over RelationNet (30.0).
  • aPY: the text says the unseen margin is 16% §4.3. Against SP-AEN (13.7) the gap is 16.7 points.
• Starred rows. SE-GZSL and f-CLSWGAN use extra classifiers at test time §4.3. They are in Table 2 but outside the paper's claims. Do not compare them directly.
• No seeds or error bars. The paper reports one number per cell, with no variance and no seed count §4.3 tab. 2. Run-to-run variance is unknown.
• Old environment. The README targets Python 3.6 and PyTorch 0.4 README.md. The scripts call yaml.load(fin) without a Loader train_gdan.py valtest_gdan.py pretrain_gdan.py. Current PyYAML (version 6 and later) requires a Loader, so these calls fail on a modern install.
• Hard-coded CUDA. Sampling and reparameterisation call .cuda() directly gdan.py gdan.py gdan.py. The code will not run on CPU as written.
• Feature input. The code's default input width is 2048 config_gdan.py and the feature file name suggests ResNet-101 features train_gdan.py. The paper does not name the backbone §3.3 §4.1; the GBU release is the stated source.
• Pretraining checkpoint. The paper does not say how many CVAE pretraining epochs it used, and the code picks a checkpoint per dataset by file name (Paper vs code, item 13). A reimplementation needs its own choice.
• Selection by unseen accuracy only. Checkpoints are compared by validation unseen accuracy valtest_gdan.py, not by H, which the paper's headline numbers use.
• No convergence analysis in the active text. The only training-length statement is 500 epochs §4.2. A validation-curve paragraph exists in the source but is commented out.

Lineage

• Builds on: conditional VAE, used for the generator §3.3; the LS-GAN loss for the discriminator §3.5; the Text-to-Image GAN idea of training with mismatched pairs as fakes §3.5; and cyclic consistency from dual-learning and image-to-image translation work (CycleGAN and DualGAN), which the paper adapts §2. Named without arXiv ids; this card uses no content from them.
• Compared against: SSE, LATEM, ALE, DEVISE, SJE, ESZSL, SYNC, SAE, DEM, RelationNet, PSR-ZSL, SP-AEN, CVAE-ZSL, SE-GZSL and f-CLSWGAN §4.3 tab. 2. The last two are starred in Table 2.
• Superseded by: not known from the content of this wiki.