HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning

In plain words

A zero-shot classifier has to name objects it never saw during training. It gets help from a description of each class, here a list of attributes such as "has stripes" or "lives in water". The hard part is that pictures and descriptions are very different kinds of data, so a shared space where they can be compared does not line up well.

This paper builds that shared space in two stages. The first stage shapes the groups of points so that the picture groups and the description groups take the same form, checked by a pair of classifiers that are pushed to disagree and then pulled back together. The second stage moves the two groups so that their overall spread matches. Both stages run inside two linked autoencoders, and a simple classifier is then trained on the shared space. The authors report that this beats earlier shared-space methods on four standard benchmarks.

TL;DR

HSVA maps image features and attribute vectors into one latent space in two steps: a structure adaptation (two encoders, a discrepancy term between two classifiers) and a distribution adaptation (a Wasserstein distance between Gaussian latents plus an inverse CORAL term against seen-unseen bias). Author- reported conventional ZSL accuracy is 70.6 on AWA1, 62.8 on CUB and 63.8 on SUN tab. 1. The official code has no license file, and it differs from the paper in several schedules and in how it picks its reported epoch (see Paper vs code).

Problem

• Zero-shot learning must recognise unseen classes, using side information such as attributes for each class §1.
• Common-space methods align visual and semantic features with one step that matches distributions. The paper argues this ignores that the two domains also differ in structure, so related classes can sit on different manifolds §1 fig. 1.
• Setting: seen-class images with labels and seen-class attributes are available for training, and unseen-class attributes are available as auxiliary data. Unseen visual features are not used for training §3 §3.5.
• Two evaluations: conventional ZSL (CZSL), which classifies only unseen classes, and generalised ZSL (GZSL), which classifies seen and unseen classes together §3 §4.

Method

Notation: x is a 2048-d visual feature from a ResNet-101 backbone and a is a class attribute vector. Seen data is (x^s, y^s, a^s); unseen attributes are a^u §3.

HSVA has two partially aligned VAEs that share one encoder §3.1 fig. 2:

1. Encoders and decoders. The visual encoder E^x and semantic encoder E^a map their inputs into a structure-aligned space. A common encoder E^z maps that space to a Gaussian latent (mean and variance) that is shared by both domains. Decoders D^x and D^a rebuild the visual feature and the attribute vector from the latent §3.1.
2. VAE losses. Each VAE has a reconstruction term and a KL term to a standard normal prior, weighted by gamma §3.1 eq. 1 eq. 2.
3. Cross reconstruction. Each decoder also rebuilds its own input from the latent of the other domain, with an L1 norm. This ties the two VAEs together §3.1 eq. 3 eq. 4 eq. 5.
4. Structure adaptation, classification. E^x, E^a and two task-specific heads CLS^1 and CLS^2 are trained with softmax cross-entropy over seen classes, on both visual and semantic embeddings §3.2 eq. 6 eq. 7.
5. Structure adaptation, discrepancy maximisation. With encoders frozen, the heads are updated to increase the discrepancy between their outputs, measured by a sliced Wasserstein discrepancy (SWD) on the two domains §3.2 eq. 8 eq. 9.
6. Structure adaptation, discrepancy minimisation. With heads frozen, the encoders are updated to decrease the same discrepancy §3.2 eq. 10.
7. Distribution adaptation. The 2-Wasserstein distance between the diagonal Gaussian latents of visual and semantic embeddings is minimised, using the common encoder E^z §3.3 eq. 11.
8. Inverse CORAL. The CORAL statistic between seen visual latents and unseen semantic latents is negated, so that the two groups are pushed apart. This targets seen-unseen bias §3.3 eq. 12.
9. Joint objective. All terms are summed with weights lambda_1, lambda_2 and lambda_3. The paper says the terms are optimised alternately in the SA steps and that the weights are the same for all datasets §3.4 eq. 13.
10. Classification. After training, the visual and attribute features are mapped to latents z = E^z(E^x(x)) and z = E^z(E^a(a)). A softmax classifier is trained on seen visual latents plus unseen attribute latents, so the method is inductive §3.5.

The pseudocode below follows the official code's order of updates. The paper's objective (eq. 13) is written as one sum, and the code runs the same terms in three blocks per batch.

# One training iteration, as the code orders the paper's terms  [§3.2–§3.4]
# [code: HSVA.py#L513-L663]
# Input:  visual batch x (B, 2048), attribute batch a (B, A), labels y (B,)
#         unseen attribute batch a_u (B, A)
# Output: none (encoders, decoders and heads are updated in place)

def train_iteration(x, a, y, a_u, lambda_1, lambda_2, lambda_3, beta):
    # Classify visual and semantic embeddings with both heads  [§3.2] [eq. 6]
    s_vis = E_x(x)                                    # (B, 2048)
    s_sem = E_a(a)                                    # (B, 2048)
    loss_cls = (ce(CLS1(s_vis), y) + ce(CLS2(s_vis), y)
                + ce(CLS1(s_sem), y) + ce(CLS2(s_sem), y))
    update(CLS1, CLS2, E_x, E_a, loss_cls)

    # Freeze encoders; heads maximise their disagreement  [§3.2] [eq. 8]
    # The classification loss is recomputed on fresh forward passes
    dis = swd(CLS1(E_x(x)), CLS2(E_x(x))) + swd(CLS1(E_a(a)), CLS2(E_a(a)))
    loss_cls = (ce(CLS1(E_x(x)), y) + ce(CLS2(E_x(x)), y)
                + ce(CLS1(E_a(a)), y) + ce(CLS2(E_a(a)), y))
    update(CLS1, CLS2, loss_cls - lambda_2 * dis)

    # Freeze heads; encoders reduce the disagreement  [§3.2] [eq. 10]
    # The paper does not state how often this runs; the code runs it twice
    # [code: HSVA.py#L563]
    for _ in range(2):
        dis = swd(CLS1(E_x(x)), CLS2(E_x(x))) + swd(CLS1(E_a(a)), CLS2(E_a(a)))
        update(E_x, E_a, lambda_2 * dis)

    # Partially aligned VAEs with distribution and inverse-CORAL terms  [§3.3]
    mu_x, logvar_x = E_z(E_x(x))                      # (B, 64) each
    mu_a, logvar_a = E_z(E_a(a))                      # (B, 64) each
    mu_u, logvar_u = E_z(E_a(a_u))                    # (B, 64) each
    z_x = sample(mu_x, logvar_x)                      # see Pitfalls
    z_a = sample(mu_a, logvar_a)
    z_u = sample(mu_u, logvar_u)

    # Within-domain and cross-domain L1 reconstruction, summed  [§3.1] [eq. 4]
    rec_within = l1(D_x(z_x), x) + l1(D_a(z_a), a)
    rec_cross = l1(D_x(z_a), x) + l1(D_a(z_x), a)

    # KL to the standard normal prior with weight beta  [§3.1] [eq. 1]
    kl = kl_to_std_normal(mu_x, logvar_x) + kl_to_std_normal(mu_a, logvar_a)

    # Wasserstein distance between paired diagonal Gaussians  [§3.3] [eq. 11]
    da = wasserstein_gaussian(mu_x, logvar_x, mu_a, logvar_a)
    # Inverse CORAL: seen visual vs unseen semantic latents  [§3.3] [eq. 12]
    coral = coral_stat(z_x, z_u)

    loss = (rec_within + beta * kl + lambda_1 * rec_cross
            + lambda_3 * (da - coral))
    update(D_x, D_a, E_z, E_x, E_a, loss)
    return loss

The discrepancy used by default is sliced Wasserstein. Its inner computation:

# Sliced Wasserstein discrepancy between two heads  [§3.2] [eq. 9]
# Input:  logits_1, logits_2 (B, K) from the two heads
# Output: scalar discrepancy

def swd(logits_1, logits_2, n_proj=128):
    # Turn logits into class probabilities  [code: HSVA.py#L231-L232]
    p1 = softmax(logits_1)                            # (B, K)
    p2 = softmax(logits_2)                            # (B, K)
    # Fresh random directions, normalised per column  [code: HSVA.py#L235-L236]
    proj = normalise_columns(randn(K, n_proj))        # (K, M)
    # Project, sort each direction over the batch, compare sorted values
    q1 = sort(p1 @ proj, axis=0)                      # (B, M)
    q2 = sort(p2 @ proj, axis=0)                      # (B, M)
    # Mean squared gap, averaged over projections  [code: HSVA.py#L241-L242]
    return mean((q1 - q2) ** 2)

Final classifier. Latents are built per class, then a softmax classifier is trained:

# Build the training set for the final classifier  [§3.5]
# [code: HSVA.py#L375-L444]
# GZSL: 200 encoded seen-image latents and 400 encoded unseen-attribute
#       latents per class  [§4]
# CZSL: 800 (AWA1), 400 (CUB), 200 (SUN) unseen-attribute latents per class
#       [§4]

def build_train_set(x_seen, y_seen, a_unseen, y_unseen, n_seen, n_unseen):
    # Resample each class with replacement when it has too few rows
    # Visual rows go through E_x then E_z; attribute rows through E_a then E_z
    # Training uses a sampled latent; testing uses the mean  [§3.5]
    # [code: HSVA.py#L351-L427]
    z_seen, labels_seen = sample_per_class(E_z(E_x(x_seen)), y_seen, n_seen)
    z_unseen, labels_unseen = sample_per_class(E_z(E_a(a_unseen)), y_unseen,
                                               n_unseen)
    return concat(z_seen, z_unseen), concat(labels_seen, labels_unseen)

Hyperparameters

Datasets §4: CUB has 11,788 images, 200 classes (150 seen, 50 unseen) and 312 attributes. SUN has 14,340 images, 717 classes (645 seen, 72 unseen) and 102 attributes. AWA1 has 30,475 images and AWA2 has 37,322 images, both with 50 classes (40 seen, 10 unseen) and 85 attributes. Splits follow the training splits of Xian et al. [ref. 44], and the evaluation protocol follows Xian et al. [ref. 43] §4.

Results

Author-reported. Setup: four benchmarks, ResNet-101 features, a softmax classifier on HSVA latents §4. CZSL is reported on AWA1, CUB and SUN only tab. 1.

Conventional ZSL, top-1 accuracy on unseen classes, source tab. 1:

Generalised ZSL, unseen (U), seen (S) and harmonic mean (H), source tab. 2:

Ablation under GZSL, source tab. 3:

The text states gains of at least 5.3, 3.0 and 2.0 points over the best common-space CZSL baseline on AWA1, CUB and SUN §4.1 tab. 1. It states harmonic-mean gains of at least 2.7, 1.0, 2.9 and 2.7 points over the best common-space GZSL baseline on AWA1, AWA2, CUB and SUN §4.1 tab. 2. The qualitative t-SNE comparison with CADA-VAE is in fig. 3. The paper presents it as evidence, not as a metric.

Paper vs code

The official code is at shiming-chen/HSVA (the first author's account, linked in the abstract). It is a flat set of Python files. The HSVA model is in HSVA.py, hyperparameters and the training entry point are in single_experiment.py, and the classifier is final_classifier_v1.py. The repository has no LICENSE file, so we cite it and do not copy from it. The README gives inference and training commands README.md.

• Adam beta_1. Paper: 0.5 §4. Code: generator, encoders, decoders and discriminator use beta_1 0.9 HSVA.py, and so do the heads CLS^1 and CLS^2 HSVA.py. Only the final softmax classifier uses 0.5 final_classifier_v1.py.
• gamma schedule. Paper: +0.0026 per epoch until epoch 90 §4. Code: linear from 0 to 0.25 over epochs 0 to 90, about 0.00278 per epoch [code: single_experiment.py#L39-L41].
• lambda_2 and lambda_3 schedule. Paper: +0.54 per epoch until epoch 22 §4. Code: one ramp to 8.0 over epochs 0 to 25, about 0.32 per epoch [code: single_experiment.py#L45-L47]. The paper has two weights, lambda_2 for the SWD term and lambda_3 for the distribution and iCORAL terms §3.4 eq. 13. The code uses the same value for both HSVA.py HSVA.py.
• lambda_1 schedule. Matches: +0.044 per epoch from epoch 21 to 75 in the paper §4; the code's cap is 2.37 over the same epochs [code: single_experiment.py#L42-L44].
• Likelihood term. The paper writes the within-domain VAE term as a decoder log-likelihood §3.1 eq. 1. The code uses a summed L1 reconstruction with weight 1 HSVA.py and the KL term with weight gamma [code: HSVA.py#L631-L632].
• One objective or blocks. The paper writes one objective with alternating updates §3.4 eq. 13. The code runs three update blocks per batch: classify, maximise, then minimise twice, and only then the VAE and distribution terms HSVA.py HSVA.py HSVA.py HSVA.py. The classification loss appears only in the first block.
• Discrepancy minimisation count. Paper: not stated. Code: two updates of the encoders per batch HSVA.py.
• Discrepancy form. The default is SWD with 128 random projections [code: HSVA.py#L235-L236] single_experiment.py. An MCD option (mean absolute difference of softmax outputs) exists but is not the default [code: HSVA.py#L226-L227].
• Seen-class features in GZSL. Paper: 200 synthesised features per seen class §4. Code: 200 encoded real seen-class images per class, with zero synthesised seen features HSVA.py single_experiment.py.
• AWA2 in CZSL. The paper's CZSL numbers cover AWA1, CUB and SUN §4 tab. 1. The code also sets a CZSL configuration for AWA2 (200 features per unseen class) single_experiment.py.
• Reparameterisation. The paper uses the reparameterisation trick without a formula §3.5. The code uses sigma = exp(0.1 · logvar), not exp(0.5 · logvar), and draws one noise value per sample that is shared across all latent dimensions HSVA.py.
• Test latents. The paper's test encoding is not specified §3.5. The code uses the mean, not a sample, for test features HSVA.py.
• Feature scaling. Not stated in the paper. The code scales features with min-max fitted on the training split and applied to the test split [code: data_loader.py#L122-L126].
• Unseen attribute sampling. The paper describes an auxiliary set of unseen classes §3. The code draws unseen attribute rows by sampling the labels of the test-unseen split data_loader.py. Class frequencies in the test split therefore shape these batches.
• Reported numbers. The released README reports GZSL numbers for AWA1 (U 61.1, S 75.2, H 67.4), AWA2 (57.8, 79.3, 66.9) and CUB (51.9, 59.5, 55.5) [code: README.md#L45-L47]. The paper reports 59.3 / 76.6 / 66.8, 56.7 / 79.8 / 66.3 and 52.7 / 58.3 / 55.3 tab. 2. SUN (48.6 / 39.0 / 43.3) and the CZSL accuracies (70.6, 62.8, 63.8) agree. The README CZSL row for AWA1 is 70.6 [code: README.md#L45].

Pitfalls

• Test-set epoch selection. Both the classifier and the generator choose the best epoch on the test split. The classifier keeps the epoch with the best harmonic mean on test accuracies final_classifier_v1.py. The generator evaluates every epoch from 100 onward and keeps the best H [code: HSVA.py#L724-L731]. A reported number is therefore optimistic unless you select on a validation split. The paper does not state how the reported epoch was chosen. The code comments say the classifier epoch count was set on a validation split single_experiment.py, but that selection is not in the code.
• Sampling scale. Training latents use sigma = exp(0.1 · logvar), with one noise value per sample shared across dimensions HSVA.py. This is not the usual reparameterisation, and the paper gives no formula. Reproduce the code's sampler before changing it.
• Batch sums, not means. Reconstruction is summed over batch and dimensions HSVA.py. The KL term and the distribution term are also summed over the batch HSVA.py HSVA.py. Changing the batch size changes the effective weights of lambda_1, lambda_2 and lambda_3. The paper states none of these reductions.
• CORAL scale. The code takes the mean over matrix entries of the squared covariance difference HSVA.py. The paper does not give the normalisation.
• Unused terms. A margin term is computed but not added to the loss [code: HSVA.py#L649] HSVA.py. The attribute argument of the heads is ignored in the forward pass models.py.
• Hard-coded GPU calls. The code calls .cuda() on several tensors and creates CUDA tensors for sampling HSVA.py HSVA.py HSVA.py. It will not run on CPU without edits.
• Hidden layer sizes. The paper gives the encoder and decoder types but not their widths §4. Widths in the table come from the code only.
• Seen-unseen bias term. The iCORAL term is what the paper credits for seen-unseen bias. Its contribution is 1.4 to 4.4 H points in the ablation §4.1 tab. 3.

Lineage

• Builds on: maximum classifier discrepancy [ref. 34], sliced Wasserstein discrepancy [ref. 35], and CORAL [ref. 39]. The code acknowledges CADA-VAE and MCD-DA README.md. §2 §3.2 §3.3.
• Compared against: DeViSE [ref. 32], ReViSE [ref. 9], DCN [ref. 10], CADA-VAE [ref. 11], SGAL [ref. 17] and the non-common-space methods in tab. 2 §4.1.
• Superseded by: not stated in the paper or in this wiki.