A Unified approach for Conventional Zero-shot, Generalized Zero-shot and Few-shot Learning

In plain words

The task is to recognise object classes the model never saw labelled images for. A common trick is to describe every class by a vector (a list of attributes such as "has stripes", or a word embedding), learn to map pictures onto those vectors for the seen classes, and then match a new picture to the nearest description of an unseen class.

This paper argues that one global mapping from pictures to descriptions is too crude. Instead, it builds one "direction" per seen class, showing how the picture lines up with that class's description. A new class is then described as a mix of the seen directions that resemble it, and the mix is chosen with help from a learned distance measure on the descriptions. The same machinery handles three settings: no labelled examples of the new classes, a mix of seen and unseen classes at test time, and a handful of labelled examples per new class.

TL;DR

• Core object: the Class Adapting Principal Direction (CAPD). Seen CAPDs come from per-class linear maps §3.1; each unseen CAPD is a linear combination of seen CAPDs §3.1.2.
• Combination weights come from semantic embeddings, using a metric learned on CAPDs §3.1.2 and an optional reduced set of nearest seen classes §3.2.
• GZSL adds a generalized seen CAPD tuned so seen and unseen score scales balance §3.3; FSL fuses the zero-shot CAPD with one from a classifier trained on the few labelled unseen images §3.4.
• Author-reported results beat listed baselines on several settings but not all (see Results). No official code was found.

Problem

Standard zero-shot learning (ZSL) classifies unseen-class images with no unseen training images §1 §3. Prior methods tend to over-fit one dataset or one feature type, and they do not scale well to generalized ZSL (GZSL), where seen and unseen classes both appear at test time, or to few-shot learning (FSL), where a few labelled unseen images are available §1 §2.

The paper's stated assumptions: semantic class embeddings are available for all classes, either attribute vectors or word vectors (word2vec, GloVe) §3 §5 Semantic Space Embeddings. Image features are fixed deep features (GoogLeNet, VGG-verydeep-19, ResNet-101) §5 Image Features.

Method

Notation §3: seen classes \mathcal{S} (count R_s), unseen \mathcal{U} (count R_u), semantic embeddings \mathbf{e}_c \in \mathbb{R}^d, image features \mathbf{x} \in \mathbb{R}^k.

3.1 Seen CAPDs §3.1.1. Each seen class s has a linear map \mathbf{W}_s. The CAPD of an image is its projection under that map eq. 1:

\mathbf{p}_s = \mathbf{W}_s^T \mathbf{x} eq. 1

\mathbf{W}_s is trained per class by SGD, with a regularised logistic-style loss that pushes \langle \mathbf{p}_s, \mathbf{e}_s \rangle up relative to the other embeddings eq. 2 §3.1.1. Positive samples are compared to the mean of the other classes' embeddings; negative samples are compared to their own class's embedding. Per-class training is independent, so all \mathbf{W}_s can be trained in parallel §3.1.1.

\min_{\mathbf{W}_s} \frac{1}{\kappa} \sum_{c=1}^{R_s} \sum_{m=1}^{n_c} \log\big(1 + \exp\{L(\mathbf{x}_c^m; \mathbf{W}_s)\}\big) + \frac{\lambda_s}{2} \lVert \mathbf{W}_s \rVert_2^2 eq. 2

with \kappa = \sum_c n_c and, in the case-split cost L defined in the text §3.1.1:

• if c \neq s: L = \langle \mathbf{p}_s, \mathbf{e}_c \rangle - \langle \mathbf{p}_s, \mathbf{e}_s \rangle;
• if c = s: L = \langle \mathbf{p}_s, \bar{\mathbf{e}}_{\neq s} \rangle - \langle \mathbf{p}_s, \mathbf{e}_s \rangle, where \bar{\mathbf{e}}_{\neq s} is the mean of the other seen embeddings §3.1.1.

For a test image, the seen CAPD matrix stacks all seen CAPDs: \mathbf{P}^{\mathcal{S}} = [\mathbf{p}_1 \dots \mathbf{p}_{R_s}] \in \mathbb{R}^{d \times R_s} §3.1.1.

3.2 Unseen CAPDs §3.1.2. A linear combination of seen CAPDs gives the unseen CAPD eq. 3:

\mathbf{p}_u = \sum_{s=1}^{R_s} \theta_{s,u} \mathbf{p}_s = \mathbf{P}^{\mathcal{S}} \boldsymbol{\theta}_u eq. 3

The paper cannot learn \boldsymbol{\theta}_u directly, because no unseen images are used. It learns a Mahalanobis metric \mathbf{M} on the CAPD space from pairs of same-class and different-class seen CAPDs §3.1.2 eq. 4:

\max_{\mathbf{M}} \min_{(i,j) \in \bar{\mathcal{A}}} d_{\mathbf{M}}^2(\mathbf{p}_i, \mathbf{p}_j) \quad \text{s.t.} \quad \sum_{(i,j) \in \mathcal{A}} d_{\mathbf{M}}^2(\mathbf{p}_i, \mathbf{p}_j) \le 1 eq. 4

where d_{\mathbf{M}}^2(\mathbf{p}_i, \mathbf{p}_j) = (\mathbf{p}_i - \mathbf{p}_j)^T \mathbf{M} (\mathbf{p}_i - \mathbf{p}_j), \mathcal{A} holds same-class pairs and \bar{\mathcal{A}} different-class pairs §3.1.2.

Coefficients are then estimated in semantic space. Approximate the unseen embedding as a combination of seen embeddings eq. 5:

\hat{\mathbf{e}}_u = \sum_{s=1}^{R_s} \alpha_{s,u} \mathbf{e}_s = \mathbf{E}^{\mathcal{S}} \boldsymbol{\alpha}_u eq. 5

and solve, with the same metric eq. 6:

\min_{\boldsymbol{\alpha}_u} \; (\hat{\mathbf{e}}_u - \mathbf{e}_u)^T \mathbf{M} (\hat{\mathbf{e}}_u - \mathbf{e}_u) + \frac{\lambda_u}{2} \lVert \boldsymbol{\alpha}_u \rVert_2^2 eq. 6

The paper assumes \boldsymbol{\theta}_u \approx \boldsymbol{\alpha}_u, giving eq. 7:

\mathbf{p}_u \approx \mathbf{P}^{\mathcal{S}} \boldsymbol{\alpha}_u eq. 7

Zero-shot prediction takes the unseen class with the largest projection of its CAPD onto its own embedding eq. 8 §3.1.2:

\hat{y} = \arg\max_u \langle \mathbf{p}_u, \mathbf{e}_u \rangle eq. 8

3.3 Reduced seen set §3.2. Only a subset of R_n < R_s seen classes is used eq. 9:

\hat{\mathbf{e}}_u = \sum_{i=1}^{R_n} \beta_{i,u} \mathbf{e}_i eq. 9

\boldsymbol{\beta}_u is found as in eq. 6; \boldsymbol{\alpha}_u is replaced by \boldsymbol{\beta}_u in eq. 7. The subset is chosen by Mahalanobis distance in semantic space §3.2. The paper also proposes an automatic choice of R_n per unseen class. It fits kernel density estimation to the mean-normalised Mahalanobis distances from \mathbf{e}_u to all seen embeddings and takes the number of seen classes with the highest density score as R_n §3.2 (unnumbered in the text; see Pitfalls).

3.4 Generalized ZSL §3.3. Seen CAPDs are biased toward seen classes, so each is replaced by a generalized version §3.3, unnumbered:

\mathbf{p}^g_s = \mathbf{P}^{\mathcal{S}} \boldsymbol{\gamma}_s eq. 10

where \boldsymbol{\gamma}_s is solved only from semantic embeddings. The objective makes the squared mean reconstruction loss of seen embeddings match that of unseen embeddings, plus an L2 penalty eq. 11 §3.3:

\min_{\boldsymbol{\gamma}} \Big\lVert \underbrace{\frac{1}{R_s} \sum_{s=1}^{R_s} (\mathbf{E}^{\mathcal{S}} \boldsymbol{\gamma}_s - \mathbf{e}_s)^2}_{\text{seen loss}} - \underbrace{\frac{1}{R_u} \sum_{u=1}^{R_u} (\mathbf{E}^{\mathcal{S}} \boldsymbol{\alpha}_u - \mathbf{e}_u)^2}_{\text{unseen loss}} \Big\rVert_2^2 + \frac{\lambda_\gamma}{2} \sum_{s=1}^{R_s} \lVert \boldsymbol{\gamma}_s \rVert_2^2 eq. 11

At test time, both generalized seen CAPDs and unseen CAPDs are scored against the union of seen and unseen embeddings, and the argmax is taken [§3.3, unnumbered]:

\hat{y} = \arg\max_{l \in \mathcal{S} \cup \mathcal{U}} \langle \mathbf{p}_l, \mathbf{v}_l \rangle

with \mathbf{p}_l \in \{\mathbf{p}^g_s\} \cup \{\mathbf{p}_u\} and \mathbf{v}_l \in \{\mathbf{e}_s\} \cup \{\mathbf{e}_u\} §3.3.

3.5 Few-shot §3.4. A separate linear map \mathbf{W}_u is learned for each unseen class from its few labelled images, giving \mathbf{p}'_u = \mathbf{W}_u^T \mathbf{x}. The two unseen CAPDs are fused eq. 12:

\mathbf{p}^f_u = \delta_u \mathbf{p}_u + \delta'_u \mathbf{p}'_u, \qquad \delta_u + \delta'_u = 1 eq. 12

The weights are the normalised sums, over seen-class training images, of the maximum projection response of each CAPD onto unseen embeddings [§3.4, unnumbered]:

\delta_u = \frac{\sum_x \max_u \langle \mathbf{p}_u, \mathbf{e}_u \rangle} {\sum_x \max_u \langle \mathbf{p}_u, \mathbf{e}_u \rangle + \sum_x \max_u \langle \mathbf{p}'_u, \mathbf{e}_u \rangle}

\delta'_u is defined symmetrically. At prediction time, \mathbf{p}^f_u replaces \mathbf{p}_u in eq. 8 §3.4.

Pseudocode for the zero-shot path eq. 1 eq. 7 eq. 8:

# Zero-shot prediction for one image  [§3.1] [eq. 1, 7, 8]
# Input:  x (k,); W[s] (k, d) per seen class; coef (R_s, R_u)
# Output: index of the predicted unseen class

def predict_zsl(x, W, coef, E_unseen):
    # One CAPD per seen class  [eq. 1]
    seen_capds = stack([W[s].T @ x for s in seen_classes])  # (d, R_s)
    # Unseen CAPDs as combinations of seen CAPDs  [eq. 7]
    unseen_capds = seen_capds @ coef                        # (d, R_u)
    # Score each unseen CAPD against its own embedding  [eq. 8]
    scores = [dot(unseen_capds[:, u], E_unseen[u])
              for u in unseen_classes]
    return argmax(scores)

Coefficients for one unseen class eq. 5 eq. 6:

# Coefficients of one unseen class  [§3.1.2] [eq. 5, 6]
# Inputs: E_seen (d, R_s), e_u (d,), metric M (d, d), reg lam_u
# Output: alpha_u (R_s,)

def unseen_coefficients(E_seen, e_u, M, lam_u):
    # Minimise the M-weighted reconstruction error plus L2 penalty  [eq. 6]
    # The paper does not state the solver.
    alpha_u = solve_quadratic(E_seen, e_u, M, lam_u)
    return alpha_u

Key tables are reproduced in Hyperparameters and Results.

Hyperparameters

Results

Author-reported. Setup: top-1 accuracy (%) or mAP, four benchmarks, deep features from GoogLeNet or VGG-verydeep-19 unless noted §5. Baselines are the best listed in each table, not the best possible.

Supervised ZSL, attributes tab. 4 (VGG on the left, GoogLeNet on the right):

Note: the "Ours" row in tab. 4 differs from the reduced-auto row in tab. 3 for AwA-G (80.83 vs 80.43), SUN-G (87.00 vs 79.00) and CUB-V (43.33 vs 43.01); see Pitfalls.

Supervised ZSL retrieval, mAP, VGG tab. 5: aPY 43.85 (best listed MFMR'17 45.60), AwA 72.87 (best listed MFMR'17 70.80), SUN 80.20 (best listed Zhang'16 80.10), CUB 36.60 (best listed MFMR'17 30.60).

Unsupervised ZSL, top-1 tab. 6: word2vec AwA V 66.26, G 66.89; CUB V 34.40, G 32.42. GloVe AwA V 62.01, G 64.73; CUB V 32.08, G 29.66. Baselines on the same rows reach at most 61.10 (word2vec, AwA-G, Xian'16) and 32.50 (GloVe, CUB-G, Xian'16) tab. 6.

GZSL, ResNet-101 features, Xian et al. 2017 split, harmonic mean (HM) tab. 7: SUN HM 31.3, CUB 43.3, AwA 54.5, aPY 37.0. ZSL on the same features: SUN 49.7, CUB 53.8, AwA 52.6, aPY 39.3 (author-tuned for GZSL).

GZSL, GoogLeNet, Chao et al. 2016 split tab. 8: AwA HM 50.8, CUB HM 29.5. Listed baselines reach AwA HM 29.6 (MFMR'17) and CUB HM 22.3 (SynC). ZSL on the same features: AwA 76.2, CUB 44.0.

Few-shot, GoogLeNet tab. 10, top-1 with attributes: AwA 87.4 (CMT 85.1), CUB 56.9 (CMT 56.7). mAP with attributes: AwA 92.0 (CMT 88.4), CUB 58.0 (CMT 58.5; the baseline is higher). The text notes three labelled images per unseen class and reports 10-trial averages §5 Benchmark Comparisons.

Full per-dataset breakdown, GZSL/OSL/FSL, including the SUN and aPY splits: tab. 11 (aPY), tab. 12 (AwA), tab. 13 (CUB), tab. 14 (SUN). On aPY attributes: OSL 71.2, FSL 83.6, ZSL 40.7 (top-1). On AwA attributes: OSL 82.8, FSL 87.4, ZSL 76.2, GZSL HM 50.8 tab. 12.

Ablation, reduced seen set tab. 3 (top-1, V / G): all seen classes: aPY 45.84 / 50.64, AwA 73.19 / 64.74, SUN 84.50 / 87.00, CUB 39.86 / 42.31. Lasso in place of ℓ2: aPY 36.54 / 37.22, AwA 74.16 / 75.76, SUN 78.50 / 84.50, CUB 37.47 / 37.37. Automatic R_n: aPY 54.69 / 55.07, AwA 78.53 / 80.43, SUN 85.00 / 79.00, CUB 43.01 / 45.31. The average number of seen classes used by the automatic rule is 10.17 (aPY, of 20 seen), 20.00 (AwA, of 40), 74.70 (CUB, of 150) and 344.40 (SUN-10, listed as 717 total seen in the table although tab. 1 says 707) with attributes tab. 2.

Figures: Figure 1 visualises CAPDs for two unseen classes on a 2D t-SNE fig. 1. Figure 3 compares nearest, mid-range, farthest and random seen subsets fig. 3. Figure 5 shows confusion matrices for AwA fig. 5. Figure 6 shows precision-recall curves for AwA fig. 6. Figure 7 shows the fusion weights \delta_u and \delta'_u for AwA and CUB fig. 7.

Paper vs code

No official code. The paper's implementation section says "the code of our method will be released" §5 Implementation Details, footnote 1; no repository is linked in the text or the arXiv metadata. Searches on GitHub for "CAPD zero-shot", "CAPD ZSL", "class adapting principal directions", "unified zero-shot generalized few-shot", and the author names (user search for "shafin rahman") returned no matching repository. Only CapDec (an image captioning project by other authors) and a recreation of it matched "CAPD" and were excluded as unrelated.

Pitfalls

• Objective direction in §3.1.2. The text says "we minimize the following objective" §3.1.2, but eq. 4 is written as a max-min problem. Implement it as the max-min the equation states; the solver is not given.
• Metric solver not stated. The method depends on \mathbf{M} from eq. 4, but no algorithm or solver is given. A reimplementation must choose one, and it will change \boldsymbol{\alpha}_u and every downstream number.
• Cost L in eq. 2 for positive samples. The text uses the mean of the other seen embeddings, which is a choice, not a formula derived from the objective. Implement the case split exactly as written §3.1.1.
• Automatic R_n is ambiguous. The text says the number of seen classes "with the highest probability score" is R_n. KDE bandwidth, the grid for evaluating the density, and how the count is taken are not stated §3.2.
• Eq. 11 is written with a squared norm of a difference of two vector losses. The text says the two components are "mean reconstruction losses" and minimises their squared difference. Whether each term is a scalar or a vector is not stated, so the norm's meaning is ambiguous.
• Lambda and learning rate tuning. λ values are cross-validated but not reported. Learning rates are constant over 100 iterations per class, with no batch size or epoch count, which makes exact reproduction hard §5.
• Table inconsistencies. The "Ours" rows in tab. 4 and tab. 3 differ for AwA-G (80.83 vs 80.43), SUN-G (87.00 vs 79.00) and CUB-V (43.33 vs 43.01). The text also refers to "seen accuracy (acc_u)" when it means acc_s §5.2 GZSL. The reported ZSL numbers in the GZSL tables are tuned for GZSL, so they should not be read as the best ZSL performance §5.2 All results.
• Reduced-set numbers. The nearest-subset result in the reduced-set experiment §5.1 is explicitly not the final automatic result; the final experiments use the automatic rule.
• Few-shot weights. Eq. for \delta_u divides by a sum that is zero only if both CAPDs give zero response on every training image. Behaviour in that case is not stated.
• Splits. SUN appears with two splits (645/72 and 707/10) and GZSL uses an 80/20 split of seen images repeated over 10 trials §5. Results on one split do not transfer to the other.

Lineage

• Builds on: the ranking loss for image tagging in Mubarak et al. (2016), cited as the motivation for eq. 2 §3.1.1. Also the ZSL compatibility and attribute literature cited in §2.
• Compared against: DeViSE, SJE, ESZSL, SSE, LATEM, ALE, SYNC, CMT, DAP/IAP, ConSE, Zhang'16, Bucher'16, DSRL'17, MFMR'17, SCoRe'17, DMaP-I'17 (only as reported in the paper's tables tab. 4 tab. 6 tab. 7).
• Superseded by: not stated.