A Simple Exponential Family Framework for Zero-Shot Learning

In plain words

Most zero-shot methods place each class at one fixed point in a space built from attributes, then label a new example with the nearest point. This paper describes each class instead as a spread of likely feature values, a probability distribution with its own centre and width. The spread for a class with no training images is computed from that class's attribute description. A rule learned from the classes that do have images maps attributes to spreads, so an unseen class still gets one. Classifying a new example then means asking which class's spread it most likely came from. The same setup can also use unlabelled examples from the unseen classes, or a handful of labelled ones, to sharpen those spreads. The rule itself is a closed-form regression, which keeps the method simple to implement.

TL;DR

GFZSL models each class as an exponential-family distribution; the paper focuses on a diagonal Gaussian. Its mean and log-variance are regressed from the class attributes by linear or kernel ridge regression fitted on seen classes, so unseen-class distributions follow from attributes alone. With the nonlinear variant the authors report 80.83% (AwA), 56.53% (CUB-200) and 86.50% (SUN) in the inductive setting tab. 1. The released code is for non-commercial research, and several settings used in the experiments appear only in that code.

Problem

• Zero-shot learning must recognise classes that have no labelled training examples §1.
• Common methods represent each class as a fixed point in a semantic space. The paper argues this ignores variability within a class §1 §3.
• Most such methods lack a generative model of the data. The paper argues that a generative model lets unlabelled data from seen and unseen classes enter parameter estimation in a principled way §1.
• Setting: S seen classes have labelled training data, U unseen classes have none. Every class has an attribute vector of size K §2. In generalised zero-shot learning, test data may come from seen or unseen classes §2 §4.

Method

Each class c, seen or unseen, has a class-conditional exponential-family density with natural parameters θ_c §2 eq. 1. The parameters are a function of the class attributes, θ_c = f(a_c) §2.1 eq. 3. Conjugate priors are assumed on θ_c eq. 2. The experiments use a D-dimensional Gaussian with diagonal covariance: μ_c = f_μ(a_c) and σ²_c = f_σ²(a_c) §2.1 eq. 4–5. A full covariance is possible but adds parameters §2.1.

1. Seen-class parameters. Estimate μ̂_c and σ̂²_c for each seen class c by MLE or MAP. The paper says labelled and unlabelled seen data may both be used §2.2 eq. 6–7. The code uses the per-class sample mean and sample variance awa1.m.
2. Regression on log-variance. Re-parameterise ρ = log σ² so that the variance regression is unconstrained §2.2.1.
3. Linear gating. μ̂ = W_μ a and ρ̂ = W_σ a §2.2.1 eq. 8–9. Ridge solutions: W_μ = M Aᵀ (A Aᵀ + λ_μ I_K)⁻¹ eq. 11, and the same form for W_σ with R in place of M and λ_σ² eq. 13. M and R are D×S, A is K×S.
4. Kernel gating. Replace the linear map with a kernel k over attributes (a quadratic kernel in the experiments §4). Coefficients are α_μ = M (K + λ_μ I_S)⁻¹ eq. 17 and α_σ² = R (K + λ I_S)⁻¹ eq. 19. The paper writes λ_μ in eq. 19; see Pitfalls §2.2.2 eq. 16, 18.
5. Unseen-class parameters. Linear: μ̂_u = Ŵ_μ a_u and σ̂²_u = exp(Ŵ_σ a_u) eq. 14–15. Kernel: μ̂_u = α_μ k_u and σ̂²_u = exp(α_σ k_u), where k_u holds the kernel values between a_u and all seen-class attributes §2.2.2 eq. 20–21.
6. Other exponential families. The same recipe applies: estimate seen-class parameters, then regress them on attributes. The paper says closed forms exist for other families but does not derive them §2.2.3.
7. Inductive classification. Assign a test example to the unseen class under which it is most likely §2 §4.1.1. Section 2 also mentions a posterior rule; no class prior is given for the experiments.

Optional refinement with unlabelled data from unseen classes §2.3. Initialise a Gaussian mixture with U components from the inductive estimates, then run EM:

# Transductive / semi-supervised refinement, Gaussian mixture  [§2.3]
# Input:  unlabelled (N, D), mu (U, D), var (U, D) from the inductive step
# Output: refined mu (U, D), var (U, D), mixing weights pi (U,)

def refine_unseen(unlabelled, mu, var, max_iters):
    # Uniform mixing weights; means and variances from the inductive step  [§2.3]
    pi = ones(U) / U
    # Paper gives no convergence test; the code caps the iterations [§2.3]
    for _ in range(max_iters):
        # E step: posterior of each sample for each unseen class  [§2.3]
        resp = pi * gaussian_density(unlabelled, mu, var)   # (N, U)
        resp = resp / resp.sum(axis=1, keepdims=True)
        # M step: re-estimate mixing weights, means and variances  [§2.3]
        pi = resp.mean(axis=0)
        mu, var = weighted_mean_and_variance(unlabelled, resp)
    return mu, var, pi

In the transductive setting the unlabelled data is the test set itself. In the semi-supervised setting it is a separate pool §2.3.

Few-shot refinement with a small number of labelled examples per unseen class §2.4. The update is written as a conjugate-style update:

# Few-shot update for one unseen class  [§2.4] [eq. 22] [eq. 23]
# Input:  mu_prior, var_prior (D,) from the zero-shot step
#         examples (N_c, D) labelled examples of this class
# Output: mu_post (D,), var_post (D,)

def few_shot_update(mu_prior, var_prior, examples):
    n_c = examples.shape[0]
    # Mean: the prior mean counts as one extra observation  [eq. 22]
    mu_post = (mu_prior + examples.sum(axis=0)) / (1 + n_c)
    # Empirical variance taken around the prior mean  [§2.4]
    var_emp = ((examples - mu_prior) ** 2).mean(axis=0)
    # Precisions add: prior precision plus n_c / var_emp  [eq. 23]
    var_post = 1 / (1 / var_prior + n_c / var_emp)
    return mu_post, var_post

The paper says this can run online as labelled examples arrive §2.4.

Inductive prediction, as used in the experiments §2 §4.1.1:

# Inductive prediction: most probable unseen class  [§2] [§4.1.1]
# Input:  features (N, D); unseen_mu (U, D); unseen_var (U, D)
# Output: predicted unseen-class index per sample (N,)

def predict_unseen(features, unseen_mu, unseen_var):
    # Diagonal Gaussian log-likelihood per sample and class  [§2]
    scores = gaussian_log_lik(features, unseen_mu, unseen_var)  # (N, U)
    return scores.argmax(axis=1)

Figure 1 is a block diagram: seen-class data gives seen-class parameters, a regression maps attributes to parameters, unseen-class parameters follow, and transductive or few-shot refinement is an optional last stage fig. 1 §2.3.

Hyperparameters

Results

Author-reported. Setup: AwA, CUB-200 and SUN with VGG-19 fc7 features §4. Baseline numbers are taken from the papers that report them §4.1.1; the paper does not say whether each baseline used the same protocol. Baseline rows keep the paper's bibliographic keys in parentheses where two entries share a first author.

Inductive ZSL, accuracy (%), source: tab. 1. Top block: non-VGG features; bottom block: VGG-19 features, per the caption.

Transductive ZSL, accuracy (%), VGG-19 features, source: tab. 2. The caption mentions a standard deviation over 100 iterations; the GFZSL rows show none.

Transductive precision and recall (%), source: tab. 3.

Few-shot accuracy (%) with 2, 5, 10, 15, 20 labelled examples per unseen class, source: tab. 4. Figure 2 plots the AwA case against the SVM and inductive ZSL fig. 2; its values are not in the text.

Generalised few-shot accuracy (%) on unseen-class test examples, source: tab. 5. The paper says the model is biased toward seen classes and that the SVM does better §4.3.

The paper reports these averages as "Average (as printed)". Recomputed from the three columns they differ; see Pitfalls.

Paper vs code

The code is at vkverma01/Zero-Shot (commit above), linked from the paper's abstract footnote. It has four MATLAB scripts (awa1.m, awa2.m, cub.m, sun.m), two evaluation functions, and a kernel and log-density helper. The repository's README states that the code is for non-commercial research only. We cite it for comparison only and do not reproduce it.

• Linear model not in code. The paper reports linear rows in Tables 1–3 and defines the linear solution in closed form §2.2.1 tab. 1 tab. 2. All four scripts compute only the kernel (quadratic) model awa1.m. The linear rows have no code path.
• Hyperparameters. The paper chooses λ_μ and λ_σ² by cross-validation on a validation split and does not give the values §4. The code fixes them per dataset and contains no tuning code: AwA1 λ_μ=0.1, λ_σ²=1e8 awa1.m; AwA2 λ_μ=0.01 awa2.m; CUB-200 λ_μ=1, λ_σ²=1e7 cub.m; SUN λ_μ=0.1, λ_σ²=1e8 sun.m.
• Eq. 19 uses λ_μ. The paper's variance solution uses λ_μ eq. 19, but the code's variance regression uses its own variance regulariser [code: awa1.m#L46]. The code's choice fits the structure of eq. 13; we read eq. 19 as a typo.
• Log-variance offset. The code regresses log(σ² + 0.001), or +0.0001 for CUB awa1.m cub.m. The paper uses no offset §2.2.
• Variance estimator. The code uses MATLAB var, which divides by N−1 [code: awa1.m#L31]. The paper does not state the denominator §2.2.
• Kernel form. The code uses (1 + a·b)², a quadratic with a constant term kernelPoly.m. The paper says only "quadratic kernel" §4.
• No normalisation. The scripts load features and attributes and use them without any normalisation step awa1.m. The paper's text does not mention normalisation §4.
• Inductive scoring. The inductive classifier adds 0.05 to the predicted variance before scoring Inductive_setting.m awa1.m. Section 2 scores by likelihood with no added term §2.
• Transductive refinement. The paper uses EM with mixing weights and no stated cap or covariance floor §2.3. The code calls MATLAB/VLFeat vl_gmm with a custom initialisation, a covariance lower bound of 0.4 and at most 1000 iterations Transductive_setting.m. It then scores with the fitted covariances plus 0.05 Transductive_setting.m.
• Splits. The README says the code follows the split proposed by Xian et al. README.md. The paper names splits for CUB-200 (Akata et al.) and SUN (Jayaraman et al.) §4 but names none for AwA. The code loads AWA1.mat in awa1.m awa1.m; awa2.m also exists, uses AWA2.mat and a different λ_μ, and is not reported in the paper awa2.m.
• Feature extraction absent. The scripts load precomputed .mat files from a dataset/ folder that the README links to a download awa1.m README.md. The repository has no VGG-19 extraction code.
• Repetitions absent. The table captions refer to 100 iterations tab. 2 tab. 3. Each script runs one pass with no repetition loop.
• Few-shot and generalised few-shot. The repository has no code for either §2.4 §4.2 §4.3.

Pitfalls

• Eq. 19 typo. The variance regression in the nonlinear case is written with λ_μ eq. 19. Implement it with the variance regulariser, as the code does awa1.m.
• Averages do not match their columns. Recomputed from the three columns shown, the GFZSL linear average in Table 1 is 72.83 (printed 72.23), the nonlinear average is 74.62 (printed 74.59), and the GFZSL kernel average in Table 2 is 81.64 (printed 80.63) tab. 1 tab. 2. Recompute the averages yourself rather than quoting the printed ones.
• Few-shot text overstates the SVM comparison. The text says the few-shot model outperforms the SVM §4.2. In Table 4, the CUB-200 SVM is higher at 10, 15 and 20 shots (68.75, 73.87, 75.42 against 68.44, 70.11, 71.23) tab. 4. AwA and SUN agree with the text.
• Spread not reported for GFZSL. The captions of Tables 2 and 3 refer to spreads over 100 iterations, but the GFZSL rows carry no spread and the repository runs each script once. Treat GFZSL numbers as single runs unless you reproduce them with repeated runs.
• Few-shot update is non-standard. Eq. 22 gives the prior mean the same weight as one data point, whatever the variances are. Eq. 23 adds precisions, but the empirical variance is measured around the prior mean, not the sample mean §2.4 eq. 22–23. No code is provided, so test this against held-out data before relying on it.
• Scale sensitivity. The code's λ_σ² values (1e7 to 1e8) only make sense relative to the scale of the kernel matrix, which depends on the attribute scale. The scripts apply no normalisation awa1.m, so check the kernel matrix's magnitude and the predicted variances before relying on the nonlinear variance branch.
• Undocumented variance terms. The code adds +0.001 or +0.0001 inside the log, +0.05 at test time in both settings, and a 0.4 covariance floor in the GMM awa1.m Inductive_setting.m Transductive_setting.m. None appears in the paper. Reproducing the printed numbers requires them.
• No prior in inductive scoring. The paper allows a posterior rule, but no class prior is given for the experiments §2 §4.1.1.
• EM stopping not stated. The paper gives no convergence test §2.3. The code caps iterations at 1000 Transductive_setting.m.
• Diagonal covariance. The experiments assume diagonal covariance; a full covariance is possible but adds parameters §2.1.
• Generalised ZSL is weak. Unseen-class accuracy under generalised zero-shot learning is low, and the paper says the multi-class SVM still beats the model even with labelled unseen examples §4.3 tab. 5.
• Licence. The code is research-only; do not port it into a commercial product. Implement from the paper and check the paper's equations.

Lineage

• Builds on: linear and kernel ridge regression and Gaussian mixture EM, named in §2 §2.2 §2.3. No prior card in this wiki.
• Compared against: Akata et al., Changpinyo et al., Bucher et al., Zhang et al., Wang et al., Romera and Torr, Guo et al. (zero-shot tables tab. 1–2); a multi-class SVM in the few-shot tables tab. 4–5.
• Superseded by: not stated in the paper or in this wiki.