Meta-Learning: A Survey

In plain words

Meta-learning means learning from past learning jobs. Instead of starting every new machine learning task from scratch, you keep records of which settings worked on which tasks, what those tasks looked like, and the models that came out of them, and you use those records to pick a good starting point for the next task.

This survey sorts the existing ideas by the kind of record they use. The first kind is just the scores of settings on past tasks: a shortlist of settings that usually work, or a guess at which knobs matter. The second kind is a description of each task (its size, its shape, simple statistics), so that similar tasks can share what they learned. The third kind is the trained models themselves: reusing their weights, or learning how to update a model quickly. The author argues that the more similar past tasks are to the new one, the more of this record is useful, and that choosing what counts as "similar" is the central difficulty.

TL;DR

A survey of meta-learning, grouped by the meta-data used: past model evaluations, task meta-features, and prior model parameters. It gives one shared formulation, a list of method families with their trade-offs, and the size of the experiments behind several cited studies. It reports no new experiments and has no official code.

Problem

Meta-learning needs a systematic way to store and reuse experience. The text lists what such meta-data contains: configurations used to train models, their evaluations, the learned parameters, and meta-features that describe the task §1. It also states that meta-learning gives no benefit when a new task is unrelated to the earlier ones, so task similarity is the key assumption §1.

Method

The survey does not propose a method. It gives one shared formulation and then groups the surveyed work into families by the meta-data each one uses §1.

Shared formulation §2. There is a set of prior tasks t_j in T, and a set of learning algorithms, each defined by a configuration theta_i in Theta. Theta may be discrete, continuous or mixed, and may cover hyperparameters, pipeline components or network architecture components §2. The prior evaluation P_ij = P(theta_i, t_j) is a scalar score under a fixed measure (for example accuracy) and a fixed evaluation technique (for example cross-validation) §2. P_new holds the evaluations already observed on a new task, and a meta-learner L is trained on P together with P_new to recommend configurations for the new task §2. P_new grows over time as the recommended configurations are evaluated §2.3.

The families:

1. Learning from model evaluations (P) §2.
   1. Task-independent recommendations: a portfolio of configurations, ranked per task by success rate, AUC or significant wins, then aggregated by average rank into a global ranking §2.1. Rankings can also trade accuracy against training time §2.1.
   2. Configuration space design: learn which hyperparameters matter, either by variance explained (functional ANOVA) or by the gain from tuning over a learned default, and learn defaults for each algorithm §2.2.
   3. Configuration transfer: measure task similarity from relative landmarks (pairwise score differences) and use active testing §2.3.1; or fit a surrogate model per prior task and weight the surrogates by how well they predict the new task §2.3.2; or learn a joint task representation, for example with a multi-task model §2.3.3; or use bandits or distribution divergences over the best configurations §2.3.4.
   4. Learning curves: reuse partial learning curves to predict the final score and stop poor runs early §2.4.
2. Learning from task properties (meta-features m(t_j)) §3.
   1. Meta-features: a hand-designed vector per task, grouped as simple, statistical, information-theoretic, complexity, model-based and landmarker features, listed with formulas in tab. 1 §3.1.
   2. Learned meta-features: a model maps task meta-features and evaluations to a learned representation, for example pairwise "A beats B" features §3.2, or a Siamese network over tasks §3.2.
   3. Warm starts: evaluate the best configurations of the most similar prior tasks first, then use them to seed a genetic search or Bayesian optimisation §3.3.
   4. Meta-models: learn a mapping from meta-features to good configurations, either as a ranking §3.4.1 or as a performance or runtime predictor §3.4.2.
   5. Pipeline synthesis: warm-start or guide search over whole pipelines and their steps §3.5.
   6. Whether to tune: predict whether tuning an algorithm on a task is worth the cost §3.6.
3. Learning from prior models (weights W and configurations) §4.
   1. Transfer learning: start from a source model and fine-tune it on the target task §4.1.
   2. Meta-learning in neural networks: learn the update rule or optimiser itself, from recurrent networks, memory-augmented networks or LSTMs §4.2.
   3. Few-shot learning: learn an initialisation (MAML, Reptile), a non-parametric matcher (matching networks), a prototype model, or a memory-based learner §4.3.
   4. Beyond supervised learning: applications to reinforcement learning, active learning, density estimation and recommendation §4.4.

Pseudocode. The survey gives no algorithm box. The two sketches below restate the text in our own structure; the function names are ours, not the paper's.

# Top-K anytime search over recommended configurations  [§2.1]
# Input:  ranked configurations theta*_1..theta*_K, new task t_new
# Output: best configuration evaluated on t_new

def top_k_search(ranked_configs, t_new, budget):
    # Try configurations in ranked order; stop after K, when the
    # budget runs out, or when a model is accurate enough  [§2.1]
    best_config = None
    best_score = None
    for config in ranked_configs:
        if out_of_budget(budget):
            break
        score = evaluate(config, t_new)
        if best_score is None or score > best_score:
            best_score = score
            best_config = config
    return best_config

# Meta-learner over evaluations  [§2]
# Input:  prior evaluations P, new-task evaluations P_new, new task t_new
# Output: recommended configurations for t_new

def recommend(P, P_new, t_new):
    # Train L on all known evaluations  [§2]
    meta_learner = fit_meta_learner(union(P, P_new))
    # Predict configurations for the new task  [§2]
    return meta_learner.predict(t_new)

Hyperparameters

Results

This survey reports no results of its own, and the text does not describe new experiments. The table lists the study sizes that the survey states for cited work. These are author-reported figures from the cited studies, not reproduced here.

Paper vs code

No official code. The paper has no code or project link; its only identifier is the author's e-mail address [title page]. A GitHub repository search for "meta-learning survey vanschoren" returned no repositories. A survey has no implementation to compare against, so nothing in the method description is checked against code.

Pitfalls

• A survey defines no algorithm an agent can implement exactly. Use the cited works for implementation details; this card only maps the field.
• The text does not fix a single meta-learner signature. Section 2 describes L as recommending configurations, section 3 as predicting performance, and section 4 as learning how to train a learner §2 §3 §4. Read each family against its own definition.
• tab. 1 prints the Fisher discriminant ratio with a difference of the two class variances in the denominator. The ratio is usually written with a sum of the two variances. Check the original source before implementing it.
• Ranking measures for top-K and portfolio methods are not fixed. Success rate, AUC, significant wins and time-aware variants are all named, with no preferred choice §2.1.
• The stopping rule for top-K search lists K, a time budget and an accuracy threshold. No threshold value is given §2.1.
• The relative-landmark method only works for configurations that were evaluated on many prior tasks §2.3.1.
• Learning one Gaussian process over all prior tasks is described as less scalable than one per task §2.3.3. A GP scales cubically, which the text gives as the reason one bandit-based method is less scalable §2.3.4.
• Collaborative filtering for configurations needs some evaluations on the new task before it can predict anything (cold start) §3.3.
• Meta-feature selection and normalisation matter; the optimal set depends on the application, and the text does not give one recipe §3.1.
• The text says that no learning-curve work for neural architecture search uses curves from other tasks. That is true as of the 2018 text only §2.4.
• Transfer from large pre-trained image models can fail when the target task is not similar §4.1.
• The text does not state the cost of the meta-gradient in MAML-style methods, or of the learned optimisers §4.2 §4.3.
• The survey was published on arXiv in October 2018. Later work is not covered.

Lineage

• Builds on: the cited works in sections 2 to 4. The survey introduces no method of its own §1.
• Compared against: not applicable.
• Superseded by: not known from content in this wiki.