A Survey of Deep Active Learning

In plain words

Deep networks need lots of labeled examples, and labels are often the expensive part: a radiologist has to outline every lesion, a linguist has to tag every sentence. Active learning flips the usual order. Instead of labeling a random pile of data up front, you train a model on a few labels, let the model point at the unlabeled examples it would learn most from, have a human label only those, retrain, and repeat until the budget runs out.

This survey asks why that classic idea does not simply carry over to deep learning, and collects what people have done about it. Three things get in the way. A network's softmax output looks like a confidence score but is often overconfident, so "pick the examples the model is least sure about" picks badly. Active learning traditionally asks for one label at a time, while deep nets need many labels and are too slow to retrain after each one. And older active-learning methods assume fixed features, while deep nets learn their features as they train.

The responses fall into a few families: ask for labels in batches that are both uncertain and different from each other; get better uncertainty from Bayesian tricks such as keeping dropout on at prediction time; pick a small set of examples that covers the whole dataset; let the model also learn from the unlabeled pool through pseudo-labels, generated data or semi-supervised training; and decide when to stop asking. The survey ends with a warning that published results disagree, sometimes even on how well plain random selection does, so comparisons across papers are unreliable.

TL;DR

A taxonomy survey (v2, December 2021; references up to November 2020), not a method paper: no experiments, no code. It names three obstacles to combining AL with DL (unreliable softmax uncertainty, too few labels for deep training, AL and DL pipelines optimized separately) §2 and maps the literature onto them tab. 1: query-strategy optimization (batch mode, uncertainty/hybrid, deep Bayesian AL, density/core-set, automated design) §3.1, data expansion of labeled samples §3.2, generic frameworks §3.3 and stopping strategies §3.4. It gives the BALD and BatchBALD acquisition functions, margin and entropy sampling, the Exploration-P uncertainty-minus-redundancy objective and the Bayesian posterior predictive eq. 1–9. The application section lists tasks and datasets across vision, NLP and other fields §4 tab. 2. Its most practically useful content is the reproducibility warning in §5. Read it for orientation and a reading list; go to the primary papers before implementing.

Problem

AL selects the most valuable unlabeled samples for an oracle to label so a model reaches a given performance with as few labels as possible §1.2; DL extracts features automatically but needs large labeled datasets, which is impractical where labeling needs experts (the survey's example is clinicians annotating COVID-19 lung lesion images) §1.1. Classic AL struggles with high-dimensional data and relies on pre-extracted features §1.2. Combining them is hard for three reasons §2:

1. Model uncertainty. Softmax probabilities are overconfident; the survey cites work where softmax response (SR) as a confidence measure did worse than random sampling §2.
2. Insufficient labels. AL yields few labels, DL needs many, and one-by-one querying does not fit DL training §2.
3. Pipeline inconsistency. AL query strategies assume fixed features, while DL learns features and classifier jointly; fine-tuning a DL model inside an AL loop, or treating the two as separate problems, can diverge §2.

The survey's literature base: about 270 papers found on DBLP with the keyword "deep active learning", filtered by hand and supplemented manually; the final references include 103 conference papers, 153 journal papers, 3 books, 1 research report, 1 dissertation and 28 unpublished papers, with the latest from November 2020 §1, footnote.

Method

The "method" of a survey is its taxonomy. Section numbering of the compiled paper: §1 Introduction (§1.1 Deep Learning, §1.2 Active Learning), §2 Necessity and challenge of combining DL and AL, §3 Deep Active Learning (§3.1 Query strategy optimization with §3.1.1 BMDAL, §3.1.2 Uncertainty-based and hybrid, §3.1.3 DBAL, §3.1.4 Density-based, §3.1.5 Automated design; §3.2 Data expansion of labeled samples; §3.3 Generic framework; §3.4 Stopping strategy), §4 Applications (§4.1 Visual data processing, §4.2 NLP, §4.3 Other), §5 Discussion and future directions, §6 Summary. The appendix is empty. Figures 1–9, tables 1–2, numbered equations 1–9 (the margin formula in §3.1.2 is inline and unnumbered).

AL scenarios §1.2

• Membership query synthesis: the learner may request a label for any point in input space, including points it generates.
• Stream-based selective sampling: decide per incoming sample whether to query its label; suited to devices with limited storage and compute.
• Pool-based: rank the whole unlabeled pool and query the best; the common setting and the one the survey uses.

The DeepAL loop §3 fig. 1c

DeepAL loop  [§3] [fig. 1c]
Input: initial labeled set L0, unlabeled pool U, oracle, label budget
1. Initialize or pretrain model parameters theta on L0
2. repeat:
   a. Extract features of samples in U with the current model
   b. Score U with the query strategy and select a batch
   c. Oracle labels the batch; add it to L and remove it from U
   d. Train the model on L
3. until the label budget is exhausted or a predefined stop condition holds (§3.4)
Output: trained model, labeled set L

Optimization target in the supervised setting, with unlabeled pool U^n of n samples, labeled set L^m of m samples, model f:\mathcal{X}\to\mathcal{Y} and loss \ell \ge 0 §3.1:

\mathop{\arg\min}_{L^m\subseteq U^n,\ (\mathrm{x,y}) \in L^m,\ (x,y) \in U^n} \mathbb{E}_{(x,y)}[\ell(f(\mathrm{x}),\mathrm{y})] eq. 1

with the goal of keeping m \ll n as small as possible at a predetermined accuracy §3.1.

Batch mode DeepAL (BMDAL) §3.1.1

The survey treats batch querying as the defining difference between DeepAL and classical AL: one-by-one querying retrains often for little new data and tends to overfit §3.1.1. At each acquisition step, pick the batch maximizing a batch acquisition function given the model trained on L (also written D_{train}):

\mathcal{B}^*= \mathop{\arg\max}_{\mathcal{B}\subseteq U} a_{batch}(\mathcal{B},f_{\theta}(L)) eq. 2

Naive top-K BALD. Score each sample independently with BALD's mutual information between prediction and parameters, then take the top K §3.1.1:

a_{\mathrm{BALD}}\left(\{x_{1}, \ldots, x_{b}\}, \mathcal{P}(\omega \mid D_{train})\right)=\sum_{i=1}^{b} \mathbb{I}\left(y_{i} ; \omega \mid x_{i}, D_{train}\right) \mathbb{I}\left(y ; \omega \mid x, D_{train}\right)=\mathbb{H}\left(y \mid x, D_{train}\right)-\mathbb{E}_{\mathcal{P}(\omega \mid D_{train})}\left[\mathbb{H}\left(y \mid x, \omega, D_{train}\right)\right] eq. 3

Higher mutual information means higher uncertainty. Because each sample is scored independently, the batch can be full of informative but near-duplicate samples §3.1.1 fig. 2.

BatchBALD scores the batch jointly §3.1.1:

a_{\text{BatchBALD}}\left(\{x_{1}, \ldots, x_{b}\}, \mathcal{P}(\omega \mid D_{train})\right)=\mathbb{I}\left(y_{1}, \ldots, y_{b} ; \omega \mid x_{1}, \ldots, x_{b}, D_{train}\right) \mathbb{I}\left(y_{1: b} ; \omega \mid x_{1: b}, D_{train}\right)=\mathbb{H}\left(y_{1: b} \mid x_{1: b}, D_{train}\right)-\mathbb{E}_{\mathcal{P}(\omega \mid D_{train})} \mathbb{H}\left(y_{1: b} \mid x_{1: b}, \omega, D_{train}\right) eq. 4

Uncertainty-based and hybrid strategies §3.1.2

• Margin sampling: M = P(y_1 \mid x) - P(y_2 \mid x) for the two most probable labels; select the top-K samples with the smallest M §3.1.2 (inline, unnumbered).
• Entropy for a k-class task; select the top-K with the largest value §3.1.2:

\mathbb{E}(x)=-\sum_{i=1}^{k} P(y_i\mid x) \cdot \log P(y_i\mid x) eq. 5

• DFAL uses DeepFool adversarial examples to approximate each sample's distance to the decision boundary, arguing plain uncertainty is fooled by adversarial examples §3.1.2.
• Exploration-P (hybrid, explicit): learn features f(x) with a deep net, set uncertainty of a selected set E(S)=\sum_{x_i\in S}\mathbb{E}(x_i), and penalize redundancy §3.1.2:

R(S) = \sum_{x_i\in S}\sum_{x_j\in S}Sim(x_i,x_j),\quad Sim(x_i,x_j) = f(x_i)\,\mathcal{M}\,f(x_j) eq. 6

\mathrm{I}(S)=E(S)-\frac{\alpha}{|S|} R(S) eq. 7

\mathcal{M} is a similarity matrix: identity gives a feature dot product, or it can be learned with f; \alpha trades uncertainty against redundancy §3.1.2. Early in training it explores with random sampling §3.1.2.

• DMBAL: K-means with informativeness as sample weights, so the selected mini-batch is informative and diverse §3.1.2.
• WI-DL: representativeness during unsupervised DBN feature learning, uncertainty during supervised fine-tuning, combined by weighted incremental dictionary learning §3.1.2.
• BADGE (implicit): sample points that are diverse and high-magnitude in a "hallucinated" gradient-embedding space; balances uncertainty and diversity with no hand-tuned trade-off hyperparameter §3.1.2.
• WAAL (explicit): model AL as distribution matching with Wasserstein distance; split into DNN parameter optimization and batch selection §3.1.2.
• TA-VAAL: add the loss-prediction module (from LLAL) and RankCGAN to VAAL so both data distribution and task uncertainty enter selection §3.1.2 fig. 3.

The survey states that diversity methods do better with more classes and larger batches, uncertainty methods with smaller batches and less varied data §3.1.2.

Deep Bayesian active learning (DBAL) §3.1.3

Probabilistic network f(\mathrm{x};\theta) with prior p(\theta) (usually Gaussian) and likelihood p(\mathrm{y}=c\mid\mathrm{x},\theta)=\mathrm{softmax}(f(\mathrm{x};\theta)) §3.1.3:

p(\theta\mid X, Y)=\frac{p(Y \mid X, \theta)\, p(\theta)}{p(Y \mid X)} eq. 8

p\left(\hat{\mathrm{y}} \mid \mathrm{x}^{*}, X, Y\right)=\int p\left(\hat{\mathrm{y}} \mid \mathrm{x}, \theta\right) p(\theta \mid X, Y)\, d \theta=\mathbb{E}_{\theta \sim p(\theta \mid X, Y)}[f(\mathrm{x} ; \theta)] eq. 9

• DBAL (Gal et al.): Bayesian CNN with a Gaussian prior on weights, variational inference for the posterior, BALD as acquisition; in practice MC-dropout supplies the posterior samples, and the survey notes dropout has been shown equivalent to variational inference §3.1.3.
• Criticisms cited: the core-set paper calls DBAL unsuitable for large datasets because of batch sampling; DAL's analysis finds MC-dropout uncertainty performs about like softmax response §3.1.3.
• DEBAL: ensembles plus MC-dropout, motivated by mode collapse in variational inference causing overconfidence §3.1.3.
• ACS-FW: batch selection as a sparse approximation of the full-data log-posterior (Bayesian coresets), solved with Frank-Wolfe and random projections; reported better coverage of the data manifold than MAXENT and BALD §3.1.3.
• DPEs: regularized ensembles approximating a deep BNN, evaluated in large-scale visual AL §3.1.3.
• ActiveLink: Bayesian neural link prediction on knowledge graphs, plus meta-learning-based incremental training to avoid both retraining from scratch and bias toward new or old samples §3.1.3.

Density-based methods §3.1.4

• FF-Active: farthest-first traversal in a representation layer's activation space §3.1.4.
• Core-set: select a subset solving the k-Center problem; needs a large distance matrix over the unlabeled pool, costly at scale §3.1.4.
• Active Palmprint Recognition and DAL (Discriminative Active Learning): treat AL as binary classification between labeled and unlabeled sets and query the samples that are most clearly distinguishable from the labeled set, so the labeled set comes to match the pool's distribution. DAL samples in proportion to density, avoiding the over-sampling of sparse regions that farthest-point coverage causes §3.1.4.
• Batch size: most work tunes small batches; Chitta et al. query 10k or 500k samples at a time using hundreds of models and reused checkpoints, and report that training on an AL-chosen subset can beat training on the full dataset §3.1.4.
• Gradient Analysis (GA): in some datasets per-sample importance differs too little for subset training to pay off §3.1.4.
• Beluch et al.: ensembles beat MC-dropout and density-based selection, giving better-calibrated uncertainty and offsetting class imbalance during acquisition §3.1.4.

Automated design §3.1.5 fig. 4

Standard AL pipeline: oracle → predictor (here a BNN) → guide (a fixed acquisition function) → oracle fig. 4a. RAL replaces the fixed guide with a probabilistic policy BNN trained by RL from oracle feedback fig. 4b. DRAL (person re-ID): for each probe, an RL agent picks gallery instances sequentially; the oracle gives binary positive/negative feedback, from which rewards are computed fig. 4c. Fang et al. recast heuristic AL as RL. Active-iNAS runs incremental neural architecture search during AL instead of assuming a fixed architecture §3.1.5. Munjal et al. study robustness and reproducibility of DeepAL §3.1.5.

Data expansion of labeled samples §3.2

• CEAL: uncertain samples go to the oracle; high-confidence samples get pseudo-labels from the CNN; both fine-tune the CNN each round §3.2 fig. 5.
• WI-DL: unsupervised feature learning on all data, supervised fine-tuning on labeled data §3.2.
• GAAL: first use of a GAN in AL querying, generating samples meant to be more informative; BGADL combines GAAL, Bayesian data augmentation, ACGAN and a VAE to generate samples in disagreement regions §3.2 fig. 6.
• VAAL: a VAE trained on labeled and unlabeled data tries to fool an adversarial discriminator into calling every point labeled; the discriminator's view of the latent space picks samples, reducing outlier sensitivity and avoiding the distance concentration p-norm methods suffer in high dimensions §3.2.
• ARAL: extends VAAL with adversarial representation learning on real and generated data; encoder E, generator G, discriminator D, classifier C, sampler S, all trained jointly §3.2 fig. 7.
• SSAL: unsupervised pretraining on all data before AL starts, then in each cycle supervised training on labeled data followed by semi-supervised training on all data; the authors find this training change helps more than the differences between sampling strategies §3.2.

Generic frameworks §3.3

Early DeepAL trained the deep model on labeled data, then sampled from its outputs, treating the two as separate problems §3.3. Tighter designs: CEAL and HDAL (face recognition) §3.3; FCN-based suggestive annotation for medical segmentation (uncertainty plus similarity on FCN features) §3.3; defect detection with an average-margin rule controlling per-category sampling ratios §3.3. The survey argues uncertainty should come from both the feature-extraction and task stages, not only the last layer fig. 8: AL-MV treats several hidden layers as multi-view data with adaptive per-layer uncertainty weights; LLAL attaches a small loss-prediction module fed by multiple hidden layers, trains it jointly with the target loss, and queries the top-K unlabeled samples by predicted loss, reported competitive on classification, detection and human pose estimation §3.3 fig. 9; a deeply supervised U-Net uses multi-level hidden outputs for finger-bone segmentation §3.3.

Stopping strategies §3.4

Most DeepAL uses predefined criteria: maximum iterations, minimum change in accuracy, minimum number of labels, or a target accuracy §3.4. Stabilizing Predictions (SP) sets aside an unlabeled stop set and stops when predictions on it stop changing; no extra labels needed §3.4.

Challenge-to-solution map tab. 1

The table lists citations rather than names; the names here are the methods the text associates with those citations.

Applications §4 tab. 2

The publication column of tab. 2 is omitted here; see the paper.

Hyperparameters

The survey trains nothing. These are the few concrete values and free parameters it states.

Results

The survey runs no experiments. Numbers below are as the survey reports them from other papers.

Conflicting findings the survey highlights §5: the core-set and DFAL papers report diversity methods always beating uncertainty methods, and uncertainty methods doing worse than random sampling; LLAL reports otherwise.

Paper vs code

No official code.

Pitfalls

• Random baselines are not comparable across papers. Reported random-sampling accuracy differs by 13% between two papers at the same CIFAR-10 setting, and one method's results by 8% between two papers §5. The survey recommends a common evaluation platform, shared experiment snapshots, several runs with different initializations, ablations and transfer experiments §5. Always rerun random sampling yourself under your own training recipe.
• Top-K on per-sample scores gives redundant batches. Eq. 3 sums independent scores, so near-duplicates all score high §3.1.1 fig. 2. Use a joint score (eq. 4) or an explicit diversity term (eq. 6–7, DMBAL, BADGE, core-set).
• Eq. 4 hides its cost. The survey does not say how to compute the joint entropy \mathbb{H}(y_{1:b}\mid\ldots); for k classes the joint label space has k^b configurations, so an implementation needs the estimator from the BatchBALD paper.
• Eq. 9 is loosely written. The integrand uses \mathrm{x} where it should be \mathrm{x}^*, and the right side is \mathbb{E}[f(\mathrm{x};\theta)] although the stated likelihood is \mathrm{softmax}(f) §3.1.3. Average the softmax probabilities over posterior samples, not the logits.
• Symbol clashes. Entropy is written \mathbb{E}(x) eq. 5, the same glyph as expectation in eq. 1, 3, 4 and 9; E(S) in eq. 7 is the summed entropy. U^n is called unlabeled but defined as \{\mathcal{X},\mathcal{Y}\} §3.1. Eq. 1 does not contain the "minimize m" goal; that lives only in the text.
• Eq. 6 includes i=j. The double sum counts each sample's self-similarity, so with \mathcal{M}=I the redundancy term also penalizes feature norm. Whether Exploration-P excludes the diagonal is not stated here.
• MC-dropout is not automatically better than softmax. The survey cites an analysis finding it performs about like softmax response §3.1.3, and ensembles giving better-calibrated uncertainty than MC-dropout §3.1.4.
• Core-set needs a pool-wide distance matrix, which becomes expensive on large unlabeled pools §3.1.4; farthest-first coverage also over-samples sparse regions §3.1.4.
• Strategy choice depends on batch size and data. Diversity methods favor larger batches and more classes, uncertainty methods smaller batches §3.1.2; some datasets gain little from subset selection at all §3.1.4 §5.
• Training recipe can dominate the query strategy. Semi-supervised and unsupervised training across cycles (SSAL) was reported to matter more than the sampler §3.2. Compare strategies with the same training method.
• Retraining vs incremental training. Retraining from scratch every cycle is expensive; naive incremental training biases the model toward new or early samples §3.1.3 §5. The survey offers no general fix beyond ActiveLink's meta-learning.
• Stopping thresholds are unspecified. Predefined criteria risk stopping early (performance loss) or late (wasted budget) §3.4; SP's stop-set size and stability measure are not given here.
• Snapshot in time. References end in November 2020 (v2 is from December 2021); later work is not covered.

Lineage

• Builds on: Settles' active learning literature survey (2009 technical report) and the classic AL query strategies it covers (uncertainty, diversity, expected model change) §1.2. No card in this wiki.
• Compared against: no methods of its own; surveys BALD, BatchBALD, DBAL, core-set, BADGE, VAAL, LLAL, CEAL, DFAL, WAAL, TA-VAAL, ARAL, SSAL and others §3.
• Superseded by: not known from content in this wiki.