fastai: A Layered API for Deep Learning

In plain words

fastai is a Python library for training deep learning models, built on top of PyTorch. It is meant for two kinds of people at once. Practitioners get a few high-level functions that pick sensible defaults, so a working image classifier takes only a handful of lines. Researchers get smaller building blocks that they can mix, replace and extend without rewriting the whole pipeline.

The trick is layering. Each high-level function is assembled from middle-level pieces, and those are built on lower-level ones. You can drop down a layer when you need control, and you never have to learn the bottom layer just to get started. Training itself is driven by callbacks: small hooks that can look at and change almost anything during training, such as the data, the predictions, the gradients or the learning rate.

The library also takes a few specific choices that the authors found work well in practice, including a one-cycle learning-rate schedule and a learning-rate finder. Its aim is to be easy for beginners, hackable for experts, and to avoid forcing a choice between the two.

TL;DR

• fastai is a layered PyTorch library: high-level one-call training for vision, text, tabular and collaborative filtering, built on mid-level data, training and optimizer APIs and low-level transform and tensor abstractions §1 §2 §4 §5.
• Its central mechanisms are the data block API, a Learner that bundles model, optimizer, loss and data, two-way callbacks that can change training at any point, and a generic optimizer built from stats and steppers §3.1 §4.1 §4.2 §4.3.
• Results are author-reported and qualitative: a course rewritten on the library, ResNet-50 on ImageNet in 18 minutes in a DawnBench entry, and short code examples. The paper has no benchmark tables §1 §8.
• The paper targets fastai v2.0.0 (pre-release); the code at the pinned commit is fastai 2.8.13, so names and defaults have drifted §1 [code: fastai/init.py#L1].

Problem

• Deep learning libraries of the time often offered either a high-level API or a flexible low-level one, and it was hard to combine both §1. Many libraries had a high-level API and a low-level API with nothing in between, so customizing the high level meant rewriting parts of it §4 intro.
• Transfer learning is sensitive to details. In the authors' tests, batch-normalization handling changed the error rate by as much as 300% depending on configuration, and only one approach worked across all their datasets §4.1.
• Adding a technique to an existing research codebase often needed a rewrite. The paper's example is mixup, whose released code worked on one dataset, metric and optimizer §3.3.
• Data preparation is a common source of mistakes, such as forgetting a validation set or shuffling it §2 intro §1.
• The paper assumes PyTorch as the base and does not aim to hide it: PyTorch modules can be used inside fastai, and fastai components can be used inside PyTorch code §1 §3.2.

Method

The library is organised in three layers, plus a development environment §1 §5:

• High level: *_learner functions and *DataLoaders classes for vision, text, tabular and collaborative filtering. They choose defaults from the data and the model §1 §2.
• Middle level: the data block API, DataLoaders, Learner, two-way callbacks, the generic optimizer, the metric API, DataLoader customisation, and architecture building blocks such as SequentialEx, MergeLayer and XResNet §4.
• Low level: transform pipelines with encode and decode, type dispatch, semantic tensor types, GPU batch augmentation and convenience utilities §5.
• nbdev: a notebook-based tool used to write the library and generate its documentation §6.

Steps as the paper describes them:

1. Describe the data as blocks. A data block set defines get-items, split, label, processing and batching in that order. Blocks are mixed and matched, for example image input with a category label §3.1. The paper gives this as a functional DataBlock with keyword arguments, not a chained API §3.1.
2. Transform items and batches. Item transforms run per item, usually on the CPU. Batch transforms run on the mini-batch, on the GPU when available. Images in a batch must share a size first §2.1. A transform can have an inverse (decode), which is used to display batches and predictions §5.2 §4.6.
3. Bundle the training state in a Learner. It holds the model, optimizer, loss, data and metrics. A splitter divides the model into parameter groups, which can be frozen or given different learning rates §4.1. The loss and optimizer are optional when they can be inferred §4.1.
4. Run training through callbacks. Each step of the loop fires an event, and callbacks can read or edit the state, skip a batch, an epoch or a phase, or cancel training §4.2. The paper's per-batch loop is summarised below.
5. Schedule the learning rate and momentum together. The one-cycle policy warms up the learning rate while momentum goes down, then anneals the learning rate while momentum goes back up §3.1 fig. 8.
6. Pick a starting learning rate. The learning-rate finder runs a short mock training with a learning rate that grows exponentially over 100 iterations, then reads a suggestion off the loss curve §3.1 fig. 9 schedule.py schedule.py.
7. Build optimizers from stats and steppers. Statistics such as moving averages are tracked separately from the update rules. Decoupled weight decay (AdamW-style) is a single switch §4.3.
8. Compute metrics incrementally. A metric has reset, accumulate and value, so it does not need to keep every prediction in memory §4.4.
9. Compose architectures from blocks. SequentialEx gives each layer access to the block input, so skip connections need no custom forward. MergeLayer takes add or concatenate. XResNet covers many ResNet variants through parameters §4.8.

The per-batch training loop from the paper (§4.2), with a callback at every step. Event names follow the paper.

# Train on one batch; every step fires a callback event  [§4.2]
# Input: batch (inputs, targets); callbacks cb(event) that may edit state
# Output: none (weights and callback state are updated in place)

def train_one_batch(batch, model, loss_func, optimizer, cb):
    # Split the batch into model inputs and targets  [§4.2]
    # [code: fastai/learner.py#L203-L205]
    inputs, targets = split(batch)
    cb('begin_batch')
    try:
        # Forward pass; callbacks can change the predictions  [§4.2]
        predictions = model(inputs)
        cb('after_pred')
        # No targets means inference only, so stop before the loss  [§4.2]
        if len(targets) == 0: return
        loss = loss_func(predictions, targets)
        cb('after_loss')
        # Backward and step only while training  [§4.2]
        if not training: return
        loss.backward()
        cb('after_back')
        optimizer.step()
        cb('after_step')
        optimizer.zero_grad()
    except CancelBatchException:
        # A callback may skip the rest of this batch  [§4.2]
        cb('after_cancel')
    finally:
        # Runs on normal exit, early return and cancel  [§4.2]
        cb('after_batch')

The one-cycle schedule as the paper describes it and as the default code computes it. The paper states no numeric defaults; the values below are the code's defaults, not the paper's.

# One-cycle schedule for learning rate and momentum  [§3.1] [fig. 8]
# Input: progress in [0, 1]; lr_max; moms = (high, low, high); pct_start
# Output: (lr, mom) for this point of training
# cosine(a, b, t): cosine interpolation from a (t=0) to b (t=1)

def one_cycle(progress, lr_max, moms, pct_start):
    # Warm-up: lr rises from lr_max / div to lr_max  [§3.1]
    # [code: fastai/callback/schedule.py#L114-L121]
    if progress < pct_start:
        fraction = progress / pct_start
        lr = cosine(lr_max / 25, lr_max, fraction)  # div=25 in code
        mom = cosine(moms[0], moms[1], fraction)
    # Annealing: lr falls to lr_max / div_final; momentum rises back  [§3.1]
    else:
        fraction = (progress - pct_start) / (1 - pct_start)
        lr = cosine(lr_max, lr_max / 1e5, fraction)  # div_final=1e5 in code
        mom = cosine(moms[1], moms[2], fraction)
    return lr, mom

The Dice metric from §4.4, reshaped in the metric's own terms. The paper gives it as Python code, and the current code matches its logic metrics.py.

# Dice coefficient for a binary target  [§4.4]
# Input per batch: scores (B, C, H, W), targets (B, H, W) in {0, 1}
# Output: one scalar, or None if nothing was accumulated

def reset():
    # Start a new epoch with empty sums  [§4.4]
    intersection = 0
    union = 0

def accumulate(scores, targets):
    # Hard predictions from the class axis  [§4.4]
    predictions = argmax(scores, axis=1)
    intersection += sum(predictions * targets)
    union += sum(predictions + targets)

def value():
    # Dice = 2 * intersection / union, computed once at the end  [§4.4]
    if union > 0: return 2 * intersection / union
    return None

Hyperparameters

The paper gives most of its values in code listings, not in tables. Values are copied from those listings unless marked as code defaults. Listing numbers are by section, since the paper does not number listings.

Results

Author-reported. The paper has no benchmark table, no baseline and no variance. Setup: examples on Oxford-IIIT Pets and CamVid §2.1, IMDb §2.2, an Adult sample §2.3, a MovieLens sample §2.4, and MNIST and COCO samples §3.1. Hardware is stated only as "a single GPU" for Pets §2.1 and for the DawnBench entry, where the paper gives no further detail §1.

The authors also say that they reimplemented recent research papers in a short amount of work and got results close to the originals §1. No per-paper numbers are given.

Paper vs code

Code: github.com/fastai/fastai at commit 46231bdc3e88a457c940dd7bde8a03c92fb42901 (default branch, shallow clone, fastai/__init__.py reports version 2.8.13), Apache-2.0. The paper's listings and text describe fastai v2.0.0 §1; this code is much newer, so the comparisons below are with the current code and are not a check of the v2.0.0 release.

• Learning-rate suggestion. Paper: a suggestion of the minimum loss value divided by 10 fig. 9. Code: the same rule exists as minimum (lr_min / 10) schedule.py, but lr_find defaults to valley, which picks a point near the middle of the longest run of decreasing losses schedule.py schedule.py. The first 10% and last 5 points are dropped before suggesting schedule.py.
• Learner arguments. Paper: Learner.__init__ has 11 keyword arguments §5.6. Code: 13 keyword arguments besides dls and model (including wd_bn_bias, train_bn, moms, default_cbs) learner.py.
• Data block method. Paper: databunch §3.1. Code: dataloaders is the method name block.py; databunch is not found in the file.
• Mish. Paper: the Mish activation is optimised with the JIT compiler §4.8. Code: the function mish calls PyTorch's built-in F.mish; the JIT version is the separate MishJit module layers.py.
• Batch norm. Paper: Learner bypasses batch-norm layers when freezing §4.1. Code: frozen batch-norm and bias parameters are flagged to keep training (force_train) by default, and get no weight decay by default learner.py. This matches the paper's intent; the mechanism differs.
• Weight decay. Paper: the decoupled switch is a one-line change §4.3. Code: each optimizer has decouple_wd=True by default, so decoupled weight decay is the default for the Adam family optimizer.py. The paper names the list steppers; the code calls it cbs.
• Dice metric. Paper and code agree on the logic §4.4 metrics.py. The code adds an explicit dim= keyword.
• Adaptive GAN switcher. Paper: switches when losses reach thresholds §4.2.1. Code: if a threshold is None, the switch happens after every batch gan.py.
• Code examples. Several listings use fastai2.* import paths §2.3 §2.4; the current package is fastai. The paper's own text notes the listings were run against v2.0.0 §1.
• Listing errors. The language-model listing ends with metrics=Perplexity()]), which has an extra bracket. The classifier TextDataLoaders.from_df(...) call is missing a comma after path=path §2.2.
• LAMB. The paper says the authors did not define the φ function, and that its code is based on the official TensorFlow implementation §4.3. This card does not check the LAMB code.

No other discrepancies were checked. Specifically not compared: the two-way callback event names and their order, the exact XResNet tweaks, and the 15 DataLoader extension points §4.6.

Pitfalls

• Version drift. The paper describes fastai v2.0.0 §1. Names and defaults differ at the pinned commit (see Paper vs code). Anyone implementing from the paper should pin a version, or use the paper's names only as a guide.
• Unstated defaults. The paper never gives the one-cycle div, div_final, pct_start, or the momentum range. Code defaults are div=25, div_final=1e5, pct_start=0.25, and momentum (0.95, 0.85, 0.95) schedule.py learner.py. The Pets example passes no learning rate, so it uses the Learner default §2.1.
• LR suggestion. The figure caption says "minimum divided by 10" fig. 9. The default suggestion is a different rule (valley), so a plot-based guess and the returned value can disagree schedule.py. Choose the suggestion function explicitly.
• Batch-norm policy is a default, not a rule. train_bn=True is the default. Setting it to False switches to the usual freezing behaviour, which the paper argues against §4.1 learner.py.
• Augmentation defaults are hidden. The Pets listing sets max_warp=0 and leaves rotation, zoom and lighting at code defaults (max_rotate=10, max_zoom=1.1, max_lighting=0.2). The paper does not list these augment.py.
• GAN switcher. With no thresholds, AdaptiveGANSwitcher changes the training target after every batch. The paper's threshold values are not stated §4.2.1 gan.py.
• Two-way callbacks can change anything. The paper's design lets any callback change data, losses, gradients and control flow §4.2. This is powerful, but the paper reports no combined cases of conflicting callbacks beyond "not yet come across any cases where mixing ... has caused any problems" §4.2. Test combinations yourself.
• Mish speed. The paper's JIT claim applies to the MishJit module, not to the mish function layers.py.
• Compute is not reported. The paper gives wall-clock examples for two tasks only ("a couple of minutes" on a single GPU for Pets §2.1, and 18 minutes for ImageNet §1). No memory use, throughput or cost comparisons are given.
• Callback and metric ordering. The paper says reset runs "at the start of training" §4.4. The implementation was not checked for when reset is called, so verify this before relying on it.
• Section numbering. The paper's own numbering comes from the MDPI class, which numbers sections automatically. Section numbers here follow the order of \section and \subsection in the source (for example §4.2.1 is the GAN case study). Check against the rendered PDF when precision matters.
• Figures. Figure numbers are by order of appearance in the source: schedule is fig. 8, the LR finder is fig. 9, LAMB is fig. 10, interpolation is fig. 11 and nbdev is fig. 12.

Lineage

• Builds on: the one-cycle policy §3.1, the learning-rate finder §3.1, ULMFiT-style NLP transfer learning and discriminative learning rates §2.2, and the batch-norm literature §4.1. These are named in the paper; this wiki has no cards for them yet.
• Builds on the frameworks PyTorch §5.1, NumPy §1, pandas, PIL and scikit-learn §5.1 §5.6.
• Compared against: Keras and TensorFlow-style high-level APIs, scikit-learn pipelines, tf.data, and Torchvision and Detectron-style task libraries §3.3 §7.
• Superseded by: not known from wiki content.