Dive into Deep Learning

In plain words

This is a free textbook on deep learning. Many resources either explain the ideas with little code, or show code without saying why it works. This book does both. Each idea is introduced just before it is needed, and it comes with code you can run.

The book is built from Jupyter notebooks. Each chapter adds a few ideas and works through several self-contained examples on real datasets. Each example is often a single notebook, so you can copy it and change it for your own project. The early chapters build the basics from scratch so you see what a framework does for you. Later chapters use the framework's high-level tools.

The book runs from basic maths and data handling, through models for images and text (convolutional networks, recurrent networks, attention and transformers), to training methods, hardware and reinforcement learning. Its text is licensed for reuse with attribution and share-alike.

TL;DR

• A book, not a method paper: it introduces no new algorithm. The authors are Zhang, Lipton, Li and Smola; the arXiv version is v5 (2023-08-22) p. xxv README.md.
• It pairs text, maths and runnable code. Code is in Jupyter notebooks, with framework modules for MXNet, PyTorch, TensorFlow and JAX .
• The chapters start from scratch and then move to high-level APIs; concepts are taught just in time p. xxvii–xxviii.
• Text is CC BY-SA 4.0, so any adapted text stays share-alike. Sample code is under a permissive MIT-0-style grant LICENSE-SAMPLECODE.
• Limits: the code commit is newer than the PDF version, and per-chapter numbers live in the notebooks, which this card does not catalogue.

Problem

The authors say that, at the time, no resource was simultaneously up to date, broad, technically deep, and interleaved with runnable code p. xxvi. They name Goodfellow et al. (2016) as a comprehensive text that does not connect its explanations to code p. xxvi. They also say that many resources were behind paywalls p. xxvi. To close that gap, the book opens with a quick bring-up on the prerequisites (linear algebra, calculus, probability) before the models §2 p. xxviii.

Method

The book's "method" is its teaching design, not an algorithm. The design, as the preface states it:

• One working example per notebook. After the opening notebooks, every chapter adds a moderate set of ideas, each shown in several examples that run on real data. Each example lives in its own notebook, so a reader can start a project by copying one p. xxvii.
• Just-in-time concepts. Concepts are introduced when a practical task needs them. The authors accept using a technique (their example is stochastic gradient descent) before they explain why it works p. xxvii.
• Scratch first, then high-level API. Basic tutorials often show two versions: one built from numerical and automatic-differentiation primitives, and one using the high-level API of a deep learning framework. Later tutorials rely on the high-level API p. xxviii.
• Source format. Equations, citations and code are written together in Jupyter notebooks and rendered to HTML and PDF. The source is shared on GitHub p. xxvii.
• Setup. Python 3.x is required; the authors tested Python 3.9 with Miniconda p. xxxiv.

Structure (printed page numbers from the table of contents p. iii–xxi; Fig. 1 shows the dependencies between chapters fig. 1):

1. Introduction §1, p. 1
2. Preliminaries: data manipulation, linear algebra, calculus, automatic differentiation, probability, documentation §2, p. 30
3. Linear neural networks for regression §3, p. 82
4. Linear neural networks for classification §4, p. 127
5. Multilayer perceptrons, numerical stability and initialization, generalization, dropout, a house-price Kaggle example §5, p. 170
6. Builders' guide: layers and modules, parameter management, initialization, custom layers, file I/O, GPUs §6, p. 212
7. Convolutional neural networks §7, p. 239
8. Modern convolutional networks (AlexNet, VGG, NiN, GoogLeNet, batch normalization, ResNet, DenseNet, RegNet) §8, p. 275
9. Recurrent neural networks, including backpropagation through time §9, p. 334
10. Modern recurrent networks (LSTM, GRU, deep and bidirectional RNNs, encoder–decoder, sequence-to-sequence, beam search) §10, p. 379
11. Attention mechanisms and transformers (attention scoring, multi-head and self-attention, the transformer, vision transformers, large-scale pretraining) §11, p. 422
12. Optimization algorithms: convexity, gradient descent, SGD, minibatch SGD, momentum, Adagrad, RMSProp, Adadelta, Adam, learning-rate scheduling §12, p. 484
13. Computational performance: compilers, asynchrony, hardware, multi-GPU training, parameter servers §13, p. 567
14. Computer vision: image augmentation, fine-tuning, object detection, semantic segmentation, style transfer, CIFAR-10 and dog-breed Kaggle examples §14, p. 614
15. Natural language processing: pretraining (word2vec, GloVe, fastText, BERT) §15, p. 716
16. Natural language processing: applications (sentiment analysis, natural language inference, fine-tuning BERT) §16, p. 771
17. Reinforcement learning, starting with Markov decision processes §17, p. 809

The table of contents read for this card ends at chapter 17. The printed book's later chapters and appendices were not read.

The book names no single algorithm to implement. The one reusable procedure its code defines is the training loop of the base Trainer class. The pseudocode below is derived from that code, not from the paper's text torch.py. Each chapter's notebook implements the standard method it covers.

# Base training loop  [code: d2l/torch.py#L277-L285]
# Input:  model (has training_step), data (train and val loaders)
# Output: model trained for max_epochs epochs

def fit(model, data, max_epochs):
    # Attach loaders and model  [code: d2l/torch.py#L278-L279]
    prepare_data(data)
    prepare_model(model)
    # Create the optimizer from the model  [code: d2l/torch.py#L280]
    optimizer = model.configure_optimizers()
    # One fit_epoch call per epoch  [code: d2l/torch.py#L284-L285]
    for epoch in range(max_epochs):
        fit_epoch(model, optimizer)


# One epoch: training pass, then validation  [code: d2l/torch.py#L294-L311]
def fit_epoch(model, optimizer):
    # Put the model in training mode  [code: d2l/torch.py#L296]
    model.train()
    for batch in train_loader:
        # Forward pass returns the loss  [code: d2l/torch.py#L298]
        loss = model.training_step(prepare_batch(batch))
        # Clear gradients from the previous step  [code: d2l/torch.py#L299]
        optimizer.zero_grad()
        # Backward, optional clipping, update  [code: d2l/torch.py#L300-L304]
        loss.backward()
        if gradient_clip_val > 0:
            clip_gradients(gradient_clip_val, model)
        optimizer.step()
    # Skip validation without a loader  [code: d2l/torch.py#L306-L307]
    if val_loader is None:
        return
    # Put the model in evaluation mode  [code: d2l/torch.py#L308]
    model.eval()
    for batch in val_loader:
        # Validation step without gradients  [code: d2l/torch.py#L309-L311]
        with no_grad():
            model.validation_step(prepare_batch(batch))

Hyperparameters

Book-wide values are not stated. Values are set per notebook. The rows below are the only ones this card verified in code.

Results

Author-reported, and the book is a teaching text rather than a benchmark study. The parts read for this card report no benchmark numbers of their own. Not stated. The table of contents lists Kaggle walkthroughs, including CIFAR-10 image classification and a dog-breed identification example p. xix. Their outcomes were not read.

Paper vs code

• Version mismatch. The PDF is arXiv v5, dated 2023-08-22 [raw/arxiv/2106.11342.json]. The code commit 23d7a5a is dated 2024-03-16 (commit date from the clone's git history). Text and notebooks may differ, and this was not checked.
• Chapter coverage. The repository has chapter folders for chapters 1–17 whose titles match the printed table of contents to . It also has folders for Gaussian processes, hyperparameter optimization, generative adversarial networks and recommender systems . I did not check their book numbering.
• License split. The GitHub API reports the repository license as "Other" (NOASSERTION). The repository's own files separate the two terms. The book text is CC BY-SA 4.0 LICENSE. The sample code is under a modified MIT license LICENSE-SUMMARY. The grant text is MIT-0 style with no attribution clause LICENSE-SAMPLECODE.
• Framework code. The d2l package has separate modules for MXNet, PyTorch, TensorFlow and JAX . The synthetic_data helper is defined in the PyTorch, TensorFlow and MXNet modules torch.py tensorflow.py mxnet.py; the JAX module was not checked for it.

Pitfalls

• Licensing. Text from this book is CC BY-SA 4.0. Any text adapted from it is share-alike, and this card's summary and structure are adapted from it. A human must confirm the card's licence before it is published. This is flagged for review.
• Scope of this read. This card was compiled from the front matter (pp. xxv–xxviii, xxxiv), the table of contents (pp. iii–xxi) and code. Chapter bodies were not read. Any claim about what a chapter teaches beyond its title is not supported by this card.
• The base PyTorch Trainer rejects GPUs. It asserts num_gpus == 0 torch.py. Multi-GPU paths in other modules were not checked.
• Sample code licence differs from text licence. The sample code is MIT-0 style with no attribution clause, while the text is share-alike. Keep the two separate when reusing either.
• Version drift. The PDF (2023) and the code (2024) may disagree. Take numbers from the notebook at code_commit, not from the PDF text alone.

Lineage

• Builds on: not stated.
• Compared against: Goodfellow et al. (2016) and Bishop (2006), named in the preface as textbooks p. xxvi p. xxvii. Neither has a card, so they are named and not linked.
• Superseded by: not stated.