A Taxonomy of Deep Convolutional Neural Nets for Computer Vision

In plain words

Before deep learning, computer vision systems relied on people designing the image features by hand, such as edge and texture detectors, and then feeding them to a classifier. Convolutional neural networks learn those features from the pictures themselves. A small filter slides across the image, and the same filter is reused at every position, so the network learns "what a useful pattern looks like" once and finds it everywhere.

This paper is a guide to that family of networks. It starts from AlexNet, the network that won a large image classification contest in 2012, and explains the pieces it is built from: filters, pooling, activation functions, dropout and gradient-based training. It then walks through the common variations that people made to adapt these networks to other jobs: finding and outlining objects, labelling every pixel, reading video, describing images in words, answering questions about images, and comparing two images for similarity.

It is written for newcomers who want to choose a starting design for a vision problem. It does not train or test any network itself, so the numbers it quotes come from the papers it cites.

TL;DR

• A survey, not a method: it describes the building blocks of convolutional networks §2.1–2.4, the AlexNet design §2.5, and how the design was adapted §3.
• Gives a recipe-style overview; no experiments and no results table of its own.
• Use it as a map of the field as of its 2016 arXiv version. Values that an implementer needs (learning rates, dropout rate, layer sizes beyond those shown) are often "not stated" here; check the cited originals.

Problem

Fully connected layers on raw images need very large weight matrices, and they ignore the 2-D layout of pixels. For a 200×200 image with 40,000 inputs and a 20,000-unit hidden layer, the first layer alone has 40,000 × 20,000 = 800 million weights, and the paper argues that learning small local filters is far more tractable §2.1.1. Earlier deep models (restricted Boltzmann machines, deep belief networks, stacked autoencoders) relied on unsupervised pre-training; the survey states that supervised end-to-end training of a CNN was enough for ImageNet-scale image classification §1.

Method

The survey has no single method. Its core content is the following set of building blocks, each with the rule the paper gives for it.

Convolutional layer. Each filter has small spatial size and spans all input channels (for RGB, 11×11×3 in the first AlexNet layer, with 96 such filters) §2.1 fig. 1. Two constraints separate a convolution from a fully connected layer: local connectivity and weight sharing §2.1.1. Practical networks use a stride larger than one, which the paper describes as equivalent to down-sampling after a regular convolution §2.1.1.

Pooling. A max filter over each n×n region keeps the largest value. The paper gives two effects: a small amount of spatial invariance, and a reduction of the activation size by a factor of n² for the next layer §2.1.2.

Non-linearity. A non-linearity follows each layer. The survey lists sigmoid and tanh (the older choice) and ReLU, max(0, x), (the modern choice) §2.1.3. It attributes faster training with ReLU to a cited study §2.1.3. Leaky ReLU is max(0, x) + α·min(0, x) with a fixed α, and a later work learns α §2.1.3.

Key equations (unnumbered in the source; cited by section):

\text{ReLU}(x) = \max(0, x) \qquad [\text{§2.1.3}]

\text{LeakyReLU}(x) = \max(0, x) + \alpha \min(0, x) \qquad [\text{§2.1.3}]

Losses. Classification uses softmax or sigmoid cross-entropy; regression uses Euclidean error. In AlexNet the last layer (FC8) has one output per class of the 1000 ILSVRC classes §2.3 fig. 1.

Optimization. Backpropagation computes gradients §2.3.1. Plain gradient descent is impractical for large datasets, so the survey uses stochastic gradient descent with mini-batches "typically ... (∼100)" data points §2.3.1. Momentum is described as largely used in practice and often considered essential; Adagrad, Nesterov, Adadelta and Adam are named as alternatives §2.3.1. The survey does not write out an update rule.

Dropout. During training each neuron is dropped with probability p, so each SGD step trains a random subset of neurons. At test time all neurons are used and each activation is multiplied by p §2.3.2. See Pitfalls for the conflict in this description.

Pre-trained features. Use the activations of a trained network as features for a new task; the survey names the FC7 activation as a generic feature descriptor §2.6.2. For fine-tuning, initialise from the trained weights, use a learning rate much lower than the original, and re-learn fewer layers when the new task is more similar to the original and when less data is available §2.6.1.

AlexNet, as described. Five convolutional layers followed by three fully connected layers §2.1 fig. 1. Max-pooling follows layers 1, 2 and 5; dropout is applied to the last two fully connected layers; layers 1 and 2 use local response normalisation §2.5. Hyperparameters were hand-tuned §2.5.

Region-based detection (R-CNN). Generate about 2000 class-agnostic region proposals with selective search §3.1; warp each proposal to a fixed size; run a CNN on it; classify each region with class-specific linear SVMs §3.1 fig. 2.

Fully convolutional segmentation. Replace the fully connected layers with convolutions so the network accepts any input size §3.2. The survey describes upsampling by learned deconvolution layers at several scales §3.2, and a second approach that removes strides and feeds the output into a CRF §3.2. fig. 3

Two-stream video. One CNN takes RGB frames; another takes stacked optical flow; their predictions are fused §3.3 fig. 4.

CNN + RNN. An RNN has at least one feedback connection and keeps an internal memory §3.4. The survey gives a toy example: a single-hidden-layer RNN that outputs the running sum of a positive sequence. With unit weights and ReLU, the update is h(t+1) = ReLU(a(t) + h(t)) and S(t+1) = ReLU(h(t+1)) §3.4 fig. 5. LRCN combines AlexNet-style per-frame CNN features (RGB and optical flow) with a single LSTM layer of 256 hidden units and averages the frame predictions §3.4.1 fig. 6.

# Toy running-sum RNN with unit weights and ReLU  [§3.4] [fig. 5]
# Input: positive sequence a_1 ... a_T
# Output: S_T, the sum of the sequence so far (as the survey states it)

def running_sum_rnn(inputs):
    # Initial hidden state is not stated; zero is assumed here
    hidden = 0
    for a in inputs:
        # Update the hidden state from the new input and the old state  [§3.4]
        hidden = relu(a + hidden)
        # Read the output from the hidden state  [§3.4]
        output = relu(hidden)
    return output

# Forward pass of one convolution block  [§2.1] [§2.1.2] [§2.1.3]
# Input:  image (3, H, W), RGB
# Output: pooled feature maps

def conv_block(image, filters, stride, n):
    # Slide each learned filter over the image with the given stride  [§2.1.1]
    maps = convolve(image, filters, stride)
    # Non-linearity after the layer; modern choice is ReLU  [§2.1.3]
    maps = relu(maps)
    # Keep the largest value in each n x n region  [§2.1.2]
    pooled = max_pool(maps, n)
    return pooled

Hyperparameters

Only values stated in the survey are listed. The survey gives no full training recipe for any network.

Results

No experiments of its own; the paper is a survey. The quantitative claims below are the survey's own statements, reported as the cited works found them. Setups are taken from the survey's text.

Caveat: the 1–2% and 3% figures are the survey's summary of other papers. This card does not re-check them.

Paper vs code

No official code. The survey has no method of its own. The arXiv source links no code repository: its footnote points to the journal article, and the authors' lab page (listed in the source) could not be reached. A GitHub repository search for "taxonomy deep convolutional computer vision" returned no results.

Pitfalls

• Layer count of AlexNet. Section 2.2 says "Modern deep networks such as AlexNet have 7 layers" §2.2. The same paper describes five convolutional and three fully connected layers, eight in total §2.1 fig. 1. Count the layers from the architecture, not from this sentence.
• Dropout probability. The text says neurons are dropped with probability p during training, and that at test time each activation is multiplied by p §2.3.2. If p is the drop probability, the test-time scaling would be the keep probability, not p. The two statements point to different conventions. The survey does not resolve this; the cited Hinton et al. (2012) source is the place to check. Do not implement from this sentence without checking.
• Toy RNN indices. The survey defines S_t as the sum a_1 ... a_t §3.4, but its recurrence gives S(t+1) from h(t+1) = ReLU(a(t) + h(t)) §3.4. The output is therefore indexed one step ahead of the definition. The initial state h(0) is not stated; the card's pseudocode assumes zero.
• Unit weights are asserted. The survey says the weights "would result in" W_ih = W_hh = W_ho = 1 for the toy problem §3.4. It does not derive this, and it is a hand-set example, not a learned result.
• Max-pooling size reduction. The factor n² assumes non-overlapping windows (stride equal to the window size) §2.1.2. The survey does not state the stride for its max-pooling layers §2.5.
• Unsourced claims. The survey asserts, without a citation, that SGD generalises better than full-batch gradient descent §2.3.1. Treat it as the survey's opinion.
• Missing AlexNet recipe. Learning rate, momentum, dropout rate and weight decay were "hand tuned" and their values are not given §2.5. An implementer must take them from the original paper.
• Costs stated for 2016 hardware. The one-month training time on two GPUs and the "little under a week" figure are the survey's statements §2.5. They are not current benchmarks.
• Coverage ends with the 2016 arXiv version. Later architectures and training methods are not covered. This card does not add them.

Lineage

• Builds on: LeCun et al.'s early CNN for handwritten digits, Neocognitron, the deep belief network and autoencoder pre-training work, and AlexNet (Krizhevsky et al., 2012) §1 §2. None of these has a card in this wiki.
• Surveyed architectures without cards: VGG, GoogLeNet, Overfeat, R-CNN, fully convolutional networks, two-stream networks, LRCN, siamese networks, and triplet networks §2 §3.
• Compared against: none in the survey's own experiments.
• Superseded by: not stated in this card. Post-2016 designs are outside its scope.