Convolutional recurrent network (CRNN)

Overview

A convolutional recurrent network (CRNN) usually has two stages. Convolutional layers extract local patterns from a two-dimensional input. A recurrent layer then reads a sequence built from those features and keeps a running summary. In that form, local feature extraction comes first and sequence modelling second 1609.04243 §2.4. The PredRNN variant does not follow this order: its convolutional recurrent cells sit in every layer, with no separate front end 2103.09504 §4.1.

The papers here differ on what the sequence is:

• Time frames of a spectrogram, for multi-label music tagging 1609.04243 §2.4.
• One observation per date in a satellite image series, with the recurrent cells themselves convolutional 1802.02080 §3.1.
• Columns or rows of a feature map, after the height has been pooled to one, for reading text at any orientation 1711.04226 §4.1.
• Columns of a horizontal word image, read by an attention decoder, with an extra loss that trains the attention to land on each character 1709.02054 §3.2.
• Unrolled steps of one layer on a single image. This is recurrence over depth, not over a sequence of inputs, despite the shared name 1811.04241 §3.
• Frames over time, with convolutional gates in every layer of a stacked predictor. Memory flows up through the layers and forward in time 2103.09504 §4.1.

Evidence is limited. Each variant comes from one author-reported study, and the music study reports no seeds or repeated runs 1609.04243 §3. The focusing study chose its loss weight on the same benchmarks it reports, with no separate validation split 1709.02054 §4.4. The PredRNN paper has the broadest evaluation here, on five datasets, but it reports no seeds or variance for any table 2103.09504 §5. The official code for the music study has no model code at the recorded commit, so its implementation details are the paper's alone README.md.

Variants

CRNN for music tagging (time axis)

Four 3×3 convolutional layers, each followed by batch normalization and an ELU activation. Max-pooling uses (2×2), (3×3), (4×4) and (4×4) windows. Together these reduce the 96 frequency bands to one, which leaves about 15 time steps. Those steps feed a two-layer GRU, and its last hidden state feeds the output. Dropout of 0.1 sits between the convolutional layers 1609.04243 §2 1609.04243 §2.4 1609.04243 tab. 1. Card: Convolutional Recurrent Neural Networks for Music Classification.

Baseline: k1c2 (1×4 kernels along time)

Four convolutional layers with 1×4 kernels that run along time only. Pooling is (1×4), (1×5), (1×8) and (1×8). Two fully connected layers follow 1609.04243 §2.1.

Baseline: k2c1 (full-band first layer)

Five convolutional layers. The first uses a 96×4 kernel that spans every frequency band, so it collapses the frequency axis in one step. The later layers use 1×4 kernels along time. Two fully connected layers follow 1609.04243 §2.2.

Baseline: k2c2 (fully convolutional)

Five convolutional layers with 3×3 kernels. Pooling shrinks the map to 1×1, so the network has no recurrent or fully connected stage 1609.04243 §2.3. The paper does not describe how the 1×1 map becomes 50 outputs 1609.04243 tab. 1. All four networks are sized to the same parameter budgets, as described under Results.

Bidirectional ConvGRU or ConvLSTM encoder over a time series (time axis)

Each pixel's observation is a stack of spectral bands plus two constant maps for year and day of year 1802.02080 §4. The recurrent cells are convolutional, so the state at each step is a map with spatial size and a fixed number of channels, not a vector 1802.02080 §3.1. The same cell runs forward and backward over the observations with shared weights and zero initial states. The two final states are concatenated and a 3×3 convolution maps them to the class channels, followed by a softmax 1802.02080 §3.3. Only the final states are read out, not one output per step.

During training, 30 observations are drawn at random from all dates while their order is kept 1802.02080 §5.2. No cloud mask is applied; observations with cloud cover below 80% are kept 1802.02080 §4. Card: Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders.

Directional sequences with attention decoding (image-width axis)

A basal CNN produces a square feature map 1711.04226 §3.1. The horizontal branch pools the height to one with shared convolution blocks and runs a bidirectional LSTM along the width. The vertical branch rotates the map by 90 degrees and does the same. The two branches share their convolution filters, which the authors report as necessary for training 1711.04226 §4.1. A placement network gives four weights per position that sum to one, and a filter gate fuses the four sequences with them 1711.04226 §4.1 1711.04226 §4.2. An attention LSTM decodes one character per step 1711.04226 §3.3. Training uses word labels only 1711.04226 §3.4. Card: AON: Towards Arbitrarily-Oriented Text Recognition.

Column sequence with a focusing loss on attention (image-width axis)

A ResNet-style encoder maps a 256 × 32 word image to 65 column features, and a bidirectional LSTM runs over those columns 1709.02054 §4.2 tab. 1. An attention decoder reads the features one character at a time 1709.02054 §3.1. A second, small network, the focusing network, takes the attention centre for each character, crops a patch of feature maps around it and predicts a label for every patch position 1709.02054 §3.2 eq. 9 eq. 10. Only images with character pixel labels carry this loss 1709.02054 §3.2 eq. 13. The two losses are mixed with a weight λ, and the authors use 0.01 1709.02054 §3.3 §4.2. Patch size is not given as a number, and the focusing head's shapes are not stated 1709.02054 §3.2. Card: Focusing Attention: Towards Accurate Text Recognition in Natural Images.

Recurrent convolutional layers inside inception-style blocks (unrolled steps)

Each recurrent convolutional layer sums a feed-forward convolution of the current input with a convolution of the previous step's output, adds a bias and applies ReLU. The layer is unrolled for a fixed number of steps; the experiments use two recurrent steps after one feed-forward convolution 1811.04241 §3 1811.04241 §4. Branches with 1×1 and 3×3 kernels and a pooled branch are concatenated, and the block input is added back 1811.04241 §3. Card: Breast Cancer Classification from Histopathological Images with Inception Recurrent Residual Convolutional Neural Network.

Stacked spatiotemporal memory (time and depth)

Each unit is a spatiotemporal LSTM (ST-LSTM) with two memories. The temporal memory C runs along time within its layer, as in an LSTM. The spatiotemporal memory M runs up from the layer below. At the bottom layer it comes from the top layer at the previous step 2103.09504 §4.1 eq. 3. The gates are convolutions, and the hidden state is a 1×1 convolution over the two memories, gated by an output gate 2103.09504 §4.2 eq. 4.

Training adds a memory decoupling penalty: the absolute cosine similarity between the increments of C and M. The penalty is used only in training and adds no parameters at inference 2103.09504 §4.2 eq. 5. Reverse scheduled sampling feeds the encoder the true frame with a probability that rises over training 2103.09504 §4.4 eq. 9. For action-conditioned prediction, the previous hidden state is multiplied by a convolution of the action before it enters the cell 2103.09504 §4.3 eq. 7. Card: PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning.

When to use which

The papers compare networks at matched parameter counts or against published baselines, and they measured little about latency or memory. The choice depends on which constraint binds.

The CRNN's margin over k2c2 is small. It is read by eye from a chart, a few thousandths of AUC at most 1609.04243 §3.1 fig. 2. With no variance reported, treat the ranking as indicative, not settled.

Implementation sketch

The steps below are the music-tagging CRNN. The paper does not state the optimizer's learning rate, batch size or early-stopping patience, so an implementation must choose them.

# Forward pass for one clip  [§2.4] [§3]
# Input:  log-mel spectrogram, 96 bands by about 1366 frames
# Output: one probability per tag, 50 tags

def crnn_tags(spectrogram):
    # Treat the spectrogram as a one-channel image  [§2]
    x = add_channel_axis(spectrogram)              # (1, 96, 1366)
    # Four conv blocks: 3x3 conv, batch norm, ELU, then pooling  [§2.4]
    for width, pool in zip(widths, [(2, 2), (3, 3), (4, 4), (4, 4)]):
        x = elu(batch_norm(conv2d(x, width, kernel=3)))
        x = max_pool(x, pool)
        # Dropout 0.1 between conv layers; exact placement not stated  [§2]
        x = dropout(x, rate=0.1)
    # Frequency is now 1, so read the time axis as a sequence  [§2.4]
    steps = time_steps(x)                          # about 15 steps
    # Two stacked GRU layers; keep the last hidden state  [§2.4] [tab. 1]
    h = gru(gru(steps, units=rnn_units_1), units=rnn_units_2)[-1]
    # One sigmoid per tag, because tags are not exclusive  [§2]
    return sigmoid(dense(h, num_outputs=50))       # output head assumed

Two notes for the implementation:

• The stated pooling gives 15 time steps only with ceiling rounding or same-padding. Applied to 1366 frames with floor rounding, the same pools give 14 steps. This is our arithmetic, not a figure in the paper 1609.04243 §2.4.
• The paper does not state the final dense layer's shape, the dropout placement relative to batch norm and ELU, or the GRU widths beyond tab. 1 1609.04243 §2.4 1609.04243 tab. 1.

The other variants keep the convolutional front end and change the recurrent loop. The time-axis encoder runs the loop over observations in forward and reverse order and reads out the two final states. The directional variant runs it over columns and rows and attends over the positions. The unrolled variant runs it over steps of one convolution on a single image. The PredRNN variant has no front end. Each layer's convolutional cell reads the layer below and the memory from the layer below, so the loop runs over layers within each step and then over steps 2103.09504 §4.1 eq. 3. Their loop bodies are given on the cards cited above.

Results

All results are author-reported.

Music tagging

Million Song Dataset with top-50 last.fm tags. Training uses 201,680 clips, validation 12,605 and test 25,940. Clips are 29 seconds 1609.04243 §3.

The paper gives no AUC values per budget in the text. The popularity of a tag shows no correlation with its AUC rank (Spearman ρ 0.077) 1609.04243 §3.3.

Land cover from a satellite time series

Sentinel-2 top-of-atmosphere reflectance over one study area, 17 crop classes, a GRU encoder with 256 cells trained for 60 epochs on both 2016 and 2017 1802.02080 §4 1802.02080 §5.2. Overall accuracy is 89.7% for 2016 and 89.5% for 2017, and overall kappa is 0.870 for both years. Class recall runs from 41.5% (peas, 2017) to 96.8% (maize, 2017) 1802.02080 tab. 2. The released code does not compute the kappa values in the table 1802.02080 §7. The claim that the network learned to ignore clouds rests on visual inspection of a few memory cells, not on a measured test 1802.02080 §5.1.

Text reading at arbitrary orientation

Lexicon-free accuracy: 76.8% on CUTE80 and 68.2% on ICDAR15, the highest of the compared methods in each column. Yang et al. is higher on SVT-Perspective with no lexicon, and the authors' STN baseline is higher with the 50-word lexicon 1711.04226 tab. 1 1711.04226 §5.3. On regular benchmarks the authors describe the results as comparable, and they trail Cheng et al. (FAN) on the unconstrained columns 1711.04226 §5.4 1711.04226 tab. 2.

Scene text with a focusing loss

Word accuracy without a lexicon, on the same model with and without the focusing network. The 0% row is the baseline without it 1709.02054 tab. 2 tab. 8 tab. 9.

The main setting trains on 4 million synthetic words, of which about 30% carry character pixel labels 1709.02054 §4.2. The paper's own table lists Baidu IDL higher on IC15 (77.0) and Jaderberg et al. 2016 higher on IC03 Full (98.6 against 97.3) 1709.02054 tab. 2. The focusing loss lowers total normalized edit distance in every column of the table 1709.02054 tab. 3. The numbers above are not comparable with the AON results, which come from a different paper and setup.

Breast histopathology

BreakHis, 70/30 patient-disjoint splits, five trials averaged 1811.04241 §4. Binary accuracy at 40× is 97.95% ± 1.07 at image level with augmentation 1811.04241 tab. 4. Eight-class accuracy at 40× is 97.09% ± 1.06 at image level with augmentation 1811.04241 tab. 3. On BCC 2015 the image-wise binary accuracy is 99.05% with resized, augmented samples 1811.04241 tab. 7. That test set is about 20 images 1811.04241 §4, so one image is about 5 percentage points of accuracy (our arithmetic). The trial count conflicts within the paper, three in one section and five in another 1811.04241 §4.

Competition entries

The ICDAR 2019 scanned-receipt report publishes ranked scores and no architectures. Two entries in its Task 2 OCR ranking have team names that mention CRNN, one with CTC after a U-Net stage and one after U-Net segmentation. They score 87.93 and 86.12 Hmean 2103.10213 fig. 4. The report describes no architecture for either, so the names are the only CRNN evidence it gives.

Video prediction (stacked ST-LSTM)

Moving MNIST, fixed 10,000-sequence training set, MSE: PredRNN-V2 reaches 48.4 against 103.3 for ConvLSTM and 52.0 for MIM 2103.09504 tab. 2. Removing reverse scheduled sampling and the decoupling loss gives 56.8. Truncating the bottom-up or top-down memory path gives 57.6 or 59.7 2103.09504 tab. 4. Adding the two techniques lowers MSE in every backbone row of the paper's table: PredRNN goes from 56.8 to 48.4, and ConvLSTM from 103.3 to 64.2 with reverse scheduled sampling alone 2103.09504 tab. 5.

On KTH, averaged over 20 future steps, PredRNN-V2 reaches PSNR 28.37 and SSIM 0.838, against 23.58 and 0.712 for ConvLSTM 2103.09504 tab. 7. On radar echo maps, PredRNN-V2 has the best CSI at 30, 40 and 50 dBZ 2103.09504 tab. 8. On BAIR with actions, its SSIM is 0.861, against 0.807 for ConvLSTM. The SVG and SV2P baselines were scored on the best of 100 samples, and the paper does not say whether PredRNN-V2 was scored the same way 2103.09504 tab. 10 2103.09504 §5.6.

The Moving MNIST MSE is a per-frame sum of squared errors over pixels, as the repository computes it, not a per-pixel mean. Do not compare it with per-pixel MSE from another paper trainer.py. The radar and Traffic4Cast results rely on pipelines and backbones that are not in the repository datasets_factory.py.

Contradictions

• Convolutional depth of the music CRNN. The architecture section and tab. 1 show four convolutional layers 1609.04243 §2.4 1609.04243 tab. 1. The training-cost section refers to five 1609.04243 §3.2. The source does not resolve this.
• Layer-type labels in the music study. tab. 1 labels k1c2 and k2c2 as 2-D convolutions and k2c1 as 1-D. The text describes k2c1 with a 96×4 first kernel, which is 2-D 1609.04243 §2.2 1609.04243 tab. 1.
• Abstract and tables in the text-reading paper. The paper claims state-of-the-art on irregular data 1711.04226 §5.3, while its table shows two columns where other methods are higher 1711.04226 tab. 1.

Papers

• Convolutional Recurrent Neural Networks for Music Classification: proposes the CRNN for music tagging and compares it with three CNN baselines at matched parameter counts and training time.
• Focusing Attention: Towards Accurate Text Recognition in Natural Images: a ResNet-style encoder with a BiLSTM over columns, an attention decoder and a focusing loss that uses character pixel labels when they exist.
• AON: Towards Arbitrarily-Oriented Text Recognition: a convolutional front end with directional BiLSTM sequences fused by learned placement weights and an attention decoder.
• Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders: a bidirectional convolutional GRU or LSTM encoder over Sentinel-2 series for per-pixel crop classification.
• Breast Cancer Classification from Histopathological Images with Inception Recurrent Residual Convolutional Neural Network: recurrent convolutional layers unrolled inside inception-style residual blocks for histopathology classification.
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning: a stacked ST-LSTM whose memory flows up the layers and forward in time, with a memory decoupling loss and reverse scheduled sampling, evaluated on video prediction.
• ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction: a competition report with no method of its own; it names two CRNN-based OCR entries in its rankings.