Convolutional Recurrent Neural Networks for Music Classification

In plain words

Music tagging means giving a song a set of labels such as rock, sad or guitar. This paper asks which neural network shape does this best when the networks are given the same memory budget or the same training time. The networks listen to a spectrogram, a picture of how loud each pitch band is over time.

Three of the networks are plain convolutional networks. They slide small windows over the spectrogram and detect patterns in each window, then combine these patterns. The proposed network adds a recurrent part on top. Its convolutional layers still pick out local patterns, but a recurrent network then reads the resulting sequence step by step and keeps a running summary of the whole clip. The idea is that some labels, such as mood, depend on the whole song, while others, such as instruments, depend on short passages.

The paper keeps everything else equal across the networks: same data, same training settings, and same activation and normalisation layers. It varies the number of parameters and the training time. Among the networks tested, the recurrent one gives the best accuracy for a given parameter count, and a fully convolutional network gives similar accuracy with shorter training time.

TL;DR

• Proposes a CRNN for music tagging: 4 two-dimensional conv layers, then a 2-layer GRU that summarises the time axis §2.4.
• Compares it with three CNNs (k1c2, k2c1, k2c2) on Million Song Dataset top-50 tags, at 0.1, 0.25, 0.5, 1.0 and 3.0 million parameters §3 tab. 1.
• Author-reported: at equal parameter count CRNN has the best AUC-ROC of the four §3.1 fig. 2. k2c2 trains faster at every size §3.2.
• The official code repository has no model code at the cited commit, so this card rests on the paper alone (see Paper vs code).

Problem

Music tagging is a multi-label task: one clip can carry many tags, so the output is one sigmoid per tag §2. Earlier CNN taggers differ in kernel shape and in how they reduce the time and frequency axes. The authors argue that CNNs summarise time only by fixed averaging and subsampling, while RNNs can adapt how they summarise §1. They also expect some tags, such as mood, to depend on global structure, and others, such as instruments, on short segments §1. The paper does not state a single failure to fix. Its stated question is which structure gives the best AUC for a given memory budget or training time §1 §3.

Method

Input. A 29-second clip is turned into a log-amplitude mel-spectrogram with 96 mel bands and 1366 frames, treated as a single-channel image §2 §3.

Shared settings. All networks use batch normalization and ELU activations in every convolutional and fully-connected layer, so that only the structure differs §2.

k1c2 (CNN baseline). Four convolutional layers with 1×4 kernels along time, with max-pooling (1×4), (1×5), (1×8), (1×8). The output is flattened and passed through two fully-connected layers §2.1 tab. 1.

k2c1 (CNN baseline). Five convolutional layers. The first has a full-band 96×4 kernel that compresses all frequency bands into one. The rest use 1×4 kernels along time, with pooling of (1×4) or (1×5). The output goes to two fully-connected layers §2.2 tab. 1.

k2c2 (CNN baseline). Five convolutional layers with 3×3 kernels and max-pooling (2×4), (2×4), (2×4), (3×5), (4×4). This reduces the feature map to 1×1, so the network is fully convolutional §2.3.

CRNN (proposed). Four convolutional layers with 3×3 kernels and max-pooling (2×2), (3×3), (4×4), (4×4). The result is a feature map of N×1×15 (feature maps × frequency × time). These 15 time steps feed a two-layer GRU. The last hidden state is connected to the output §2.4. A weak dropout of 0.1 is applied between convolutional layers to limit overfitting of the recurrent layers §2.

# CRNN forward pass for one clip  [§2.4] [tab. 1]
# Input:  mel (96, 1366) log-amplitude spectrogram  [§3]
# Output: 50 tag probabilities, each in (0, 1)  [§2]

def crnn_tags(mel):
    # Treat the spectrogram as a one-channel image  [§2]
    features = add_channel_axis(mel)           # (1, 96, 1366)
    for layer in range(4):
        # 3x3 convolution, widths per layer  [§2.4] [tab. 1]
        features = conv2d(features, widths[layer], kernel=3)
        # Batch norm and ELU in every layer  [§2]
        features = elu(batch_norm(features))
        # Max-pool: (2x2), (3x3), (4x4), (4x4)  [§2.4]
        features = max_pool(features, pool_sizes[layer])
        # Weak dropout between conv layers; exact placement not stated  [§2]
        if layer < 3:
            features = dropout(features, rate=0.1)
    # Read the feature map as a sequence of 15 time steps  [§2.4]
    sequence = to_time_sequence(features)      # (15, N)
    # Two stacked GRU layers; keep the last hidden state  [§2.4] [tab. 1]
    hidden = gru_stack(sequence, layers=2, units=rnn_units)
    # Sigmoid output per tag, since tagging is multi-label  [§2]
    return sigmoid(dense(hidden[-1], num_tags=50))

Scaling. Each structure is sized to a parameter budget of 0.1, 0.25, 0.5, 1.0 or 3.0 million, within 2% tolerance. Depth and kernel shapes stay fixed and only layer widths change. The authors treat 3M as an approximate upper bound on useful complexity given hardware and dataset size §2.5.

Training. Adam, binary cross-entropy, early stopping on validation AUC-ROC, where one epoch is one pass over the training data §3.

# Training with early stopping  [§3]
# Input:  train set (201,680 clips), validation set (12,605 clips)
# Output: model with the best validation AUC-ROC seen so far

def train(model, train_set, valid_set):
    best_auc = -infinity
    while not stopped:
        # One full pass over the training data  [§3]
        for clip, tags in train_set:
            # Binary cross-entropy over the 50 tags  [§3]
            loss = binary_cross_entropy(model(clip), tags)
            # Adam update; learning rate not stated  [§3]
            adam_step(model, loss)
        # Validation AUC-ROC after each pass  [§3]
        auc = mean_auc_roc(model, valid_set)
        if auc > best_auc:
            best_auc = auc
            keep_weights(model)
        else:
            # Stop when AUC stops improving; patience not stated  [§3]
            stopped = True

Hyperparameters

Layer widths. Number of feature maps for conv layers, hidden units for FC and RNN layers tab. 1. Each row is one layer; the columns are the parameter budgets (×10⁶). Layer types follow the table labels.

Source: tab. 1. The layer-type labels in the source table are inconsistent with the text (see Pitfalls).

Results

Author-reported. Setup: Million Song Dataset with last.fm top-50 tags, AUC-ROC on the test set, 29-second clips, models trained at five parameter budgets §3 fig. 2. The paper does not report seeds or repeated runs.

Values read by eye from the unlabelled bars of fig. 2: CRNN about 0.851 at 0.1M and about 0.861 at 3.0M; k2c2 about 0.846 and 0.856; prior state of the art 0.851 (stated in its caption). Treat these as approximate. The text does not give AUC values per budget.

Paper vs code

The paper links an official repository, https://github.com/keunwoochoi/icassp_2017 §3, whose owner matches the first author. At code_commit it contains a single README.md and no model code. The README says the repository will contain the data split, model code and pretrained weights README.md. No architecture, hyperparameter or training code can be compared, so this card's hyperparameters come from the paper alone. The README also names two other repositories (a split repository and an earlier tagger). They are not part of this card. The repository has no LICENSE file, so no code may be quoted from it.

Pitfalls

• No model code. The official repository lacks code and weights at the commit recorded here. Every architectural detail (for example, the output head, the exact dropout placement, and the optimiser settings) is therefore unverified by code, and an implementation must make its own choices where the paper is silent.
• CRNN convolution depth is inconsistent. Section 2.4 and tab. 1 show four convolutional layers. The computation section says "5 convolutional layers" (15 time steps in the RNN, depth up to 20) §3.2. Implement four, as the architecture section and the table agree.
• Time steps depend on rounding in pooling. The paper states 15 time steps §2.4. Applying the stated pool sizes to 1366 frames gives 14 steps with floor rounding and 15 with ceiling rounding (derived, not stated in the paper). Use same-padding or ceiling-mode pooling to match.
• Output head is not in the table. The CRNN text connects the last hidden state to the output §2.4, but tab. 1 lists no dense layer. The k2c2 text reduces the map to 1×1 §2.3, but how that map becomes 50 tag outputs is not described. Both are assumed to end in a 50-unit sigmoid layer; this is an assumption, not a stated fact.
• Layer-type labels in tab. 1 conflict with the text. The table labels k1c2 and k2c2 as conv2d and k2c1 as conv1d. The text describes k1c2 with 1×4 kernels and k2c1 with a 96×4 first layer. Follow the text for kernel shapes.
• Wrong figure reference. The k2c2 description cites the k2c1 panel of fig. 1 ("dia_b"). The k2c2 panel is dia_c.
• Input size does not match the stated clip. 29 s at 12 kHz with hop 256 gives about 1359 frames (derived). The stated input is 1366 frames. The padding or framing used is not stated, so reproduce the 1366 shape only if the framing is known.
• Split counts do not add up. 201,680 + 12,605 = 214,285, but the text gives 214,284 for training plus validation §3. The test count of 25,940 is given separately.
• Commented-out results. The LaTeX source has an AUC and a time row in tab. 1 commented out, with different values from the figures. They are not in the published paper and are not used here.
• Small gaps, no variance. The CRNN margin over k2c2 in fig. 2 looks like roughly 0.004 to 0.005 AUC, eyeballed from the chart. No seeds, repeated runs or confidence intervals are reported §3. Treat the ranking as indicative.
• Early stopping is loosely specified. The paper says training stops when validation AUC does not improve over one full pass §3. Patience and the maximum number of epochs are not stated.
• Hardware and timing. The hardware is not specified, so the training-time comparison §3.2 is only meaningful relative to the other networks in the same run.
• Accuracy vs memory vs time. The paper's conclusion depends on the budget. k2c2 is faster to train at every size, while CRNN has higher AUC for the same parameter count. Choose by the binding constraint, as the paper does §4.

Lineage

• Builds on: the CRNN structure was first proposed for document classification by Tang et al. (2015), and later applied to images by Zuo et al. (2015) and to music transcription by Sigtia et al. (2016), as cited in §1. Their arXiv ids are not known to this wiki.
• Builds on: the k1c2 and k2c1 layer widths follow Dieleman and Schrauwen (2014), and k2c2 follows Choi et al. (2016, ISMIR), which the paper uses as its state-of-the-art baseline §2.5 §3.1.
• Uses batch normalization (arXiv:1502.03167), ELU (arXiv:1511.07289) and GRU (arXiv:1409.1259) as components §2 [bibliography].
• Compared against: k1c2, k2c1 and k2c2, all defined in this paper §2.
• Superseded by: not known from content already in this wiki.