Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

In plain words

Residual networks are very deep stacks of layers, where each layer adds a correction to its input. The paper notices that if you reuse one layer's weights at every depth, the stack becomes the same as a small recurrent network that repeats one update many times. So a deep residual net and a shallow recurrent net can compute the same thing, and the recurrent version needs far fewer parameters while doing about as well.

The authors then build a more general recurrent model with several stages that feed each other and themselves, loosely modelled on the visual pathway in the brain. They test it on CIFAR-10 and ImageNet. Their main point is that the success of very deep networks may come from their ability to repeat computations, not just from having many layers.

TL;DR

A residual network whose layers share weights is the same as a shallow RNN with identity shortcuts §2. A direct RNN implementation performs similarly to the ResNet with far fewer parameters, and the paper proposes multi-state fully recurrent networks (FRNNs) with time-specific batch norm §1, §3. On CIFAR-10 a 3-state FRNN reaches 6.86% error at readout time 10 under a 60-epoch schedule, above the 6.61% of a ResNet-110 trained for 200 epochs in the same table tab. 1. Results are author-reported.

Problem

Very deep residual networks reach strong accuracy, but the paper asks why depth helps and whether it is needed. Its premise is that feedforward depth can be replaced by repeated computation over time. It also notes that cortical areas show few feedforward stages, so unrolled recurrence is a candidate explanation §1. The paper gives no compute or latency benchmarks for the models.

Method

Equivalence (§2). A ResNet block updates a state as h_t = K(h_{t-1}) + h_{t-1}, where K is a nonlinear operator and the identity term is the shortcut eq. 1 §2.1. Unrolling this over time with the same weights at each step gives a residual network with shared weights fig. 1 §2.1. In the homogeneous, time-invariant case the output after n steps is (K+I)^n applied to the input eq. 4 §2.2. Ordinary ResNets with unshared weights are the time-variant case §2.2.

Generalized model (§3). A model is a directed graph of stages (states), for example V1, V2, V4 and IT, with transition functions on its edges, and it may contain cycles §3, "Multi-state Graph" fig. 2A. A pre-net feeds the first state and a post-net reads the readout state; all parts train end to end with backpropagation §3, "Pre-net and Post-net". A transition matrix (i, j, t) gives the function from state i to state j at time t, so weights may differ over time and edges may be blocked for some steps §3, "Transition Matrix". Incoming transitions to one state are summed in ResNets and averaged in FRNNs; the paper reports averaging gives about 1% better results for FRNNs [§3, "Transition Matrix"].

Time and readout (§3). Each transition is treated as taking one time unit. At t=0 only the first state holds data, and a transition starts once its input state is filled. The readout time t is when the post-net reads the last state, and the paper proposes that wall-clock time is about 20t to 50t ms §3, "Notations".

Time-specific batch norm (§3). Normalization statistics are computed separately for each time step t, using the training set. Shared statistics across time were found to hurt, and time-specific statistics restore performance; the paper also says the model then trains with ReLUs §1 §3, "Batch Normalizations for RNNs".

Weight sharing (§3). Tied weights start from the same values, and each tied weight receives the sum of the gradients of all its copies [§3, "Weight sharing"].

Transition functions (fig. 2). The transitions are BN-ReLU-Conv pipelines (BRCx2 or BRCx3) or BN-ReLU-Deconv for upsampling, with an identity shortcut (+I) fig. 2C. Between nearby states the spatial size changes by 2x, using stride-2 convolution or 2x deconvolution. Intermediate feature counts are the average of the input and output counts fig. 2 caption.

# Shared-weight residual unroll  [§2.1] [eq. 1] [fig. 1]
# Input:  x0 (B, C, H, W) from the pre-net
# Output: h_T (B, C, H, W)

def unrolled_resnet(x0, K, T):
    # h_0 is the input; K uses one set of weights at every step  [§2.2]
    h = x0
    for t in range(T):
        # h_t = K(h_{t-1}) + h_{t-1}: residual branch plus identity  [eq. 1]
        h = K(h) + h
    return h

# Fully recurrent multi-state forward  [§3] [fig. 2]
# Input:  image (B, 3, 32, 32); transitions[(i, j)] = module
# Output: post-net applied to the last state
# Simplified: the paper gives no pseudocode for this loop.

def frnn_forward(image, pre_net, transitions, readout_time):
    # Pre-net output fills state 0 at t = 0  [§3, "Pre-net and Post-net"]
    states = {0: pre_net(image)}
    for t in range(readout_time):
        # A transition runs once its input state is filled  [§3, "Notations"]
        updates = {}
        for (i, j), transition in transitions.items():
            if i in states:
                # t is passed so the batch norm can be time-specific  [§3]
                updates.setdefault(j, []).append(transition(states[i], t))
        # FRNN averages incoming transitions; a ResNet sums them  [§3]
        for j, outputs in updates.items():
            states[j] = mean(outputs)              # (B, C_j, H_j, W_j)
    return post_net(states[last_state])

# Time-specific batch norm  [§3, "Batch Normalizations for RNNs"]
# Input:  activations (T, B, C, H, W) for one state; training-set statistics
# Output: normalized activations, same shape

def time_specific_bn(activations, mean_t, std_t):
    # Statistics are per time step t, not shared across time  [§3]
    for t in range(T):
        activations[t] = (activations[t] - mean_t[t]) / std_t[t]
    return activations

# Gradient of a tied weight  [§3, "Weight sharing"]
# Input:  gradient of the loss for each copy of one tied weight
# Output: gradient used for the update

def shared_weight_gradient(copy_gradients):
    # The update uses the SUM over all copies, not the mean  [§3]
    return sum(copy_gradients)

Training details §5 are summarized under Hyperparameters.

Hyperparameters

Results

Author-reported. CIFAR-10 setup: 60 epochs, batch 64, translation and mirror augmentation tab. 1 §5. ImageNet setup: 4-state ResNet, 12 epochs fig. 6. Hardware and wall-clock times are not stated.

The table does not say whether its errors are validation or test errors. The paper says the shared-weight 2-state model is better on CIFAR-10 than the non-shared one fig. 4A. It also says weight sharing in the first state is often beneficial, even on ImageNet, while sharing in later stages sometimes lowers performance fig. 4 caption §6. The parameter counts are shown in fig. 4 but are not given in the text.

Paper vs code

The abstract links the code at gitlab.com/liaoq/biological_rnn_resnet (GitLab owner "Q Liao", the first author). Its code_commit is the latest one seen through the GitLab API (2021-10-06, "Update README.md"). I did not clone it: GitLab is not cloneable here, so I read it through WebFetch and the GitLab API.

What it contains: two ImageNet checkpoints, my_resnet/net-epoch-12.mat and my_resnet_shared/net-epoch-12.mat, pretrained models for a ResNet and a ResNet with weights shared across layers net-epoch-12.mat, plus MatConvNet DagNN helper functions in +dagnn/ and @DagNN/. The README describes the models as ImageNet pretrained and cites this paper.

What it does not contain: no CIFAR-10 training script or config, no FRNN code, no multi-state graph definition, no time-specific batch norm code, and no LICENSE file.

Discrepancies: none can be checked. The ImageNet checkpoint is named epoch 12, which matches the 12 epochs in fig. 6. I could not read the .mat contents, so the architecture, learning rate schedule and batch size cannot be confirmed against the paper. The CIFAR-10 results in Table 1 have no released code or weights.

Pitfalls

• CIFAR-10 results cannot be reproduced from the code. Only ImageNet checkpoints are released. The training recipe in §5 is the only source for the CIFAR models.
• Table 1 compares unequal schedules. The FRNN models use 60 epochs; the ResNet rows use 200 epochs or more tab. 1 caption. The authors say they did not focus on absolute accuracy.
• Source version. The fetched LaTeX is a 2016 draft (its date footnote says it was written in 2016 and released as it is). arXiv lists version 2 updated on 2020-12-31. I did not compare the two versions.
• Equation 4 mixes indices. The text writes h_n = f(h_t; w_t) = (K+I)^n of x_0 eq. 4. Read it as n applications of the shared update to x_0. Equation 2 uses x_t for the input, while equation 3 uses x_{t-1}. Implement the update from equation 1, h_t = K(h_{t-1}) + h_{t-1}.
• Readout time versus depth. Each transition counts as one time unit §3, "Notations". The unrolled depths 13 and 23 in Table 1 are counted with the table's definition (weighted and pooling layers) tab. 1 caption. The paper does not show how these numbers are derived for the 3-state model.
• Time-specific batch norm beyond training. Statistics are computed per time step from the training set §3. The paper does not say which statistics to use for readout times beyond those seen in training, though it tests generalization to t=7 to 23 when training used t=13 fig. 5.
• Gradient sum, not mean. Tied weights receive the sum of their copies' gradients §3. A mean would change the effective learning rate.
• Batch norm scale and shift. Learnable scale and shift are off in the recurrent parts for CIFAR-10 but on for ImageNet §3 §5. Copying one setting into both experiments changes the model.
• Convergence is not guaranteed. The paper says the representation may not converge as t grows, though it observes good generalization over t §5.
• Compute. Four-state fully recurrent nets become very heavy for large t or many feature maps §5. Restricting transitions to neighbouring states and self-loops scales linearly with the number of states §5.
• ReLU trainability depends on time-specific batch norm. The paper reports no training difficulty with ReLU recurrent nets when time-specific batch norm is used, where earlier work needed careful initialization §1.
• Average versus sum. The paper gives about 1% for averaging in FRNNs §3. The combine rule is part of the model, not an implementation detail.
• Biological claims are conjectures. The paper presents the link to the visual cortex as a conjecture §6 and does not test it directly.
• Not stated: hardware, training time, weight decay, ImageNet batch size, parameter counts, and whether Table 1 errors are validation or test.

Lineage

• Builds on: residual learning He et al. 2015 and its pre-activation form He et al. 2016, which the paper says it adopts §2.1. Also recursive neural networks (Socher et al. 2011) and convolutional recurrent networks (Eigen et al. 2013; Liang et al. 2015) §4.
• Compared against: All-CNN, Highway Network and ResNet-110, -164, -1001 on CIFAR-10 tab. 1. The paper discusses stochastic depth §4.
• Superseded by: not stated in this card's sources.