RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks

In plain words

A standard convolution slides one small filter over an image. The same numbers are used at every position, which keeps the model small but means the filter cannot treat a pixel differently depending on where it sits inside the window. Attention layers add per-location weights to fix this, but the weights they learn are shared by all the overlapping windows that cover the same pixel, so they do not help much when the filter is large.

RFAConv (receptive-field attention convolution) changes what gets weighted. It first spreads the input out so that each window becomes its own small block of values, one per position in the filter. It then learns a separate weighting over those positions for each window. Each window therefore gets its own weights, and a normal strided convolution finishes the job. It can replace a standard convolution layer without redesigning the network.

The authors report a modest accuracy gain on image classification and object detection, for a small extra cost in parameters and compute. The extra memory cost is larger than the parameter count suggests. The numbers come from their own training setups, so they do not transfer directly to other settings.

TL;DR

RFAConv is a drop-in replacement for a k×k convolution. Each receptive-field window gets its own softmax attention over its k×k positions, so weights are not shared across overlapping windows §3.3. The authors report gains over standard convolution and over other attention modules on ImageNet, COCO, VOC and Roboflow-100 §4. The main cost is memory: the authors acknowledge a larger memory overhead §4.6, which follows from an intermediate tensor k² times the input size RFAresnet.py.

Problem

Standard convolution uses the same kernel at every position, so it cannot respond to location-specific content §1 §3.1. The authors argue that current spatial attention does not fix this for kernels larger than 1×1: the attention map has the input's size, so the weights it learns are shared by every window that covers a position §3.1. Self-attention would model position-specific weights, but the authors state that it adds significant computation and memory §3.3. The paper's question is whether spatial attention can be applied to the receptive-field features instead, at a small cost §1 §3.3.

Method

Equations are numbered in the compiled paper in order of appearance: eq. 1 is the 1×1 case in §3.1, eq. 2 the 3×3 expansion in §3.1, and eq. 3 the RFA formula in §3.3.

Overview. RFAConv (i) expands the input so each k×k window is a block of k² values per channel, (ii) learns a softmax over the k² slots of each channel at each window position, (iii) multiplies the two elementwise, and (iv) reshapes the result so each window is a k×k block. A stride-k convolution then reduces it back to one output per window §3.3 fig. 4.

1. Start from the 1×1 case. Multiplying the input by an attention weight and then by the kernel makes the product act as a per-position kernel eq. 1. With a 3×3 kernel, neighbouring windows overlap, so one attention weight is reused by several windows eq. 2 §3.1 fig. 2.

2. Make windows non-overlapping (receptive-field spatial feature). Expand each 3×3 window into its own non-overlapping block, so no weight is shared between windows §3.2 fig. 3.

3. Expand the features with a grouped convolution. The paper uses a k×k grouped convolution (one group per channel, k² outputs per channel), then batch norm and ReLU, as a faster substitute for PyTorch's Unfold. The output has C·k² channels at each position, the same layout Unfold would give §3.3 eq. 3 RFAresnet.py.

4. Compute the attention. Average-pool the input over each k×k window with the layer's stride, then apply a 1×1 grouped convolution to C·k² channels, then a softmax over the k² slots of each channel at each position §3.3 eq. 3 RFAresnet.py.

5. Weight the expanded features. Multiply the features from step 3 by the attention from step 4, elementwise eq. 3 RFAresnet.py.

6. Adjust the shape. Rearrange the C·k² slots of each window into a k×k spatial block, so the map is k times larger in height and width §3.3 fig. 4 RFAresnet.py.

7. Reduce with a stride-k convolution. A k×k convolution with stride k maps the rearranged map to the output channels. The paper specifies only the kernel and stride here §3.3 RFAresnet.py.

# RFAConv forward, grouped-convolution variant  [§3.3] [eq. 3]
# Input:  features (B, C, H, W); window size k; stride s
# Output: features (B, C_out, H', W')

def rfa_conv(features, k, s):
    # Attention logits, one k*k map per channel  [§3.3] [eq. 3]
    pooled = avg_pool(features, k, s)           # (B, C, H', W')
    logits = grouped_conv_1x1(pooled)           # (B, C*k*k, H', W')
    # Softmax over the k*k slots of each channel  [§3.3] [eq. 3]
    attention = softmax(
        reshape(logits, B, C, k*k, H', W'),
        axis=2,
    )
    # Expanded window features, learned, then ReLU(BN(.))  [§3.3] [eq. 3]
    expanded = grouped_conv_kxk(features, k, s)  # (B, C*k*k, H', W')
    expanded = relu(batch_norm(expanded))
    # Weight each slot of each window  [§3.3] [eq. 3]
    weighted = reshape(expanded, B, C, k*k, H', W') * attention
    # Rearrange each window's k*k slots into a k x k block  [§3.3] [fig. 4]
    tiled = adjust_shape(weighted, k)           # (B, C, k*H', k*W')
    # Non-overlapping k x k convolution, stride k  [§3.3]
    return conv_kxk(tiled, k, stride=k)         # (B, C_out, H', W')

Variants (RFCBAMConv, RFCAConv). The authors apply the same expansion in front of CBAM and Coordinate Attention, giving RFCBAMConv and RFCAConv §3.3 fig. 7. In RFCBAMConv, Squeeze-and-Excitation replaces CBAM's channel module; channel and spatial weights are applied together rather than one after the other §3.3. The code computes the spatial weights from max and mean statistics over channels, which feed a 3×3 convolution and a sigmoid model.py. RFCAConv applies Coordinate Attention's directional pooling to the expanded map §3.3 model.py.

Hyperparameters

Results

Author-reported. Setup: ImageNet-1k (1.28M training images, 50k validation) with ResNet18 and ResNet34, 100 epochs, batch 128 §4.1. COCO2017 with YOLOv5n, YOLOv7-tiny and YOLOv8n, 300 epochs, batch 32 §4.3. Baselines are the unmodified models in the same tables. The paper does not state whether baselines were re-run or copied.

Per-image validation time (author-measured, hardware not stated per row): YOLOv5n 4.4 ms, RFAConv 5.3 ms, RFCAConv 5.5 ms; YOLOv8n 4.2 ms, RFAConv 4.5 ms, RFCAConv 4.7 ms; YOLOv7-tiny 3.4 ms, RFAConv 4.4 ms tab. 7. The authors state that many replaced convolutions cost more time, so the number of replacements should be limited when real-time speed matters §4.3.

Paper vs code

Code: model.py, , at commit df71c1a6235148b6cfb05d8ebb83fe529bcaa08b. The repository has no LICENSE file at its root. The Detection/YOLOv5/ folder is a copy of YOLOv5, which is GPL-3.0 LICENSE; this card cites code paths and does not quote it.

• BN and ReLU after the stride-k convolution. The paper's text gives only a k×k convolution with stride k §3.3. The classification copy matches that RFAresnet.py, and BatchNorm plus ReLU come from the BasicBlock RFAresnet.py. The two other copies add BatchNorm and ReLU after the convolution model.py common.py. Which one the authors used for the reported numbers is not stated.
• Two classes named RFAConv in model.py. The GroupConv version is at model.py; a second class named RFAConv, based on Unfold, is at model.py. The second definition shadows the first if the module is imported by name. The paper uses the GroupConv version §3.3.
• "Parameterless" Unfold. The paper says the Unfold method is parameterless §3.3. The Unfold class in model.py has a learned 1×1 grouped convolution and BatchNorm model.py.
• Replacement scope in detection. The paper says all 3×3 convolutions in the YOLOv5 and YOLOv8 yaml files are replaced §4.3. The shipped YOLOv5n yaml replaces only the stride-2 convolutions and leaves one as CAConv yolov5n.yaml (NovelConv). The YOLOv8n yaml does the same at its stride-2 convolutions YOLOv8n.yaml.
• YOLOv7-tiny placement. The paper says the first 3×3 convolution in each ELAN block of the backbone is replaced §4.3. In the shipped yaml, a standard 3×3 convolution comes first and NovelConv follows it, so NovelConv replaces the second 3×3 in each block YOLOv7-tiny.yaml.
• Learning-rate schedule. No discrepancy. The paper's "every 30 epochs" over 100 epochs matches milestones 30, 60, 90 §4.1 config.py.
• Detection registration. The yaml files use NovelConv and CAConv, but no Python file in the repository defines NovelConv, and models/yolo.py (the usual YOLOv5 parser) is not in the repository. The README says RFAConv must be registered as a component README.md. See Pitfalls.

No other discrepancies found in the classification training loop, the ResNet block structure or the RFAConv arithmetic.

Pitfalls

• Memory. The expanded tensor has C·k²·H·W elements, k² times the input RFAresnet.py. The authors state this memory cost and say that devices with limited storage may not be able to afford 3×3 non-shared weights §4.6. A 2×2 variant is described as possible but is not tested §4.6.
• Attention is a per-channel gain. The grouped 1×1 convolution has one input channel per group and no bias, so each slot's logit is a learned gain times that channel's pooled value RFAresnet.py. There is no mixing across channels in the attention. This is inferred from the code.
• Expanded features are learned. The paper describes the expansion as a fast stand-in for Unfold, which would copy pixels. The grouped convolution has learned filters, then BatchNorm and ReLU eq. 3 RFAresnet.py. The expanded values are therefore learned features, not a fixed rearrangement of the input.
• Softmax axis. The softmax runs over the k² slots of each channel at each output position RFAresnet.py. It does not run over channels or over the whole window.
• Padding and stride of the branches. The paper does not state padding or stride for the pooling and expansion branches. The code pads by k//2 and uses the layer's stride RFAresnet.py. The final convolution has no padding RFAresnet.py.
• Crash in RFCBAMConv. Both model.py and common.py call nn.ReLu(), which does not exist in PyTorch, so constructing these modules fails model.py common.py. The classification copy does not contain this typo.
• Assert that never fires. The kernel-size check is assert("..."), which always passes, so even kernel sizes are accepted model.py common.py.
• Missing detection wiring. The detection folder has no models/yolo.py, and NovelConv is undefined. The yaml files cannot be parsed as shipped . The README asks the user to register components README.md. The detection numbers cannot be reproduced from this repository alone.
• Training script defaults. The --evaluate default is a Windows path RFAtrain.py. Because the default is truthy, the script checks that file and raises an error unless it exists RFAtrain.py. Several flags use type=bool, so a value such as False on the command line is parsed as true RFAtrain.py.
• Channel and reduction defaults are code-only. The SE ratio of 16 and the Coordinate Attention reduction of 32 are not in the paper [code: model.py#L80, #L149]. The spatial statistics for RFCBAM are computed before the SE reweighting model.py.
• Dataset descriptions. The text says ImageNet-200 has 100000 training and 10000 validation images §4.2. The paper does not describe how this subset was built, and the name does not say whether it has 200 classes.
• Gains are uneven. The authors note smaller gains on VOC and weaker results for RFAConv on semantic segmentation, where Coordinate Attention and CAM do better §4.6. The RFCAConv variant is the one that leads on segmentation tab. 10.
• Inconsistent wording. §4.1 says Grad-CAM is used; §4.6 says "CAM algorithm" for the same figure. The paper does not say which is correct.
• Latency is per image on unnamed hardware. Table 7 gives per-image times without hardware per row tab. 7.
• Segmentation comparison is partial. The authors trained only the attention variants whose pretrained backbone weights were available §4.5.

Lineage

• Builds on: CBAM (Woo et al., 2018) and Coordinate Attention (Hou et al., 2021), see Coordinate Attention for Efficient Mobile Network Design. RFCBAMConv and RFCAConv modify these two modules §3.3. Deformable convolution (Dai et al., 2017) is cited as a related idea that changes sampling positions rather than weights §2.1.
• Compared against: ECA (Wang et al., 2020), CAM, TA (Misra et al., 2021), MCA (Yu et al., 2023), HAM (Li et al., 2022), GAM (Liu et al., 2021), EMAS, QEA, QCA, AFF-CAM and LMA §4.1 §4.4.
• Superseded by: not stated.