Channel attention

Overview

A channel-attention module gives each feature channel a weight, computed from the input, and multiplies the feature map by those weights. The most common form is squeeze-and-excitation (SE): summarise each channel, pass the summary through a bottleneck, then apply a sigmoid gate 2204.07756 §2.1.1 eq. 1 2402.15648 §3.3 2103.12376 §4.3. NAM is a different form: it has no squeeze or bottleneck, and its weights come from batch-norm scale factors 2111.12419 §3. Because the gate only rescales channels, the output has the same shape as the input, so the module can be inserted into an existing block without changing tensor sizes.

Not every module called channel attention is a gate. RFAConv's own attention weights the positions inside each convolution window, one softmax per channel over those positions 2304.03198 §3.3. Its channel-gate variants are covered below. The U-shape Transformer's channel module attends across channels and outputs a context map, not one weight per channel 2111.11843 eq. 5-6. It is covered as the last variant.

Variants

SE-style gate inside a residual block (MambaIR)

MambaIR places a channel-attention gate after a convolution bottleneck in the local branch of each residual state-space block 2402.15648 §3.3. The gate averages each channel, reduces the channels with a 1×1 convolution, applies ReLU, expands them back with a second 1×1 convolution, and applies a sigmoid mambair_arch.py. The squeeze factor is 30 in the code, which leaves 6 hidden channels at C = 180 and 2 at C = 60 mambair_arch.py. The paper names the module but does not state the reduction ratio in its text 2402.15648 §3.3. Card: MambaIR: A Simple Baseline for Image Restoration with State-Space Model.

The paper's ablation removes the convolution and the channel attention together in one row, so no row removes the gate alone 2402.15648 tab. 1. On Set5, removing only the convolution gives 38.48 dB, and removing the convolution and the gate gives 38.55 dB. The setting is not stated, so this does not measure the gate's contribution.

SE-style feature attention after each denoising module (RIDNet)

RIDNet stacks four enhancement attention modules. Each one ends with a feature attention unit, applied after an enhanced residual block 1904.07396 §3.2. The unit averages each channel over the whole map, reduces the channels by 16 with a 1×1 convolution, applies a shrinkage function, restores the channel count with a second 1×1 convolution, and applies a sigmoid. Each channel is then multiplied by its gate 1904.07396 §3.2.1 1904.07396 eq. 7-9. The paper leaves the shrinkage threshold unstated. The released code uses a ReLU in that place instead ridnet.py. Card: Real Image Denoising with Feature Attention.

Table 1 is the only cited ablation that toggles this unit with the other components fixed. Adding it to the plain model gives 28.45 to 28.52 dB on BSD68 1904.07396 tab. 1, configs 1 and 5. Removing it from the full configuration gives 28.90 to 28.86 dB 1904.07396 tab. 1, configs 8 and 4. The table is at 2e5 iterations, and the card records no seeds or variance for it. The gap is 0.04 to 0.07 dB, so it is small.

Gate over fused two-stream features (face forgery detection)

The face forgery detector concatenates the high-level features of its RGB and noise streams at the end of the exit flow, then applies channel-wise attention to the concatenation before the prediction 2103.12376 §4.3. The paper cites the squeeze-and-excitation work for this step. The reduction ratio is not stated, and the card does not describe the gate's internal layers 2103.12376 §4.3. The ablation rows do not vary this gate 2103.12376 tab. 3. Card: Generalizing Face Forgery Detection with High-frequency Features.

Block at the start of each unfolded stage (DGUNet)

DGUNet unfolds a proximal gradient method into seven stages. Each stage's proximal mapping begins with a channel-attention block that extracts shallow features from the stage's gradient-step output 2204.13348 §3.2. The placement is different from the two variants above: the gate runs before the encoder-decoder, not inside a residual block or on a fused feature map. The paper does not state the block's reduction ratio or its internal layers 2204.13348 §3.2. Its ablation removes the gradient module, the spatially adaptive fusion or the inter-stage fusion, and not this block 2204.13348 tab. 6. Card: Deep Generalized Unfolding Networks for Image Restoration.

Batch-norm scales as channel weights, with a sparsity penalty (NAM)

NAM takes its channel weights from a batch norm placed in the block. Each channel's scale factor gamma is divided by the sum of all scale factors in the module, and a sigmoid of the normalised, weighted features gives the mask 2111.12419 §3 2111.12419 eq. 2. The channel part adds no fully connected or convolutional layers. It needs one scale per channel, so its parameter count is 3,840 for ResNet-50 against 696,320 for the CBAM channel part 2111.12419 app. A.1 2111.12419 tab. 3. The loss adds an L1 penalty on the scale factors to push weak channels toward zero, with p = 0.0001 for CIFAR-100 and 0.001 for ImageNet 2111.12419 §4 2111.12419 eq. 4. The released code has the penalty off by default and applies it only with a flag train_cifar100.py.

On CIFAR-100 with ResNet-50, the channel-only variant has the lowest top-1 error of the compared modules: 19.09%, against 19.44% for CBAM, 19.97% for BAM, 20.15% for TAM, 20.29% for SE and 22.74% for the plain network 2111.12419 tab. 1. The margin over CBAM is 0.35 points from one run. The paper reports no seeds or variance 2111.12419 §4. The plain-network row removes the whole module, so it does not isolate the gate alone. The card does not say whether the reported NAM runs used the penalty. The paper's spatial submodule is absent from the released code, and the code has no ImageNet training script attention.py. The repository has no license file, so the card records no code license and no code is quoted. Implement from the paper's equations, not by copying the repository. Card: NAM: Normalization-based Attention Module.

Weights pooled across resolutions, with a spatial vector (Lite-HRNet)

Lite-HRNet's conditional channel weighting sits inside a shuffle block. It keeps the first half of each branch's channels and rescales the second half 2104.06403 §3.1 2104.06403 eq. 2. The weights come from two parts. The cross-resolution part pools every branch except the lowest to the lowest branch's size, concatenates them, and passes them through a 1×1 convolution, ReLU, another 1×1 convolution and a sigmoid. The output is split into one map per branch, and each map is upsampled back to its branch 2104.06403 §3.2 2104.06403 eq. 3-4. The spatial part gives each branch one vector, from global average pooling and two fully connected layers with a sigmoid, and the same vector is used at every position 2104.06403 §3.2 2104.06403 eq. 6. The two are multiplied into the channels 2104.06403 eq. 5. The authors say the cross-resolution weights do the work of a 1×1 convolution for exchanging information across channels 2104.06403 §3.2. Neither the paper's text nor its equations give the reduction ratios. The code uses 8 for the cross-resolution weights and 4 for the spatial weights, and adds batch normalization that the paper's equations omit litehrnet_18_coco_256x192.py litehrnet.py litehrnet.py.

The ablation in table 7 compares the two weight types on Lite-HRNet-18 at 256×192 COCO val. Spatial weights alone give 62.6 AP, cross-resolution weights alone give 63.0 and both give 64.8 2104.06403 tab. 7. No row removes the weighting entirely, so the table shows how the two parts combine, not what the module gains over no weighting. Table 6 compares the weighting with 1×1 convolutions, but its rows differ in parameter count 2104.06403 tab. 6. The card records no seeds or variance, and we did not re-run the numbers. Card: Lite-HRNet: A Lightweight High-Resolution Network.

Directional pooling that keeps row and column position (coordinate attention)

Coordinate attention pools each channel along the width to get a height vector, and along the height to get a width vector 2103.02907 §3.2.1 2103.02907 eq. 4-5. One shared 1×1 convolution with reduction ratio r mixes the two vectors. Two more 1×1 convolutions and sigmoids give one gate per row and one per column, and the input is multiplied by both 2103.02907 §3.2.2 2103.02907 eq. 6-9. The default r is 32 coordatt.py. The paper leaves the non-linearity unnamed. The code uses BatchNorm followed by h-swish, and floors the bottleneck width at 8 coordatt.py coordatt.py. Because the two gates multiply, each output value is scaled by two numbers between 0 and 1. The paper does not discuss how this changes the activation scale 2103.02907 eq. 9. Card: Coordinate Attention for Efficient Mobile Network Design.

On MobileNetV2, the authors report 72.3% ImageNet top-1 for the baseline, 73.5% with SE and 74.3% with coordinate attention. The parameter counts are 3.5M, 3.89M and 3.95M 2103.02907 tab. 1. The coordinate-attention row uses r = 32 and the SE row uses r = 24, so the comparison is not at matched r. At r = 16 for coordinate attention and r = 12 for SE, the figures are 74.7% and 74.1% 2103.02907 tab. 4. On Pascal VOC detection the authors report 73.1 mAP, against 71.7 for both the baseline and SE 2103.02907 tab. 7. On Cityscapes at output stride 8, they report 74.0 mIoU, against 72.2 for SE and 71.4 for the baseline 2103.02907 tab. 9. The latency column is from one device, a Pixel 4, and the card records no variance for any of these numbers.

Window-level attention inside a convolution, with channel-gate variants (RFAConv)

RFAConv replaces a 3×3 convolution. It expands each window into k² values per channel, learns a softmax over the k² slots of each channel at each window position, multiplies the two, rearranges the slots into a k×k block, and ends with a stride-k convolution 2304.03198 §3.3 2304.03198 eq. 3. Its attention weights positions inside a window, not channels. The softmax runs over the slots of each channel, not across channels [code: classification/RFAresnet.py#L37]. The authors name the memory cost of the expansion as the main cost of the method 2304.03198 §4.6.

Two variants add channel attention to the expanded map. RFCBAMConv replaces CBAM's channel module with squeeze-and-excitation, and applies channel and spatial weights together. RFCAConv applies coordinate attention's directional pooling to the expanded map 2304.03198 §3.3 model.py. On ImageNet-1k with ResNet18, the baseline is 69.59% top-1. RFAConv gives 71.23% 2304.03198 tab. 3, and RFCBAMConv and RFCAConv give 72.15% and 72.01% 2304.03198 tab. 4. On COCO2017 with YOLOv5n, the baseline is 27.5 AP, RFAConv gives 29.0 and RFCAConv gives 29.5 2304.03198 tab. 7. The paper does not say whether the baselines were re-run or copied, and the card records no variance. RFCAConv is 0.78 points above RFAConv on ResNet18, but that gap comes from two different tables. The card does not show a run that isolates the channel module inside the same setup. Card: RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks.

Dual attention in a decoder (camouflaged-object model)

The joint camouflaged-object model's fixation decoder reduces each backbone stage to 32 channels and then applies dual residual attention, which the paper takes from DANet: position attention and channel attention 2103.04011 §3.2 fig. 3. The code's decoders use RCAB channel-attention blocks instead ResNet_models.py. The paper does not state the channel module's layers. Its ablation (tab. 5) compares single-task and joint models, not the attention block, so it gives no evidence on the channel module alone 2103.04011 tab. 5. Card: Simultaneously Localize, Segment and Rank the Camouflaged Objects.

Forms named in the survey (SE and CBAM)

The survey files channel attention under soft attention and gives two formulas. The first is the SE gate: global pooling, a fully connected layer with ReLU, a second fully connected layer and a sigmoid 2204.07756 §2.1.1 eq. 1. The second is the CBAM channel gate. It pools with both max and average, passes both descriptors through one shared MLP, sums them and applies a sigmoid 2204.07756 §2.1.1 eq. 2. The survey also lists other channel-attention methods (ECA, split attention, second-order and frequency-channel attention) without formulas, except that second-order attention takes a power below one half 2204.07756 §2.1.1. The survey runs no experiments of its own, so it gives no evidence on which form performs better; the numbers it quotes come from the original papers 2204.07756 §1. Card: Visual Attention Methods in Deep Learning: An In-Depth Survey.

Channel gate in every nested node (DNANet)

DNA-Net finds single-frame infrared small targets with a U-shaped network whose nodes are densely nested. Each node applies two 3×3 convolution stages with batch norm, then a channel-then-spatial block (CSAM) 2106.00487 §3.2.3. The channel part is the CBAM form: a shared one-hidden-layer MLP over the global max and global average of each channel, then a sigmoid 2106.00487 eq. 3-4. The paper gives no reduction ratio, but the code uses a hidden width of C/16 model_DNANet.py. The gate runs at every fusion point of the nest, not on one stage's feature map. Card: Dense Nested Attention Network for Infrared Small Target Detection.

The paper's ablation removes the channel attention and keeps the spatial attention, which isolates the channel gate in a segmentation network 2106.00487 tab. VI. On NUDT-SIRST, IoU falls from 87.09 to 86.27, Pd from 98.73 to 96.96 and Fa rises from 4.223 to 4.881. On NUAA-SIRST, IoU falls from 77.47 to 76.20 and Fa rises from 2.353 to 12.69. These numbers are author-reported, and the card records no seeds or variance for them.

Channel-axis transformer in place of skip connections (U-shape Transformer)

The U-shape Transformer's CMSFFT is a transformer block that replaces the U-Net skip connections. It builds queries for each of the four encoder scales. Keys and values come from all four scales concatenated along the channel axis 2111.11843 §3.2.3 2111.11843 eq. 4. The attention map is channel by channel: it is instance-normalised and passed through a softmax, then applied to the values 2111.11843 eq. 5. The heads are averaged, a feed-forward block follows, and the step repeats four times [arxiv:2111.11843 eq. 6-7]. Nothing here produces a per-channel weight. The paper's text and equations disagree on the orientation of the added query term, and the code adds the block input instead 2111.11843 eq. 6 CMSFFT.py. Card: U-shape Transformer for Underwater Image Enhancement.

The ablation adds CMSFFT to the pix2pix baseline. PSNR rises from 19.34 to 22.47 dB on Test-L400 and from 19.36 to 21.72 dB on Test-U90 2111.11843 tab. 6. This measures a transformer block that replaces the skips, not channel reweighting alone. The table has no row that removes CMSFFT from the full model, and the card records no variance.

Fusion gate in a depth decoder (HR-Depth)

HR-Depth's decoder has a grid of nodes. Some of its fusion steps are an fSE block, which pools each channel over the whole map, passes the summary through two fully connected layers with a sigmoid, multiplies the features by the weights, and then fuses the channels with a 1×1 convolution [arxiv:2012.07356 §3.4] fig. 3. The paper sets the reduction ratio to 4 in all experiments 2012.07356 §3.4 eq. 9. The released code uses 16 [code: layers.py#L309], and adds a ReLU between the two fully connected layers that the paper does not describe layers.py. Card: HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation.

On KITTI at 640×192 with monocular training, Table 4 gives Abs rel 0.115 for Monodepth2, 0.114 with fSE alone, 0.112 with dense skip connections, 0.110 with dense skips and a row labelled "SE", and 0.109 with dense skips and fSE 2012.07356 tab. 4. The card does not say what the "SE" row uses, so it does not show whether the fusion step or the plain gate is responsible. The gaps are 0.001 to 0.003 in Abs rel, and the card records no variance. The fSE row has 14.62M parameters and the SE row 16.23M 2012.07356 tab. 4.

Gate in the fusion step and in every residual block (ESRT)

ESRT's high preserving block ends with an SE-style channel gate after a 1×1 fusion convolution 2108.11084 §3.1. Its residual block (ARFB) is described without a channel gate 2108.11084 §3.2.1 fig. 6. The released code adds one after the 1×1 convolution of every ARFB esrt.py. The code's reduction is 16, which gives 2 hidden channels at 32 input channels esrt.py. The paper does not state a ratio. Card: Transformer for Single Image Super-Resolution.

The ablation on Set5 ×4 (tab. 3) keeps the high-frequency branch and the ARFB, and varies only the channel-attention column. With the gate it reaches 32.19 dB at 751K parameters, and without it 32.08 dB at 724K 2108.11084 tab. 3. The table does not remove the gate from the ARFB blocks that the code adds, so it does not measure that gate. The card records no variance for these numbers.

Channel attention on CSP layers of a real-time detector (RTMDet)

RTMDet's backbone and neck use CSP-style layers. Each one can add a channel-attention module on its concatenated features csp.py csp.py. The card does not describe the module's layers, and the card gives no reduction ratio for it. Card: RTMDet: An Empirical Study of Designing Real-Time Object Detectors.

The change is part of the model-balance study. Reducing the second and third stages from 9 to 6 blocks cut latency by about 20% and cost 0.5 AP, from 51.4 to 50.9 2212.07784 tab. 5 §4.3. Adding channel attention to the 3-6-6-3 model recovered 0.4 AP, to 51.3, at 2.40 ms against 2.11 ms without it, with 52.30M parameters against 50.92M 2212.07784 tab. 5 §4.3. Latency is FP16 with TensorRT, batch 1, on one NVIDIA 3090 2212.07784 §4.1. The ablations are single runs with no spreads reported 2212.07784 tab. 5.

Per-time-step gate on spiking membrane potential (Attention SNN)

Attention Spiking Neural Networks adds the CBAM channel module to conv-based spiking layers. Its channel gate pools over space at each time step, gives one weight per channel and step, and shares the MLP across steps 2209.13929 §3.3 eq. 10-11. The paper recommends applying it to the integrated membrane potential, not to the convolution output, and its channel equation uses the membrane input 2209.13929 §6.1 eq. 14. The released code places the gate on the batch-normalised convolution output, before the spiking step LIF_Module.py, and its gate has no membrane term Attention.py. The paper does not state the reduction ratio r_c eq. 10. The ImageNet residual model uses a ratio of 8 in its channel-spatial block resnet_2.py.

On DVS128 Gesture with the three-layer model, the vanilla network reaches 90.63 ± 0.58% 2209.13929 tab. 1. The channel gate alone reaches 93.88%, and the full temporal, channel and spatial combination reaches 96.53% 2209.13929 tab. 6. The placement table gives the channel gate 91.84% on the convolution output before the spiking step, 91.58% after it and 93.88% on the membrane potential 2209.13929 tab. 4. On ImageNet-1K at one time step, the channel gate adds 1.7 points to MS-Res-SNN-18 (61.70% to 63.42%), 3.8 to MS-Res-SNN-34 and 0.5 to MS-Res-SNN-104 2209.13929 tab. 3. The authors say the same module gives +0.7 on CBAM's CNN ResNet-34 against +5.0 on the SNN 2209.13929 §5.2.2. Gesture rows are mean ± std over 10 runs, and the ImageNet rows are single runs 2209.13929 tab. 1. Energy figures come from an operation-count formula, not from hardware 2209.13929 §4. Card: Attention Spiking Neural Networks.

Per-stream gate after spatial alignment in a two-stream encoder (SwinNet)

SwinNet encodes the colour image and a second modality (depth or thermal) with two Swin-B streams. At each level the two streams are aligned by a shared spatial attention map. Each aligned stream then gets its own channel attention: a global max pool, a 1×1 convolution and a sigmoid. Each channel weight multiplies that stream's original, unaligned feature 2204.05585 eq. 4-6. The edge branch uses a channel-attention block with a residual connection 2204.05585 eq. 8. The paper's alignment ablation replaces the whole alignment module with BBS-Net's depth-enhanced module, which gives NLPR S-measure 0.936 against 0.941 for the full model 2204.05585 tab. 4. No row removes the channel re-calibration alone, so the table does not isolate this gate. The paper's text uses one 1×1 convolution, and the released code uses two with reduction 16 and a ReLU between them 2204.05585 eq. 5 Swin_Transformer.py. Card: SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection.

When to use which

Implementation sketch

Pseudocode in the same style as the paper cards. The first function follows the MambaIR form. The CBAM comment shows the change to the pooling step. The coordinate-attention function pools along each axis and gives row and column gates. The NAM function uses batch-norm scales as weights, with the penalty below it. The last function applies the gate to a fused two-stream input. The face forgery card does not give that gate's structure, so the SE form is an assumption, and the ratio is a parameter because the paper does not state it.

# SE-style channel gate, MambaIR form  [arxiv:2402.15648 §3.3]
# Input:  x (B, C, H, W); output: x with each channel rescaled
# Squeeze factor r = 30 in the code  [code: basicsr/archs/mambair_arch.py#L41]
# RIDNet uses the same shape with r = 16 and a shrinkage in place of the ReLU
# (the code uses ReLU)  [arxiv:1904.07396 eq. 8]

def channel_attention(x, r):
    C = x.shape[1]
    # one summary value per channel
    s = avg_pool(x)                          # (B, C, 1, 1)
    # bottleneck C -> C/r -> C, with ReLU in between
    s = relu(conv1x1(s, out=C // r))
    s = conv1x1(s, out=C)
    # per-channel weights in (0, 1)
    w = sigmoid(s)
    return x * w                             # broadcast over H and W

# CBAM-style channel attention  [arxiv:2204.07756 §2.1.1 eq. 2]
# Replaces the avg_pool line above. Both pooled descriptors go through
# the same bottleneck, and the two outputs are summed before the sigmoid.
#   w = sigmoid(mlp(max_pool(x)) + mlp(avg_pool(x)))
# DNANet uses this form in every nested node, with hidden width C/16 in code
#  [arxiv:2106.00487 eq. 3-4] [code: model/model_DNANet.py#L33-L35]

# Coordinate attention: one gate per row and per column  [arxiv:2103.02907 §3.2]
# Input: x (B, C, H, W); r = 32 by default  [code: coordatt.py#L23]
# The paper leaves the non-linearity open; the code uses BatchNorm + h-swish
def coordinate_attention(x, r):
    C, H, W = x.shape[1], x.shape[2], x.shape[3]
    # pool along width (keep H) and along height (keep W)  [eq. 4-5]
    x_h = avg_pool(x, over=W)                # (B, C, H, 1)
    x_w = transpose(avg_pool(x, over=H))     # (B, C, W, 1)
    # one shared bottleneck over the joined strip  [eq. 6]
    y = concat([x_h, x_w], axis=2)           # (B, C, H + W, 1)
    y = act(bn(conv1x1(y, out=max(8, C // r))))
    y_h, y_w = split(y, [H, W], axis=2)
    # one 1x1 conv and sigmoid per direction  [eq. 7-8]
    g_h = sigmoid(conv1x1(y_h, out=C))                  # (B, C, H, 1)
    g_w = sigmoid(conv1x1(y_w, out=C))                  # (B, C, W, 1)
    # multiply by both gates at each position  [eq. 9]
    return x * g_h * transpose(g_w)          # (B, C, H, W)

# NAM channel gate, batch-norm scales as weights  [arxiv:2111.12419 §3] [eq. 2]
# Input: x (N, C, H, W); bn is a BatchNorm over C channels in the module
def nam_channel(x, bn):
    # normalise with the module's own batch norm  [eq. 1]
    y = bn(x)
    # channel weights: |gamma| divided by its sum in the module  [§3]
    g = abs(bn.weight)                       # (C,)
    w = g / sum(g)                           # no gradient flows through w
    # sigmoid of the weighted features is the mask  [eq. 2]
    m = sigmoid(y * w.view(1, C, 1, 1))
    # the code returns mask * x, not the mask alone  [code: MODELS/attention.py#L23]
    return m * x

# L1 penalty on the scales, added to the training loss  [eq. 4] [§4]
# p = 1e-4 for CIFAR-100, 1e-3 for ImageNet. The code adds p * sign(gamma)
# to the gradient instead  [code: train_cifar100.py#L55-L69]
loss = cross_entropy(model(x), target) + p * sum(abs(g) for g in nam_gammas)

# Gate over two concatenated streams  [arxiv:2103.12376 §4.3]
# Input:  rgb (B, C1, H, W); noise (B, C2, H, W); r not stated in the paper
def fusion_gate(rgb, noise, r):
    # concatenate the two streams along the channel axis
    fused = concat([rgb, noise], axis=channel)
    # assumption: the card names channel attention but not its structure,
    # so this reuses the SE-style gate above; choose and test r
    return channel_attention(fused, r)

HR-Depth's fSE block is the same gate applied to a fusion input: pool each channel, two fully connected layers with a sigmoid, multiply, then a 1×1 convolution 2012.07356 §3.4. Its reduction is a parameter (the paper uses 4, the code 16). The code also adds a ReLU between the two layers, which the paper does not describe layers.py.

Contradictions

The cards disagree on what channel attention is. Most rescale channels with a gate, but the U-shape Transformer's channel module attends across channels and outputs a context map 2111.11843 eq. 5-6. Among the gates, the cards place them in different parts of the network and measure different things. Five ablations isolate a gate: DNANet's 2106.00487 tab. VI, RIDNet's 1904.07396 tab. 1, ESRT's 2108.11084 tab. 3, RTMDet's 2212.07784 tab. 5 and Attention SNN's channel-only rows 2209.13929 tab. 6. The others do not (see the variants above). HR-Depth's fusion-gate table has a row labelled "SE" that the card cannot define 2012.07356 tab. 4. HR-Depth's paper gives a reduction of 4 and its code 16, and the code adds a ReLU between the two fully connected layers 2012.07356 §3.4 eq. 9 layers.py layers.py. ESRT's paper describes a channel gate in the high preserving block only, and its code adds one to every residual block as well 2108.11084 §3.1 2108.11084 §3.2.1 esrt.py. The camouflaged-object paper names DANet's channel attention for its decoders, but its code uses RCAB blocks 2103.04011 §3.2 ResNet_models.py. RIDNet's paper and code differ on the activation: soft-shrinkage in the text, ReLU in the code 1904.07396 §3.2.1 ridnet.py. The U-shape Transformer's paper and code differ on the residual in its channel module and on the attention scale, which the paper gives as the square root of C and the code as the square root of 480 2111.11843 eq. 5-6 CMSFFT.py CMSFFT.py. The NAM paper and its code differ in three places: the code takes the absolute value of each scale factor, the code returns the mask times the input, and the paper's spatial submodule is absent from the code 2111.12419 §3 attention.py attention.py. The NAM ImageNet row is described as channel and spatial attention combined 2111.12419 §4 2111.12419 tab. 2, but the code has no ImageNet training script. The survey's own method counts disagree between its abstract, introduction and conclusion 2204.07756 §1 2204.07756 §5. It files harmonious attention under channel attention, although that method mixes soft and hard attention 2204.07756 §2.1.1. Coordinate attention's paper leaves the non-linearity open, and its code uses BatchNorm with h-swish 2103.02907 §3.2.2 coordatt.py. Attention SNN's paper and code differ in three places. The paper pools the channel gate over space at each time step eq. 10, while the code pools over time, height and width together, giving one gate per sample and channel Attention.py. The paper's channel equation takes the membrane input eq. 14, and the code's gate runs on the output of the temporal gate, with no membrane term Attention.py. The paper recommends the membrane potential as the location, and the code uses the convolution output 2209.13929 §6.1 LIF_Module.py. SwinNet's paper gives one 1×1 convolution for its channel re-calibration eq. 5, and the code gives two with reduction 16 and a ReLU between them 2204.05585 eq. 5 Swin_Transformer.py.

Papers

• MambaIR: A Simple Baseline for Image Restoration with State-Space Model: an SE-style gate in the local branch of each residual state-space block.
• Generalizing Face Forgery Detection with High-frequency Features: channel-wise attention over the fused RGB and noise features before the classifier.
• Deep Generalized Unfolding Networks for Image Restoration: a channel-attention block at the start of each stage's proximal mapping.
• Dense Nested Attention Network for Infrared Small Target Detection: a CBAM-style channel gate, then a spatial gate, in every node of a densely nested U-shaped network.
• Simultaneously Localize, Segment and Rank the Camouflaged Objects: DANet-style position and channel attention in the decoders of a joint localization and segmentation model.
• Visual Attention Methods in Deep Learning: An In-Depth Survey: a taxonomy that gives the SE and CBAM channel-attention formulas, with no experiments of its own.
• NAM: Normalization-based Attention Module: a channel gate whose weights are batch-norm scale factors, with an L1 penalty on those factors and no extra layers for the channel part.
• Real Image Denoising with Feature Attention: an SE-style feature attention unit after each of four enhancement attention modules in a single-stage denoiser.
• Coordinate Attention for Efficient Mobile Network Design: pools along height and width to give one gate per row and per column, with a shared bottleneck.
• RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks: window-level attention inside a convolution, with SE and coordinate-attention channel variants (RFCBAMConv, RFCAConv).
• U-shape Transformer for Underwater Image Enhancement: a channel-axis transformer (CMSFFT) in place of the skip connections, not a per-channel gate.
• Lite-HRNet: A Lightweight High-Resolution Network: cross-resolution and spatial channel weights inside shuffle blocks of a small pose and segmentation network.
• HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation: an SE-style fusion block in the decoder nodes of a self-supervised depth network.
• Transformer for Single Image Super-Resolution: an SE-style gate in the fusion step of the high preserving block, plus a gate the code adds to every residual block.
• RTMDet: An Empirical Study of Designing Real-Time Object Detectors: an optional channel-attention module on the CSP layers of a real-time detector.
• SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection: a per-stream channel re-calibration after spatial alignment in a two-stream Swin encoder, plus a channel-attention block in the edge branch.
• Attention Spiking Neural Networks: a CBAM-style channel gate per time step on the membrane potential of spiking layers, with temporal and spatial variants.