---
kind: concept
slug: channel-attention
title: Channel attention
summary: >-
  Channel attention rescales a feature map's channels with a learned gate, usually squeeze-and-excitation. Seventeen papers cover it, mostly placing the gate in residual, fused, unfolded, nested, decoder, denoising, batch-norm, detector, spiking or two-stream blocks. Removing DNANet's gate costs 0.82 IoU on NUDT-SIRST.
tags: [attention, architecture, feature-reweighting, segmentation, image-restoration, pose-estimation, depth-estimation, super-resolution, object-detection, spiking, multimodal]
papers: ["2402.15648", "2103.12376", "2204.13348", "2106.00487", "2204.07756", "2111.12419", "1904.07396", "2103.04011", "2111.11843", "2103.02907", "2304.03198", "2104.06403", "2012.07356", "2108.11084", "2212.07784", "2204.05585", "2209.13929"]
related: [attention-mechanism, self-attention, state-space-model, vision-transformer, image-super-resolution, image-denoising, semantic-segmentation, multi-scale-feature-fusion, residual-learning, image-enhancement, efficient-vision-backbone, convolutional-neural-network, depthwise-separable-convolution, monocular-depth-estimation, object-detection, spiking-neural-network, salient-object-detection, multimodal-fusion]
status: draft
updated_at: 2026-11-01
---

## Overview

A channel-attention module gives each feature channel a weight, computed from the
input, and multiplies the feature map by those weights. The most common form is
squeeze-and-excitation (SE): summarise each channel, pass the summary through a
bottleneck, then apply a sigmoid gate [arxiv:2204.07756 §2.1.1 eq. 1]
[arxiv:2402.15648 §3.3] [arxiv:2103.12376 §4.3]. NAM is a different form: it has
no squeeze or bottleneck, and its weights come from batch-norm scale factors
[arxiv:2111.12419 §3]. Because the gate only rescales channels, the output has
the same shape as the input, so the module can be inserted into an existing block
without changing tensor sizes.

Not every module called channel attention is a gate. RFAConv's own attention
weights the positions inside each convolution window, one softmax per channel
over those positions [arxiv:2304.03198 §3.3]. Its channel-gate variants are
covered below. The U-shape Transformer's channel module attends across channels
and outputs a context map, not one weight per channel [arxiv:2111.11843 eq. 5-6].
It is covered as the last variant.

## Variants

### SE-style gate inside a residual block (MambaIR)

MambaIR places a channel-attention gate after a convolution bottleneck in the
local branch of each residual state-space block [arxiv:2402.15648 §3.3]. The
gate averages each channel, reduces the channels with a 1×1 convolution, applies
ReLU, expands them back with a second 1×1 convolution, and applies a sigmoid
[code: basicsr/archs/mambair_arch.py#L54-L59]. The squeeze factor is 30 in the
code, which leaves 6 hidden channels at C = 180 and 2 at C = 60
[code: basicsr/archs/mambair_arch.py#L41]. The paper names the module but does
not state the reduction ratio in its text [arxiv:2402.15648 §3.3]. Card:
[MambaIR: A Simple Baseline for Image Restoration with State-Space Model](/papers/2402.15648).

The paper's ablation removes the convolution and the channel attention
together in one row, so no row removes the gate alone [arxiv:2402.15648 tab. 1].
On Set5, removing only the convolution gives 38.48 dB, and removing the
convolution and the gate gives 38.55 dB. The setting is not stated, so this
does not measure the gate's contribution.

### SE-style feature attention after each denoising module (RIDNet)

RIDNet stacks four enhancement attention modules. Each one ends with a feature
attention unit, applied after an enhanced residual block [arxiv:1904.07396 §3.2].
The unit averages each channel over the whole map, reduces the channels by 16
with a 1×1 convolution, applies a shrinkage function, restores the channel count
with a second 1×1 convolution, and applies a sigmoid. Each channel is then
multiplied by its gate [arxiv:1904.07396 §3.2.1] [arxiv:1904.07396 eq. 7-9]. The
paper leaves the shrinkage threshold unstated. The released code uses a ReLU in
that place instead [code: TestCode/code/model/ridnet.py#L10-L23]. Card:
[Real Image Denoising with Feature Attention](/papers/1904.07396).

Table 1 is the only cited ablation that toggles this unit with the other
components fixed. Adding it to the plain model gives 28.45 to 28.52 dB on BSD68
[arxiv:1904.07396 tab. 1, configs 1 and 5]. Removing it from the full
configuration gives 28.90 to 28.86 dB [arxiv:1904.07396 tab. 1, configs 8 and 4].
The table is at 2e5 iterations, and the card records no seeds or variance for
it. The gap is 0.04 to 0.07 dB, so it is small.

### Gate over fused two-stream features (face forgery detection)

The face forgery detector concatenates the high-level features of its RGB and
noise streams at the end of the exit flow, then applies channel-wise attention
to the concatenation before the prediction [arxiv:2103.12376 §4.3]. The paper
cites the squeeze-and-excitation work for this step. The reduction ratio is not
stated, and the card does not describe the gate's internal layers
[arxiv:2103.12376 §4.3]. The ablation rows do not vary this gate
[arxiv:2103.12376 tab. 3]. Card:
[Generalizing Face Forgery Detection with High-frequency Features](/papers/2103.12376).

### Block at the start of each unfolded stage (DGUNet)

DGUNet unfolds a proximal gradient method into seven stages. Each stage's
proximal mapping begins with a channel-attention block that extracts shallow
features from the stage's gradient-step output [arxiv:2204.13348 §3.2]. The
placement is different from the two variants above: the gate runs before the
encoder-decoder, not inside a residual block or on a fused feature map. The
paper does not state the block's reduction ratio or its internal layers
[arxiv:2204.13348 §3.2]. Its ablation removes the gradient module, the spatially
adaptive fusion or the inter-stage fusion, and not this block
[arxiv:2204.13348 tab. 6]. Card:
[Deep Generalized Unfolding Networks for Image Restoration](/papers/2204.13348).

### Batch-norm scales as channel weights, with a sparsity penalty (NAM)

NAM takes its channel weights from a batch norm placed in the block. Each
channel's scale factor gamma is divided by the sum of all scale factors in the
module, and a sigmoid of the normalised, weighted features gives the mask
[arxiv:2111.12419 §3] [arxiv:2111.12419 eq. 2]. The channel part adds no fully
connected or convolutional layers. It needs one scale per channel, so its
parameter count is 3,840 for ResNet-50 against 696,320 for the CBAM channel part
[arxiv:2111.12419 app. A.1] [arxiv:2111.12419 tab. 3]. The loss adds an L1 penalty
on the scale factors to push weak channels toward zero, with p = 0.0001 for
CIFAR-100 and 0.001 for ImageNet [arxiv:2111.12419 §4] [arxiv:2111.12419 eq. 4].
The released code has the penalty off by default and applies it only with a flag
[code: train_cifar100.py#L48].

On CIFAR-100 with ResNet-50, the channel-only variant has the lowest top-1 error
of the compared modules: 19.09%, against 19.44% for CBAM, 19.97% for BAM, 20.15%
for TAM, 20.29% for SE and 22.74% for the plain network [arxiv:2111.12419 tab. 1].
The margin over CBAM is 0.35 points from one run. The paper reports no seeds or
variance [arxiv:2111.12419 §4]. The plain-network row removes the whole module, so
it does not isolate the gate alone. The card does not say whether the reported NAM
runs used the penalty. The paper's spatial submodule is absent from the released
code, and the code has no ImageNet training script [code: MODELS/attention.py#L6-L36].
The repository has no license file, so the card records no code license and no
code is quoted. Implement from the paper's equations, not by copying the
repository. Card:
[NAM: Normalization-based Attention Module](/papers/2111.12419).

### Weights pooled across resolutions, with a spatial vector (Lite-HRNet)

Lite-HRNet's conditional channel weighting sits inside a shuffle block. It
keeps the first half of each branch's channels and rescales the second half
[arxiv:2104.06403 §3.1] [arxiv:2104.06403 eq. 2]. The weights come from two
parts. The cross-resolution part pools every branch except the lowest to the
lowest branch's size, concatenates them, and passes them through a 1×1
convolution, ReLU, another 1×1 convolution and a sigmoid. The output is split
into one map per branch, and each map is upsampled back to its branch
[arxiv:2104.06403 §3.2] [arxiv:2104.06403 eq. 3-4]. The spatial part gives each
branch one vector, from global average pooling and two fully connected layers
with a sigmoid, and the same vector is used at every position
[arxiv:2104.06403 §3.2] [arxiv:2104.06403 eq. 6]. The two are multiplied into
the channels [arxiv:2104.06403 eq. 5]. The authors say the cross-resolution
weights do the work of a 1×1 convolution for exchanging information across
channels [arxiv:2104.06403 §3.2]. Neither the paper's text nor its equations
give the reduction ratios. The code uses 8 for the cross-resolution weights and
4 for the spatial weights, and adds batch normalization that the paper's
equations omit [code: configs/top_down/lite_hrnet/coco/litehrnet_18_coco_256x192.py#L58]
[code: models/backbones/litehrnet.py#L134] [code: models/backbones/litehrnet.py#L67-L82].

The ablation in table 7 compares the two weight types on Lite-HRNet-18 at
256×192 COCO val. Spatial weights alone give 62.6 AP, cross-resolution weights
alone give 63.0 and both give 64.8 [arxiv:2104.06403 tab. 7]. No row removes
the weighting entirely, so the table shows how the two parts combine, not what
the module gains over no weighting. Table 6 compares the weighting with 1×1
convolutions, but its rows differ in parameter count [arxiv:2104.06403 tab. 6].
The card records no seeds or variance, and we did not re-run the numbers. Card:
[Lite-HRNet: A Lightweight High-Resolution Network](/papers/2104.06403).

### Directional pooling that keeps row and column position (coordinate attention)

Coordinate attention pools each channel along the width to get a height vector,
and along the height to get a width vector [arxiv:2103.02907 §3.2.1]
[arxiv:2103.02907 eq. 4-5]. One shared 1×1 convolution with reduction ratio r
mixes the two vectors. Two more 1×1 convolutions and sigmoids give one gate per
row and one per column, and the input is multiplied by both
[arxiv:2103.02907 §3.2.2] [arxiv:2103.02907 eq. 6-9]. The default r is 32
[code: coordatt.py#L23]. The paper leaves the non-linearity unnamed. The code
uses BatchNorm followed by h-swish, and floors the bottleneck width at 8
[code: coordatt.py#L28] [code: coordatt.py#L31-L32]. Because the two gates
multiply, each output value is scaled by two numbers between 0 and 1. The paper
does not discuss how this changes the activation scale [arxiv:2103.02907 eq. 9].
Card:
[Coordinate Attention for Efficient Mobile Network Design](/papers/2103.02907).

On MobileNetV2, the authors report 72.3% ImageNet top-1 for the baseline, 73.5%
with SE and 74.3% with coordinate attention. The parameter counts are 3.5M,
3.89M and 3.95M [arxiv:2103.02907 tab. 1]. The coordinate-attention row uses
r = 32 and the SE row uses r = 24, so the comparison is not at matched r. At
r = 16 for coordinate attention and r = 12 for SE, the figures are 74.7% and
74.1% [arxiv:2103.02907 tab. 4]. On Pascal VOC detection the authors report
73.1 mAP, against 71.7 for both the baseline and SE [arxiv:2103.02907 tab. 7]. On
Cityscapes at output stride 8, they report 74.0 mIoU, against 72.2 for SE and
71.4 for the baseline [arxiv:2103.02907 tab. 9]. The latency column is from one
device, a Pixel 4, and the card records no variance for any of these numbers.

### Window-level attention inside a convolution, with channel-gate variants (RFAConv)

RFAConv replaces a 3×3 convolution. It expands each window into k² values per
channel, learns a softmax over the k² slots of each channel at each window
position, multiplies the two, rearranges the slots into a k×k block, and ends
with a stride-k convolution [arxiv:2304.03198 §3.3] [arxiv:2304.03198 eq. 3].
Its attention weights positions inside a window, not channels. The softmax runs
over the slots of each channel, not across channels [code:
classification/RFAresnet.py#L37]. The authors name the memory cost of the
expansion as the main cost of the method [arxiv:2304.03198 §4.6].

Two variants add channel attention to the expanded map. RFCBAMConv replaces
CBAM's channel module with squeeze-and-excitation, and applies channel and
spatial weights together. RFCAConv applies coordinate attention's directional
pooling to the expanded map [arxiv:2304.03198 §3.3] [code: model.py#L172-L195].
On ImageNet-1k with ResNet18, the baseline is 69.59% top-1. RFAConv gives 71.23%
[arxiv:2304.03198 tab. 3], and RFCBAMConv and RFCAConv give 72.15% and 72.01%
[arxiv:2304.03198 tab. 4]. On COCO2017 with YOLOv5n, the baseline is 27.5 AP,
RFAConv gives 29.0 and RFCAConv gives 29.5 [arxiv:2304.03198 tab. 7]. The paper
does not say whether the baselines were re-run or copied, and the card records no
variance. RFCAConv is 0.78 points above RFAConv on ResNet18, but that gap comes
from two different tables. The card does not show a run that isolates the
channel module inside the same setup. Card:
[RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](/papers/2304.03198).

### Dual attention in a decoder (camouflaged-object model)

The joint camouflaged-object model's fixation decoder reduces each backbone
stage to 32 channels and then applies dual residual attention, which the paper
takes from DANet: position attention and channel attention
[arxiv:2103.04011 §3.2] [fig. 3]. The code's decoders use RCAB channel-attention
blocks instead [code: joint_cod_col/model/ResNet_models.py#L170-L172]. The paper
does not state the channel module's layers. Its ablation (tab. 5) compares
single-task and joint models, not the attention block, so it gives no evidence
on the channel module alone [arxiv:2103.04011 tab. 5]. Card:
[Simultaneously Localize, Segment and Rank the Camouflaged Objects](/papers/2103.04011).

### Forms named in the survey (SE and CBAM)

The survey files channel attention under soft attention and gives two formulas.
The first is the SE gate: global pooling, a fully connected layer with ReLU, a
second fully connected layer and a sigmoid [arxiv:2204.07756 §2.1.1 eq. 1]. The
second is the CBAM channel gate. It pools with both max and average, passes both
descriptors through one shared MLP, sums them and applies a sigmoid
[arxiv:2204.07756 §2.1.1 eq. 2]. The survey also lists other channel-attention
methods (ECA, split attention, second-order and frequency-channel attention)
without formulas, except that second-order attention takes a power below one
half [arxiv:2204.07756 §2.1.1]. The survey runs no experiments of its own, so it
gives no evidence on which form performs better; the numbers it quotes come from
the original papers [arxiv:2204.07756 §1]. Card:
[Visual Attention Methods in Deep Learning: An In-Depth Survey](/papers/2204.07756).

### Channel gate in every nested node (DNANet)

DNA-Net finds single-frame infrared small targets with a U-shaped network whose
nodes are densely nested. Each node applies two 3×3 convolution stages with
batch norm, then a channel-then-spatial block (CSAM) [arxiv:2106.00487 §3.2.3].
The channel part is the CBAM form: a shared one-hidden-layer MLP over the global
max and global average of each channel, then a sigmoid [arxiv:2106.00487 eq. 3-4].
The paper gives no reduction ratio, but the code uses a hidden width of C/16
[code: model/model_DNANet.py#L33-L35]. The gate runs at every fusion point of
the nest, not on one stage's feature map. Card:
[Dense Nested Attention Network for Infrared Small Target Detection](/papers/2106.00487).

The paper's ablation removes the channel attention and keeps the spatial
attention, which isolates the channel gate in a segmentation network
[arxiv:2106.00487 tab. VI]. On NUDT-SIRST, IoU falls from 87.09 to 86.27, Pd from
98.73 to 96.96 and Fa rises from 4.223 to 4.881. On NUAA-SIRST, IoU falls from
77.47 to 76.20 and Fa rises from 2.353 to 12.69. These numbers are author-reported,
and the card records no seeds or variance for them.

### Channel-axis transformer in place of skip connections (U-shape Transformer)

The U-shape Transformer's CMSFFT is a transformer block that replaces the U-Net
skip connections. It builds queries for each of the four encoder scales. Keys
and values come from all four scales concatenated along the channel axis
[arxiv:2111.11843 §3.2.3] [arxiv:2111.11843 eq. 4]. The attention map is
channel by channel: it is instance-normalised and passed through a softmax, then
applied to the values [arxiv:2111.11843 eq. 5]. The heads are averaged, a
feed-forward block follows, and the step repeats four times [arxiv:2111.11843 eq.
6-7]. Nothing here produces a per-channel weight. The paper's text and equations
disagree on the orientation of the added query term, and the code adds the block
input instead [arxiv:2111.11843 eq. 6] [code: net/CMSFFT.py#L287-L290]. Card:
[U-shape Transformer for Underwater Image Enhancement](/papers/2111.11843).

The ablation adds CMSFFT to the pix2pix baseline. PSNR rises from 19.34 to 22.47
dB on Test-L400 and from 19.36 to 21.72 dB on Test-U90 [arxiv:2111.11843 tab. 6].
This measures a transformer block that replaces the skips, not channel
reweighting alone. The table has no row that removes CMSFFT from the full model,
and the card records no variance.

### Fusion gate in a depth decoder (HR-Depth)

HR-Depth's decoder has a grid of nodes. Some of its fusion steps are an fSE
block, which pools each channel over the whole map, passes the summary through
two fully connected layers with a sigmoid, multiplies the features by the
weights, and then fuses the channels with a 1×1 convolution [arxiv:2012.07356
§3.4] [fig. 3]. The paper sets the reduction ratio to 4 in all experiments
[arxiv:2012.07356 §3.4] [eq. 9]. The released code uses 16 [code:
layers.py#L309], and adds a ReLU between the two fully connected layers that the
paper does not describe [code: layers.py#L313-L317]. Card:
[HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation](/papers/2012.07356).

On KITTI at 640×192 with monocular training, Table 4 gives Abs rel 0.115 for
Monodepth2, 0.114 with fSE alone, 0.112 with dense skip connections, 0.110 with
dense skips and a row labelled "SE", and 0.109 with dense skips and fSE
[arxiv:2012.07356 tab. 4]. The card does not say what the "SE" row uses, so it
does not show whether the fusion step or the plain gate is responsible. The gaps
are 0.001 to 0.003 in Abs rel, and the card records no variance. The fSE row has
14.62M parameters and the SE row 16.23M [arxiv:2012.07356 tab. 4].

### Gate in the fusion step and in every residual block (ESRT)

ESRT's high preserving block ends with an SE-style channel gate after a 1×1
fusion convolution [arxiv:2108.11084 §3.1]. Its residual block (ARFB) is
described without a channel gate [arxiv:2108.11084 §3.2.1] [fig. 6]. The released
code adds one after the 1×1 convolution of every ARFB [code: model/esrt.py#L89-L110].
The code's reduction is 16, which gives 2 hidden channels at 32 input channels
[code: model/esrt.py#L25]. The paper does not state a ratio. Card:
[Transformer for Single Image Super-Resolution](/papers/2108.11084).

The ablation on Set5 ×4 (tab. 3) keeps the high-frequency branch and the ARFB,
and varies only the channel-attention column. With the gate it reaches 32.19 dB
at 751K parameters, and without it 32.08 dB at 724K [arxiv:2108.11084 tab. 3].
The table does not remove the gate from the ARFB blocks that the code adds, so
it does not measure that gate. The card records no variance for these numbers.

### Channel attention on CSP layers of a real-time detector (RTMDet)

RTMDet's backbone and neck use CSP-style layers. Each one can add a
channel-attention module on its concatenated features [code: csp.py#L232-L233]
[code: csp.py#L244-L245]. The card does not describe the module's layers, and
the card gives no reduction ratio for it.
Card: [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](/papers/2212.07784).

The change is part of the model-balance study. Reducing the second and third
stages from 9 to 6 blocks cut latency by about 20% and cost 0.5 AP, from 51.4 to
50.9 [arxiv:2212.07784 tab. 5] [§4.3]. Adding channel attention to the 3-6-6-3
model recovered 0.4 AP, to 51.3, at 2.40 ms against 2.11 ms without it, with
52.30M parameters against 50.92M [arxiv:2212.07784 tab. 5] [§4.3]. Latency is
FP16 with TensorRT, batch 1, on one NVIDIA 3090 [arxiv:2212.07784 §4.1]. The
ablations are single runs with no spreads reported [arxiv:2212.07784 tab. 5].

### Per-time-step gate on spiking membrane potential (Attention SNN)

Attention Spiking Neural Networks adds the CBAM channel module to conv-based
spiking layers. Its channel gate pools over space at each time step, gives one
weight per channel and step, and shares the MLP across steps
[arxiv:2209.13929 §3.3] [eq. 10-11]. The paper recommends applying it to the
integrated membrane potential, not to the convolution output, and its channel
equation uses the membrane input [arxiv:2209.13929 §6.1] [eq. 14]. The released
code places the gate on the batch-normalised convolution output, before the
spiking step [code: MA_SNN/module/LIF_Module.py#L212-L228], and its gate has no
membrane term [code: MA_SNN/module/Attention.py#L91-L97]. The paper does not
state the reduction ratio r_c [eq. 10]. The ImageNet residual model uses a ratio
of 8 in its channel-spatial block [code: Att_Res_SNN/models/resnet_2.py#L179-L206].

On DVS128 Gesture with the three-layer model, the vanilla network reaches
90.63 ± 0.58% [arxiv:2209.13929 tab. 1]. The channel gate alone reaches
93.88%, and the full temporal, channel and spatial combination reaches 96.53%
[arxiv:2209.13929 tab. 6]. The placement table gives the channel gate 91.84%
on the convolution output before the spiking step, 91.58% after it and 93.88%
on the membrane potential [arxiv:2209.13929 tab. 4]. On ImageNet-1K at one time
step, the channel gate adds 1.7 points to MS-Res-SNN-18 (61.70% to 63.42%),
3.8 to MS-Res-SNN-34 and 0.5 to MS-Res-SNN-104 [arxiv:2209.13929 tab. 3]. The
authors say the same module gives +0.7 on CBAM's CNN ResNet-34 against +5.0 on
the SNN [arxiv:2209.13929 §5.2.2]. Gesture rows are mean ± std over 10 runs, and
the ImageNet rows are single runs [arxiv:2209.13929 tab. 1]. Energy figures
come from an operation-count formula, not from hardware [arxiv:2209.13929 §4].
Card: [Attention Spiking Neural Networks](/papers/2209.13929).

### Per-stream gate after spatial alignment in a two-stream encoder (SwinNet)

SwinNet encodes the colour image and a second modality (depth or thermal) with
two Swin-B streams. At each level the two streams are aligned by a shared
spatial attention map. Each aligned stream then gets its own channel attention:
a global max pool, a 1×1 convolution and a sigmoid. Each channel weight
multiplies that stream's original, unaligned feature [arxiv:2204.05585 eq. 4-6].
The edge branch uses a channel-attention block with a residual connection
[arxiv:2204.05585 eq. 8]. The paper's alignment ablation replaces the whole
alignment module with BBS-Net's depth-enhanced module, which gives NLPR S-measure
0.936 against 0.941 for the full model [arxiv:2204.05585 tab. 4]. No row
removes the channel re-calibration alone, so the table does not isolate this
gate. The paper's text uses one 1×1 convolution, and the released code uses two
with reduction 16 and a ReLU between them [arxiv:2204.05585 eq. 5]
[code: models/Swin_Transformer.py#L799-L814]. Card:
[SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection](/papers/2204.05585).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Residual conv block where you want a cheap per-channel reweighting | SE-style gate after a conv bottleneck, as in MambaIR | The paper uses it in every residual block [arxiv:2402.15648 §3.3]. The hidden width is small (6 channels at C = 180) [code: basicsr/archs/mambair_arch.py#L41] |
| Deep residual denoiser that needs a cheap gate after each module, with a fixed reduction of 16 | Feature attention after each module, as in RIDNet | The paper's ablation adds the unit at 28.45 to 28.52 dB and removes it at 28.90 to 28.86 dB on BSD68 [arxiv:1904.07396 tab. 1]. The shrinkage in the paper and the ReLU in the code differ, so pick one and test it |
| Narrow feature maps (for example C = 60) | Check the squeeze factor before reusing 30 | At factor 30 and C = 60 the hidden layer has 2 channels [code: basicsr/archs/mambair_arch.py#L41]. The cards list no test of other factors |
| Two or more feature streams combined before a classifier | Gate over the concatenated features, as in the face forgery detector | The paper places it at the fusion point [arxiv:2103.12376 §4.3]. The ratio is not stated, so you must choose and test it |
| Decoder that predicts a map from multi-stage features and runs a second task on the same features | Dual attention (position and channel) in the decoder, as in the camouflaged-object model | The paper places DANet-style channel attention in each decoder [arxiv:2103.04011 §3.2]. Its code uses RCAB blocks instead [code: joint_cod_col/model/ResNet_models.py#L170-L172], so choose one form and test it. No ablation isolates the channel part |
| Iterative restoration where each stage refines an image estimate | Gate at the start of the stage's proximal network, as in DGUNet | The paper places the block there to extract shallow features [arxiv:2204.13348 §3.2]. No ablation removes it alone, so keep it and test it |
| The model already has batch norm, and you want per-channel weights with no extra layers | NAM's batch-norm scale weights [arxiv:2111.12419 §3] | The channel part adds 3,840 parameters for ResNet-50 [arxiv:2111.12419 app. A.1]. The reported error is the lowest of the compared modules, but the margin over CBAM is 0.35 points from one run [arxiv:2111.12419 tab. 1] |
| You want the penalty to drive channel selection or compression | NAM with the L1 term on the scales [arxiv:2111.12419 eq. 4] | The paper uses the scales as channel importance [arxiv:2111.12419 §3]. The cards report no pruning result, and the code leaves the penalty off by default |
| You need the spatial submodule, or a reliable margin over CBAM | Use SE or CBAM, then test | The spatial NAM has no released code, and the paper reports no variance [arxiv:2111.12419 §4] |
| You want the max-pooled descriptor as well as the average | CBAM-style gate [arxiv:2204.07756 §2.1.1 eq. 2] | The survey gives the formula but no measured comparison with SE. Test it against the SE form |
| Dense or nested U-shaped network for small-target segmentation, where each fusion mixes scales | Channel gate, then a spatial gate, at every node, as in DNANet | The paper places the gate in every node [arxiv:2106.00487 §3.2.3]. Removing the channel gate alone lowers IoU by 0.82 on NUDT-SIRST and 1.27 on NUAA-SIRST [arxiv:2106.00487 tab. VI]. The card records no variance, so test on your data |
| Multi-scale encoder-decoder where you can afford tens of millions of parameters and want features mixed across channels | Channel-axis attention in place of skips, as in the U-shape Transformer | The full model has 65.6 M parameters and 66.2 GFLOPs at 256 × 256, more than FUnIE and UIE-DAL [arxiv:2111.11843 tab. 3]. This is a block that replaces skips, not a gate, so the gate's cost does not apply to it |
| Mobile backbone or dense prediction where the gate should keep row and column position, at close to SE's cost | Coordinate attention, as in Coordinate Attention | The authors report 74.3% against 73.5% for SE on MobileNetV2, with 0.06 M more parameters than SE [arxiv:2103.02907 tab. 1]. The rows use different reduction ratios, and the latency is from one device |
| Decoder that fuses skip features at high input resolution, where the extra cost must stay small | Fusion gate in the decoder nodes, as in HR-Depth [arxiv:2012.07356 §3.4] | With dense skips, the table moves Abs rel from 0.112 to 0.109 with fSE on KITTI at 640×192 [arxiv:2012.07356 tab. 4]. The paper's reduction ratio (4) differs from the code's (16), so choose one and test it |
| Lightweight super-resolution network where a small gate may help, and the fusion step is a natural place for it | Gate in the fusion step, as in ESRT's high preserving block [arxiv:2108.11084 §3.1] | The ablation gives 32.19 dB with the gate and 32.08 dB without it on Set5 ×4, at 751K against 724K parameters [arxiv:2108.11084 tab. 3]. The code also adds a gate in every residual block, which this ablation does not test |
| Real-time detector where you cut blocks to save latency and want some accuracy back | Channel attention on the CSP layers, as in RTMDet [arxiv:2212.07784 tab. 5] | Adding it to the 3-6-6-3 model recovers 0.4 AP, at 2.40 ms against 2.11 ms without it [arxiv:2212.07784 tab. 5]. The ablations are single runs, so measure on your own hardware |
| Spiking network where the gate should act on the membrane potential | CBAM channel gate on the membrane (Activate-PRE), as in Attention SNN [arxiv:2209.13929 tab. 4] | On Gesture the placement gives 93.88% on the membrane against 91.84% and 91.58% on the convolution output [arxiv:2209.13929 tab. 4]. The code places it before the spiking step, so match the paper's placement if you rely on its numbers [code: MA_SNN/module/LIF_Module.py#L212-L228] |
| Two modality streams that must be aligned, with a gate on each stream after alignment | Per-stream channel re-calibration, as in SwinNet [arxiv:2204.05585 eq. 4-6] | The paper gives the gate and its placement, but no ablation of the gate alone. The paper's one-layer form differs from the code's two-layer form, so pick one and test it [code: models/Swin_Transformer.py#L799-L814] |
| Window-level weighting inside a 3×3 convolution, with memory to spare | RFAConv, or its RFCAConv variant, as in RFAConv [arxiv:2304.03198 §3.3] | RFAConv raises ResNet18 top-1 from 69.59% to 71.23% [arxiv:2304.03198 tab. 3]. The paper names memory as the main cost [arxiv:2304.03198 §4.6]. The card records no variance, so test the plain and channel-gate variants on your data |
| Small multi-resolution network (for example pose or segmentation) where channels should exchange information across resolutions at low cost | Conditional channel weighting, as in Lite-HRNet [arxiv:2104.06403 §3.2] | Lite-HRNet-18 gives 64.8 AP at 0.20 GFLOPs, against 55.2 AP at 0.54 GFLOPs for Small HRNet-W16. Both rows are trained without pretraining [arxiv:2104.06403 tab. 3]. The ratios are not stated in the paper, and the code's pose recipe differs from the paper in several places |
| Need evidence that channel attention helps your task | Run your own ablation with and without the gate | Five cited ablations isolate a gate: DNANet's table [arxiv:2106.00487 tab. VI], RIDNet's table 1 [arxiv:1904.07396 tab. 1], ESRT's table 3 [arxiv:2108.11084 tab. 3], RTMDet's table 5 [arxiv:2212.07784 tab. 5] and the Attention SNN channel-only rows against the vanilla model [arxiv:2209.13929 tab. 3] [tab. 6]. HR-Depth's table 4 varies its fusion gate, with one row unexplained [arxiv:2012.07356 tab. 4]. The other cited ablations do not isolate it [arxiv:2402.15648 tab. 1] [arxiv:2103.12376 tab. 3] [arxiv:2204.13348 tab. 6] [arxiv:2204.05585 tab. 4]. NAM's table compares whole modules against the plain network [arxiv:2111.12419 tab. 1] |

## Implementation sketch

Pseudocode in the same style as the paper cards. The first function follows the
MambaIR form. The CBAM comment shows the change to the pooling step. The
coordinate-attention function pools along each axis and gives row and column
gates. The NAM function uses batch-norm scales as weights, with the penalty
below it. The last function applies the gate to a fused two-stream input. The
face forgery card does not give that gate's structure, so the SE form is an
assumption, and the ratio is a parameter because the paper does not state it.

```pseudocode
# SE-style channel gate, MambaIR form  [arxiv:2402.15648 §3.3]
# Input:  x (B, C, H, W); output: x with each channel rescaled
# Squeeze factor r = 30 in the code  [code: basicsr/archs/mambair_arch.py#L41]
# RIDNet uses the same shape with r = 16 and a shrinkage in place of the ReLU
# (the code uses ReLU)  [arxiv:1904.07396 eq. 8]

def channel_attention(x, r):
    C = x.shape[1]
    # one summary value per channel
    s = avg_pool(x)                          # (B, C, 1, 1)
    # bottleneck C -> C/r -> C, with ReLU in between
    s = relu(conv1x1(s, out=C // r))
    s = conv1x1(s, out=C)
    # per-channel weights in (0, 1)
    w = sigmoid(s)
    return x * w                             # broadcast over H and W

# CBAM-style channel attention  [arxiv:2204.07756 §2.1.1 eq. 2]
# Replaces the avg_pool line above. Both pooled descriptors go through
# the same bottleneck, and the two outputs are summed before the sigmoid.
#   w = sigmoid(mlp(max_pool(x)) + mlp(avg_pool(x)))
# DNANet uses this form in every nested node, with hidden width C/16 in code
#  [arxiv:2106.00487 eq. 3-4] [code: model/model_DNANet.py#L33-L35]

# Coordinate attention: one gate per row and per column  [arxiv:2103.02907 §3.2]
# Input: x (B, C, H, W); r = 32 by default  [code: coordatt.py#L23]
# The paper leaves the non-linearity open; the code uses BatchNorm + h-swish
def coordinate_attention(x, r):
    C, H, W = x.shape[1], x.shape[2], x.shape[3]
    # pool along width (keep H) and along height (keep W)  [eq. 4-5]
    x_h = avg_pool(x, over=W)                # (B, C, H, 1)
    x_w = transpose(avg_pool(x, over=H))     # (B, C, W, 1)
    # one shared bottleneck over the joined strip  [eq. 6]
    y = concat([x_h, x_w], axis=2)           # (B, C, H + W, 1)
    y = act(bn(conv1x1(y, out=max(8, C // r))))
    y_h, y_w = split(y, [H, W], axis=2)
    # one 1x1 conv and sigmoid per direction  [eq. 7-8]
    g_h = sigmoid(conv1x1(y_h, out=C))                  # (B, C, H, 1)
    g_w = sigmoid(conv1x1(y_w, out=C))                  # (B, C, W, 1)
    # multiply by both gates at each position  [eq. 9]
    return x * g_h * transpose(g_w)          # (B, C, H, W)

# NAM channel gate, batch-norm scales as weights  [arxiv:2111.12419 §3] [eq. 2]
# Input: x (N, C, H, W); bn is a BatchNorm over C channels in the module
def nam_channel(x, bn):
    # normalise with the module's own batch norm  [eq. 1]
    y = bn(x)
    # channel weights: |gamma| divided by its sum in the module  [§3]
    g = abs(bn.weight)                       # (C,)
    w = g / sum(g)                           # no gradient flows through w
    # sigmoid of the weighted features is the mask  [eq. 2]
    m = sigmoid(y * w.view(1, C, 1, 1))
    # the code returns mask * x, not the mask alone  [code: MODELS/attention.py#L23]
    return m * x

# L1 penalty on the scales, added to the training loss  [eq. 4] [§4]
# p = 1e-4 for CIFAR-100, 1e-3 for ImageNet. The code adds p * sign(gamma)
# to the gradient instead  [code: train_cifar100.py#L55-L69]
loss = cross_entropy(model(x), target) + p * sum(abs(g) for g in nam_gammas)

# Gate over two concatenated streams  [arxiv:2103.12376 §4.3]
# Input:  rgb (B, C1, H, W); noise (B, C2, H, W); r not stated in the paper
def fusion_gate(rgb, noise, r):
    # concatenate the two streams along the channel axis
    fused = concat([rgb, noise], axis=channel)
    # assumption: the card names channel attention but not its structure,
    # so this reuses the SE-style gate above; choose and test r
    return channel_attention(fused, r)
```

HR-Depth's fSE block is the same gate applied to a fusion input: pool each
channel, two fully connected layers with a sigmoid, multiply, then a 1×1
convolution [arxiv:2012.07356 §3.4]. Its reduction is a parameter (the paper
uses 4, the code 16). The code also adds a ReLU between the two layers, which
the paper does not describe [code: layers.py#L313-L317].

## Contradictions

The cards disagree on what channel attention is. Most rescale channels with a
gate, but the U-shape Transformer's channel module attends across channels and
outputs a context map [arxiv:2111.11843 eq. 5-6]. Among the gates, the cards
place them in different parts of the network and measure different things. Five
ablations isolate a gate: DNANet's [arxiv:2106.00487 tab. VI], RIDNet's
[arxiv:1904.07396 tab. 1], ESRT's [arxiv:2108.11084 tab. 3], RTMDet's
[arxiv:2212.07784 tab. 5] and Attention SNN's channel-only rows
[arxiv:2209.13929 tab. 6]. The others do not (see the variants above).
HR-Depth's fusion-gate table has a row labelled "SE" that the card cannot
define [arxiv:2012.07356 tab. 4].
HR-Depth's paper gives a reduction of 4 and its code 16, and the code adds a ReLU
between the two fully connected layers [arxiv:2012.07356 §3.4] [eq. 9]
[code: layers.py#L309] [code: layers.py#L313-L317].
ESRT's paper describes a channel gate in the high preserving block only, and its
code adds one to every residual block as well [arxiv:2108.11084 §3.1]
[arxiv:2108.11084 §3.2.1] [code: model/esrt.py#L89-L110].
The camouflaged-object paper names DANet's channel attention for its decoders,
but its code uses RCAB blocks [arxiv:2103.04011 §3.2]
[code: joint_cod_col/model/ResNet_models.py#L170-L172].
RIDNet's paper and code differ on the activation: soft-shrinkage in the text,
ReLU in the code [arxiv:1904.07396 §3.2.1] [code: TestCode/code/model/ridnet.py#L10-L23].
The U-shape Transformer's paper and code differ on the residual in its channel
module and on the attention scale, which the paper gives as the square root of
C and the code as the square root of 480 [arxiv:2111.11843 eq. 5-6]
[code: net/CMSFFT.py#L171-L174] [code: net/CMSFFT.py#L287-L290]. The NAM paper and its code differ in
three places: the code takes the absolute value of each scale factor, the code
returns the mask times the input, and the paper's spatial submodule is absent
from the code [arxiv:2111.12419 §3] [code: MODELS/attention.py#L18]
[code: MODELS/attention.py#L23]. The NAM ImageNet row is described as channel and
spatial attention combined [arxiv:2111.12419 §4] [arxiv:2111.12419 tab. 2], but
the code has no ImageNet training script. The survey's own method counts disagree
between its abstract, introduction and conclusion [arxiv:2204.07756 §1]
[arxiv:2204.07756 §5]. It files harmonious attention under channel attention,
although that method mixes soft and hard attention [arxiv:2204.07756 §2.1.1].
Coordinate attention's paper leaves the non-linearity open, and its code uses
BatchNorm with h-swish [arxiv:2103.02907 §3.2.2] [code: coordatt.py#L31-L32].
Attention SNN's paper and code differ in three places. The paper pools the
channel gate over space at each time step [eq. 10], while the code pools over
time, height and width together, giving one gate per sample and channel
[code: MA_SNN/module/Attention.py#L49-L55]. The paper's channel equation takes
the membrane input [eq. 14], and the code's gate runs on the output of the
temporal gate, with no membrane term [code: MA_SNN/module/Attention.py#L91-L97].
The paper recommends the membrane potential as the location, and the code uses
the convolution output [arxiv:2209.13929 §6.1] [code: MA_SNN/module/LIF_Module.py#L212-L228].
SwinNet's paper gives one 1×1 convolution for its channel re-calibration [eq. 5],
and the code gives two with reduction 16 and a ReLU between them
[arxiv:2204.05585 eq. 5] [code: models/Swin_Transformer.py#L799-L814].

## Papers

- [MambaIR: A Simple Baseline for Image Restoration with State-Space Model](/papers/2402.15648): an SE-style gate in the local branch of each residual state-space block.
- [Generalizing Face Forgery Detection with High-frequency Features](/papers/2103.12376): channel-wise attention over the fused RGB and noise features before the classifier.
- [Deep Generalized Unfolding Networks for Image Restoration](/papers/2204.13348): a channel-attention block at the start of each stage's proximal mapping.
- [Dense Nested Attention Network for Infrared Small Target Detection](/papers/2106.00487): a CBAM-style channel gate, then a spatial gate, in every node of a densely nested U-shaped network.
- [Simultaneously Localize, Segment and Rank the Camouflaged Objects](/papers/2103.04011): DANet-style position and channel attention in the decoders of a joint localization and segmentation model.
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](/papers/2204.07756): a taxonomy that gives the SE and CBAM channel-attention formulas, with no experiments of its own.
- [NAM: Normalization-based Attention Module](/papers/2111.12419): a channel gate whose weights are batch-norm scale factors, with an L1 penalty on those factors and no extra layers for the channel part.
- [Real Image Denoising with Feature Attention](/papers/1904.07396): an SE-style feature attention unit after each of four enhancement attention modules in a single-stage denoiser.
- [Coordinate Attention for Efficient Mobile Network Design](/papers/2103.02907): pools along height and width to give one gate per row and per column, with a shared bottleneck.
- [RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](/papers/2304.03198): window-level attention inside a convolution, with SE and coordinate-attention channel variants (RFCBAMConv, RFCAConv).
- [U-shape Transformer for Underwater Image Enhancement](/papers/2111.11843): a channel-axis transformer (CMSFFT) in place of the skip connections, not a per-channel gate.
- [Lite-HRNet: A Lightweight High-Resolution Network](/papers/2104.06403): cross-resolution and spatial channel weights inside shuffle blocks of a small pose and segmentation network.
- [HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation](/papers/2012.07356): an SE-style fusion block in the decoder nodes of a self-supervised depth network.
- [Transformer for Single Image Super-Resolution](/papers/2108.11084): an SE-style gate in the fusion step of the high preserving block, plus a gate the code adds to every residual block.
- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](/papers/2212.07784): an optional channel-attention module on the CSP layers of a real-time detector.
- [SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection](/papers/2204.05585): a per-stream channel re-calibration after spatial alignment in a two-stream Swin encoder, plus a channel-attention block in the edge branch.
- [Attention Spiking Neural Networks](/papers/2209.13929): a CBAM-style channel gate per time step on the membrane potential of spiking layers, with temporal and spatial variants.