NAM: Normalization-based Attention Module

In plain words

An attention module helps a network focus on the parts of its internal features that matter. Most attention modules add small extra layers that learn how much to weight each channel, which costs parameters and computation.

NAM skips the extra layers. Batch normalization already learns one scale value per channel, and the authors treat a larger scale as a sign that the channel carries more useful variation. NAM uses those existing scale values as the channel weights. A penalty added to the training loss pushes weak scale values toward zero, so the weak channels get suppressed.

The authors report similar or better accuracy than other attention modules, with far fewer extra parameters for the channel part. The paper also describes a spatial version. The released code has only the channel version.

TL;DR

NAM is a lightweight attention module for convolutional networks. It places an extra batch norm in each block, takes the normalized scale values as channel weights, and gates the features with a sigmoid. A sparsity penalty on those scale values is added to the loss. Author-reported on CIFAR-100 with ResNet-50, the channel-only variant has the lowest top-1 error of the compared modules (19.09%). The repository implements only the channel part, and it has no license file.

Problem

Attention modules such as SE, BAM, CBAM and TAM add fully connected layers, convolutions or rotations of the feature maps to learn channel or spatial weights. The authors argue these designs ignore what the trained weights already say about importance §1 §2. They also state that recognizing less salient features is key for model compression [abstract]. NAM tries to reuse the batch-norm scale factors instead, which avoids the added fully connected and convolutional layers §1.

Method

NAM keeps the module placement of CBAM and redesigns the two submodules §3. It is placed at the end of each network block. In a residual network it goes at the end of the residual structure, before the skip connection is added §3 model_resnet.py model_resnet.py.

Channel submodule. A batch norm normalizes the block features eq. 1. Its scale factors gamma measure how much each channel varies, and the paper uses them as channel importance. The channel weight is the scale factor divided by the sum of all scale factors of the module §3. The channel mask is a sigmoid of the weighted, normalized features eq. 2. The paper's formula gives the mask; the code multiplies the mask with the module input (see Pitfalls).

Spatial submodule. The paper applies a batch-norm scaling factor along the spatial dimensions as well, calling it "pixel normalization". Its spatial mask is a sigmoid of the spatially weighted features, with weights lambda_i / sum_j lambda_j §3 eq. 3. The paper does not define the spatial batch norm in more detail. The released code does not contain this submodule.

Loss. The training loss adds an L1 penalty on the channel scale factors gamma and on the spatial scale factors lambda, balanced by one coefficient p §3 eq. 4.

# NAM channel submodule  [§3] [eq. 1] [eq. 2]
# Input:  features (N, C, H, W); channel_bn is a batch norm over C channels
# Output: gated features (N, C, H, W)

def nam_channel(features, channel_bn):
    # Normalise with the module's own batch norm  [eq. 1]
    normalised = channel_bn(features)            # (N, C, H, W)
    # Channel weights from scale factors  [§3] [code: MODELS/attention.py#L18]
    scales = abs(channel_bn.weight)              # (C,)
    weights = scales / sum(scales)               # (C,)
    # Weight each channel of the normalised features  [eq. 2]
    scores = normalised * weights.view(1, -1, 1, 1)  # (N, C, H, W)
    # Sigmoid gate  [eq. 2] [code: MODELS/attention.py#L23]
    mask = sigmoid(scores)                       # (N, C, H, W)
    # Apply the gate to the module input  [code: MODELS/attention.py#L23]
    return mask * features

# Sparsity penalty on channel scale factors  [eq. 4] [§4]
# Called after loss.backward(), before optimizer.step()
# p is 0.0001 for CIFAR-100 and 0.001 for ImageNet  [§4]

def add_sparsity_gradient(nam_layers, p):
    for layer in nam_layers:
        # Subgradient of the L1 term p * |gamma|  [eq. 4]
        # The code adds it to .grad  [code: train_cifar100.py#L55-L69]
        gamma = layer.nam.channel_bn.weight
        gamma.grad = gamma.grad + p * sign(gamma)

Parameter count app. A.1: the channel part needs one scale per channel, and the spatial part needs one scale per spatial position (H x W, with the paper assuming a 32 x 32 input for its count).

Hyperparameters

Paper values come from §4. Code values are defaults in the training script train_cifar100.py, which covers CIFAR-100 only. Values the paper defers to CBAM are not in this paper, so they are marked "not stated" unless the code gives them.

Results

Author-reported. The paper does not state the number of seeds or any variance. Table 1 is CIFAR-100 with ResNet-50, and Table 2 is ImageNet with MobileNet V2 §4.

CIFAR-100, ResNet-50 tab. 1

* ch: channel attention only; sp: spatial attention only tab. 1.

ImageNet, MobileNet V2 tab. 2

The text describes the ImageNet NAM row as channel and spatial attention combined §4. A channel-only ImageNet row appears only as a commented-out line in the source tab. 2.

Parameter overhead of the attention modules, ResNet-50 app. A.1. The appendix compares only the attention overhead, not the full model. Channel part: CBAM 696,320 parameters vs NAM 3,840. Spatial part: CBAM 392 vs NAM 1,360 tab. 3 tab. 4. The appendix says NAM has fewer parameters overall, because the channel saving is much larger than the spatial increase.

Paper vs code

The official repository is Christian-lyc/NAM (commit at code_commit), linked from the abstract. It has no LICENSE file, so code_license is recorded as none and no code is quoted verbatim. It contains MODELS/attention.py (NAM), MODELS/cbam.py, MODELS/bam.py, MODELS/model_resnet.py and train_cifar100.py. The README is only the title.

• No spatial submodule. The paper defines a spatial NAM §3 eq. 3. The code has only Channel_Att, and Att wraps just that. The no_spatial argument is accepted but not used attention.py model_resnet.py. The spatial result (NAM sp*) and the combined ImageNet result tab. 1 tab. 2 cannot be reproduced from this code.
• Absolute values in the weights. The paper writes gamma_i / sum_j gamma_j §3. The code uses the absolute value of each scale factor [code: MODELS/attention.py#L18].
• Output of the module. Eq. 2 defines the mask M_c, and the text says M_c is the output §3 eq. 2. The code returns the sigmoid mask multiplied by the module input attention.py.
• Separate batch norm. The code adds a dedicated BatchNorm inside the module, on top of the block's own BN attention.py model_resnet.py. The paper's Eq. 1 is generic about which batch norm is used §3.
• Penalty scale. Eq. 4 writes the data term as a sum over samples eq. 4. The code uses the batch-mean cross-entropy and adds p * sign(gamma) to the gradient train_cifar100.py train_cifar100.py train_cifar100.py. The relative weight of the penalty therefore differs from the literal equation. The paper does not say which convention its runs used.
• Penalty off by default. The sparsity term runs only with --sparsity-regularization train_cifar100.py. The paper does not say whether the reported NAM runs used it, although it gives p for CIFAR-100 §4.
• Normalization constants. The CIFAR-100 run uses CIFAR-10 mean and std, because data_transforms_cifar10 is called for both datasets [code: train_cifar100.py#L127]. A CIFAR-100 function exists but is not used train_cifar100.py.
• Missing ImageNet code. The repository has no MobileNet V2 model and no ImageNet training script. The ResNet builder has an ImageNet branch model_resnet.py, but the training script builds only CIFAR-100 models train_cifar100.py. The ImageNet results tab. 2 are therefore not reproducible from this repository.
• Unused arguments. Channel_Att takes t=16, and Att takes shape, but neither is used attention.py model_resnet.py.

Pitfalls

• The mask is not the output. Eq. 2 gives only the sigmoid mask. An implementer who returns the mask alone will get a different network. The code multiplies the mask with the input attention.py.
• Initial scale. The code normalizes the scale factors by their sum, so each channel weight starts near 1/C, with the batch-norm gamma initialized to 1 model_resnet.py. Inferred from the code, not measured: the weighted scores then start close to zero, and the sigmoid mask starts near 0.5 for every channel.
• Division by a near-zero sum. The weights divide by the sum of the absolute scale factors with no epsilon attention.py. If the scale factors shrink toward zero under the penalty, this division becomes numerically sensitive. The paper does not discuss it.
• No gradient through the weights. The weights read .data, so the normalization is treated as a constant in backpropagation [code: MODELS/attention.py#L18].
• Penalty targets only the NAM scale factors. The code adds the penalty to the NAM batch-norm weight of each block, not to the block's own batch norms train_cifar100.py. Weight decay of 5e-4 also acts on these scale factors train_cifar100.py.
• Sign at zero. The penalty gradient uses torch.sign, which is 0 at exactly zero train_cifar100.py. The paper does not state how zero is handled.
• Small margin, one run. The headline channel result is 0.35 points better top-1 error than CBAM (19.09 vs 19.44) tab. 1. No variance, number of seeds or significance test is reported, so the gap may be within run-to-run noise. The paper does not say how many runs were averaged.
• Spatial cost grows with resolution. The spatial scale count is H x W per block app. A.1, so the overhead depends on the input resolution, which the paper assumes is 32 x 32 for its count.
• Table and figure numbers are assigned in source order. The appendix channel table is labelled table:4 in the source and renders as Table 3, and the spatial one renders as Table 4. Figure labels follow the same pattern: the channel figure (fig:2 in source) is Figure 1 and the spatial figure is Figure 2 fig. 1 fig. 2. These numbers come from the LaTeX order; the compiled PDF was not checked.
• Appendix claims. The appendix counts are simple arithmetic on the listed layer sizes app. A.1 tab. 3 tab. 4. They are not measured parameter counts. The model totals in tab. 1 and tab. 2 are the authors' own counts.
• Hidden costs. The paper reports FLOPs and parameters only. It does not report training time or memory overhead for the extra batch norm.

Lineage

• Builds on: batch normalization and its scale factors §3, with the L1 penalty on scale factors taken from network slimming (Liu et al., 2017, cited as liu2017learning) §1 §3.
• Compared against: SE, BAM, CBAM and TAM, on the same ResNet and MobileNet setups §4 tab. 1 tab. 2.
• Superseded by: not stated.