Dynamic High-frequency Convolution for Infrared Small Target Detection

In plain words

Infrared pictures can hide a tiny bright target, but they also contain many other sharp bright spots and edges, such as bright corners and broken clouds. Detectors have to tell the target apart from these look-alikes. Most networks learn one fixed set of small filters and use it everywhere in the picture.

This paper changes one building block of such networks, the ordinary convolution, so that its filters adapt to each spot. At every location, a small learned step looks at the nearby pixels and writes out its own bank of filters for that spot. The values in those filters are kept inside a range centred on zero, which suits the sharp brightness changes the detector needs to pick up. The filtered result is added to the original pixels, and then an ordinary convolution runs as usual.

The new block is a drop-in replacement, so it can be placed into existing detectors without redesigning them. According to the authors, detection improves across the networks they tried, at a small cost in speed.

TL;DR

• DHiF is a convolution whose k x k filters are generated per location from the local input, bounded to (-1, 1) by tanh §II-B.3 eq. 5 eq. 6.
• The generated filters are applied to the local patch, and the sum of the patch and its filtered version feeds a standard convolution §II-B.3 eq. 7 eq. 8.
• Author-reported: in the encoders of seven SIRST networks on IRSTD-1K and NUAA-SIRST, DHiF beats the standard convolution on IoU in every case tab. 1.
• The paper's motivation for the zero-centred range relies on Fourier-property arguments §II-B.2 eq. 3 eq. 4; the code enforces the range only, not the zero-sum property the argument mentions.
• Code exists and is linked from the paper. Its normalisation of the generator input and its layer placement differ from the paper's description (see Paper vs code).

Problem

Single-frame infrared small target (SIRST) detection must separate tiny, locally salient targets from clutter. Both targets and clutter (bright corners, broken clouds, other fragments) are high-frequency components of the image, and they share similar local contrast §I. Traditional methods use hand-made, fixed-parameter operators such as local contrast measures and top-hat transforms, which do not adapt well across scenes §I. Learned detectors fit features from annotated data, but the paper argues they do not explicitly model or discriminate the different high-frequency categories inside local regions §I §II-A. The method assumes the detector is built from convolutional layers: DHiF is designed for networks with convolutional structures §IV [supp. §IV].

Method

DHiF keeps the usual sliding window of a convolution and adds a second, local path. For each window it (a) normalises the local input, (b) generates a bank of k^2 filters, one per tap, each of length k^2, via a linear projection followed by tanh, (c) applies that bank to the local patch of every channel, (d) adds the result to the original patch, and (e) runs a standard convolution over the sum §II-B.1 §II-B.3.

The filter-sensitivity argument is as follows. The paper notes that a derivative in the spatial domain corresponds to multiplying by jω in the Fourier domain eq. 3, and that a constant (DC) offset adds an impulse at zero frequency eq. 4. From this it concludes that filter kernels should contain positive and negative values, with a sum close to zero, to respond to high frequencies rather than to flat regions §II-B.2. The paper implements this by restricting each kernel entry to an interval centred at zero §II-B.2.

The DHiF operation, as written in the paper:

1. Local feature vector. For each window, flatten the normalised single-channel local map to length k^2. The paper describes this as a normalisation layer applied to the input features; the method of normalisation is not specified §II-B.3.
2. Filter generation. A linear projection maps the k^2 values to k^2 vectors of length k^2. Each entry is passed through tanh §II-B.3 eq. 5 eq. 6.

\boldsymbol{v}_1, \dots, \boldsymbol{v}_{k^2} = \rho\big(\mathrm{vec}(\boldsymbol{F}^{local}_{norm})\big) eq. 5

\boldsymbol{W}_{f_1}, \dots, \boldsymbol{W}_{f_{k^2}} = \theta(\boldsymbol{v}_1, \dots, \boldsymbol{v}_{k^2}), \quad \theta = \tanh eq. 6

3. Filtering. Flatten the C-channel local input to C x k^2. Multiply it by the k^2 x k^2 filter matrix W_f. The same bank is applied to every channel §II-B.3 eq. 7.

\boldsymbol{F}^{local}_{filt} = \mathrm{vec}(\boldsymbol{F}^{local}_{in}) \, \boldsymbol{W}_{\boldsymbol{f}} eq. 7

4. Output. Add the flattened input to the filtered features, multiply element-wise by the weights of each of the C' output filters, and sum. Each output filter has shape C x k^2 §II-B.3 eq. 8.

\boldsymbol{F}^{local}_{out}[j] = \sum_{1}^{k^2} \Big( \big( \mathrm{vec}(\boldsymbol{F}^{local}_{in}) + \boldsymbol{F}^{local}_{filt} \big) \odot \boldsymbol{W}_j \Big) eq. 8

Equation 8 writes the sum over k^2 only; the weights W_j have shape C x k^2, so the sum also runs over channels (see Paper vs code).

Repeating these steps for every window gives the output map §II-B.3. The paper calls the block a drop-in replacement for standard convolution in SIRST networks §II-B.3 §III-B. In the residual block, DHiF replaces the first convolution, and the DHiF-Res block is defined as follows [supp. §II-B] [supp. eq. 1].

\mathrm{Res}_{DHiF}(\boldsymbol{X}) = \sigma\Big( \kappa(\boldsymbol{X}) + \gamma\big( \sigma( DHiF(\boldsymbol{X}) ) \big) \Big) [supp. eq. 1]

Here DHiF(·) includes its batch-norm layer, γ is a standard convolution followed by batch norm, σ is the activation and κ the shortcut [supp. §II-B].

The DHiF-Res block in pseudocode, from the supplementary equation:

# DHiF-Res block  [supp. eq. 1] [supp. §II-B]
# Input and output: features (B, C, H, W)

def dhif_res_block(inputs, dhif_layer, conv, bn_out):
    # Shortcut kappa; the paper leaves its form open (see Pitfalls)
    shortcut = shortcut_path(inputs)
    # DHiF, then batch norm, then activation  [supp. eq. 1]
    hidden = relu(dhif_layer(inputs))
    # Standard convolution, then batch norm  [supp. eq. 1]
    hidden = bn_out(conv(hidden))
    # Add the shortcut and apply the activation  [supp. eq. 1]
    return relu(shortcut + hidden)

The DHiF forward pass in pseudocode, from eq. 5 to eq. 8:

# DHiF forward pass for one input map  [§II-B.3] [eq. 5]-[eq. 8]
# Input:  features (B, C, H, W); kernel size k
# Output: features (B, C_out, H, W)

def dhif(features, k, linear, conv_weight):
    # Gather the k x k neighbourhood of every location  [§II-B.3]
    patches = unfold(features, k)              # (B, C, H, W, k*k)
    # Reduce the channels to one normalised value per tap  [§II-B.3]
    # (paper: normalisation layer; code: L2 norm, see Paper vs code)
    local = channel_norm(patches)              # (B, H, W, k*k)
    # Project to k*k filter vectors of length k*k  [eq. 5]
    projected = linear(local)                  # (B, H, W, k*k, k*k)
    # Bound every filter entry to (-1, 1) with tanh  [eq. 6] [§II-B.2]
    filters = tanh(projected)                  # (B, H, W, k*k, k*k)
    # Apply each location's filter bank to every channel  [eq. 7]
    filtered = apply_bank(patches, filters)    # (B, C, H, W, k*k)
    # Add the local input to the filtered response  [eq. 8]
    combined = patches + filtered              # (B, C, H, W, k*k)
    # Standard weights over channels and taps  [eq. 8]
    return weighted_sum(combined, conv_weight)  # (B, C_out, H, W)

Hyperparameters

Values from the paper and its supplementary material, with the official code where the paper is silent. Code values are facts about the repository at code_commit, not statements by the authors.

Results

Author-reported. Setup: IRSTD-1K and NUAA-SIRST, seven SIRST networks split into CNN (FC3Net, DNANet, ISNet, MSHNet) and CNN-Transformer hybrid (MTU-Net, SCTransNet, APTNet) §III-B tab. 1. Each row is the same network with the encoder convolutions replaced. The table below gives the standard convolution and DHiF rows; the other replacements (CDC, WTConv, PConv) are in tab. 1. The paper does not state the number of runs or any variance.

Source: tab. 1. Units: IoU, nIoU, P_d in 10^-2; F_a in 10^-5 §III-A.

Competing replacements, as the paper reports them tab. 1: CDC, WTConv and PConv each lower IoU below standard in some networks. On NUAA-SIRST the paper says all other replacements, except DHiF, cause severe drops in most networks §III-B. PConv in SCTransNet fails to converge (IoU 0.04 on IRSTD-1K) and is reported with a footnote tab. 1.

Placement of DHiF in the DNANet encoder, on NUAA-SIRST tab. 2:

Kernel size on NUAA-SIRST [supp. tab. II]:

Non-linearity on NUAA-SIRST [supp. tab. III]:

Block structure on NUAA-SIRST [supp. tab. I]:

Cost, DNANet at 512 x 512 [supp. tab. IV]:

Salt-and-pepper noise, DNANet variants [supp. tab. V]:

Qualitative results and feature visualisations are shown in fig. 3 and in the supplement [supp. fig. 3] [supp. fig. 4]. The paper says the visualisation shows DHiF producing negative responses inside bright-clutter regions and positive responses along edges §III-C.2 fig. 3.

Paper vs code

Searched: the paper's own text and abstract link https://github.com/TinaLRJ/DHiF (the owner name matches the first author's homepage, linked in the author biography of the paper). Owner is the first author, so it qualifies as official. Cloned at commit 06862c844bc16a3b5ee0d0019e5ae24f0f1bb4e7. The repository has no LICENSE file, and its README is a single title line README.md. Nothing was quoted from the code. The code of DHiF is in model/basic.py, and the networks that use it are under model/.

• Generator input normalisation. Paper: a normalisation layer on the single-channel input map, then flatten and project §II-B.3 eq. 5. Code: no normalisation layer in DHiF. The generator input is the L2 norm over channels of each k x k patch basic.py. Results depend on this choice, and the paper does not name its normalisation.
• Output sum. Paper: sum over k^2 only eq. 8. Code: the weighted sum runs over channels and taps basic.py. The weight W_j has shape C x k^2, so the code is consistent with the shape the paper gives.
• Generator structure. Paper: linear projection then tanh eq. 5 eq. 6. Code: one nn.Linear(k*k, (k*k)*(k*k)) followed by Tanh basic.py. The reshape to k^2 x k^2 matches the paper. The code has a bias; the paper does not mention one.
• Shortcut and attention in the DNANet block. Paper's block equation has a shortcut and no attention [supp. eq. 1]. Code: the DNANet DHiF block applies channel attention and spatial attention (CBAM) after the second batch norm, before the residual addition model_DNANet.py. The baseline DNANet block uses the same attention model_DNANet.py. The shortcut is a 1x1 convolution plus batch norm when shapes change model_DNANet.py; the paper leaves κ unspecified.
• Layer placement in DNANet. Paper: replacements in encoder levels 2 to 5 for the table; levels 2 and 3 in the main experiments §III-C.1 §III-C.2 tab. 2. Code: DHiF is used only in conv1_0 and conv2_0, i.e. encoder levels 2 and 3 model_DNANet.py.
• Input-layer use in ISNet. Paper: DHiF at the input layer provides no gain and reduces P_d; the paper advises replacing encoder hidden layers §III-C.1. Code: the ISNet stem convolution becomes DHiF when conv == 'DHiF' model_ISNet.py. The paper does not say whether the ISNet rows of tab. 1 used this stem.
• Learning-rate schedule. Paper: decay by 0.5 after every 50 epochs [supp. §III-B]. Code: MultiStepLR with milestones 100, 150, ..., 500 and gamma 0.5 train.py. The first decay is at epoch 100, not 50.
• Initialisation. Paper: Kaiming initialisation for the convolution layers [supp. §III-B]. Code: DHiF's weight uses PyTorch's kaiming_uniform_ with a = sqrt(5) basic.py, which is the PyTorch default for Conv2d and not the He initialisation the paper cites. The helper weights_init_kaiming exists in utils.py utils.py, but a search of the training path found no call to it.
• Loss and deep supervision. Paper: no loss stated. Code: soft IoU loss loss.py net.py. DNANet has deep supervision on by default and returns four outputs in training model_DNANet.py model_DNANet.py. The paper does not mention either.
• Augmentation. Paper: no augmentation stated. Code: random crop, then random flips and transpose with probability 0.5 each dataset.py dataset.py.
• Checkpoint and threshold. Paper: not stated which checkpoint is reported. Code: saves a checkpoint at the final epoch train.py and tests with threshold 0.5 train.py.
• Dataset name. Text: "IRSDT-1k". Tables and code: "IRSTD-1K" §III-A utils.py.
• Compute cost. Supplement: "introduces only 0.01 MB new parameters". Table: the parameter counts are in M, and the difference is 0.01 M [supp. tab. IV]. The unit in the text is wrong.

No discrepancies found in the DHiF filter equations beyond those listed above: the generator, the tanh bound and the filtering step match eq. 5-eq. 7 basic.py basic.py.

Pitfalls

• Normalisation is unspecified. The paper names a normalisation layer for the generator input but does not define it. The code uses an L2 norm over channels with no learned or batch normalisation basic.py. An implementer must choose; the code's choice is the only one tested.
• Zero-centred range is a bound, not a sum constraint. The argument for sensitivity to high frequencies asks for kernels with a sum close to zero §II-B.2. The implementation only bounds each entry to (-1, 1) with tanh eq. 6. Nothing enforces zero sum or symmetry. The supplement's non-linearity ablation shows that removing tanh hurts [supp. tab. III], but it does not test the zero-sum property directly.
• Compute and memory per pixel. Each location builds a k^2 x k^2 filter bank regardless of channel count basic.py, so filter tensors scale with H x W x k^4. The unfold step materialises a k^2 copy of the feature map basic.py. The paper reports 8.61 to 8.02 FPS for DNANet (6.85% slower) [supp. tab. IV], but nothing on peak memory.
• Level-ablation table is ambiguous. tab. 2 lists one row per level, but the caption and text do not say whether other levels were also replaced. The main configuration (levels 2 and 3 §III-C.2) gives 77.68 IoU in tab. 1 and [supp. tab. II], which matches no single row of tab. 2. The text's "2nd to 5th" claim is an average across rows §III-C.1.
• Input layer is not recommended. The paper finds that DHiF on the input layer reduces P_d §III-C.1, but the ISNet code puts DHiF in the stem model_ISNet.py. Do not copy the ISNet stem without a test.
• Sum notation in equation 8. The sum is written over k^2 only, but the weights have shape C x k^2. An implementer following the equation literally would sum over the wrong axis; sum over channels and taps, as the code does basic.py.
• Outliers and failures. FC3Net with standard convolution on NUAA-SIRST has IoU 31.09 and F_a 30.19, far below the other networks tab. 1; the paper gives no explanation. PConv in SCTransNet does not converge tab. 1.
• Single numbers with no variance. The paper does not say how many training runs back each number, or whether the reported checkpoint was chosen on the test set (NUAA-SIRST has train and test splits only [supp. §III-B]). Differences of 1 to 2 IoU points between rows should be read with that in mind.
• Attention is part of the block in code. An implementation that follows [supp. eq. 1] alone may not reproduce the code's DNANet numbers; the code was not run here, so this is a risk, not a measured gap. The CBAM attention is in both the standard and DHiF blocks model_DNANet.py.
• Units in the complexity text. "0.01 MB new parameters" in the supplement should read 0.01 M parameters [supp. tab. IV].

Lineage

• Builds on: the residual block [he2016deep] as the place where DHiF is inserted, and DNANet [li2022dense] as the main test network. The paper gives no arXiv ids for either. Also related: fixed local-contrast and pixel-difference operators such as ALCNet and PDC, which the paper contrasts with its learned, per-location filters [supp. §I-B].
• Compared against: CDC (MDIGCNet, IEEE JSTARS 2024), WTConv (wavelet convolutions, ECCV 2024) and PConv (pinwheel-shaped convolution, AAAI 2025) tab. 1. None has an arXiv id in the paper's bibliography, so none is linked here.
• Superseded by: not known.