wikillm

Paper · 2023-03-28 · cs.CV

Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks

EAEF fuses RGB and thermal feature maps at each encoder scale. An attention-interaction branch gates both streams with one shared cross-modal weight, a complement branch gates them with its inverse, and a per-pixel softmax mixes the results. Author-reported: 58.9 mIoU on MFNet (+1.6 over MFTNet).

Multimodal fusionChannel attentionSemantic segmentationSalient object detectionrgb-thermalmultimodal-fusionsemantic-segmentationobject-detectionsalient-object-detectioncrowd-counting

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A colour camera struggles in the dark or in fog, while a thermal camera still sees heat from people, cars and road surfaces. The hard part is combining the two images inside one neural network. Earlier approaches mixed them at the input, stacked their features, or added attention to each image on its own. The authors argue these leave the network unsure how the two sources help each other.

Their idea is to handle three cases separately: both sources are informative, only one is, or neither is. Two small side paths cover these cases. One path boosts features where both sources are strong. The other leans on whichever source is still useful when the other is weak. The results are combined into one feature set per source. The block sits between the two image encoders at several depths, and the same design is used for segmentation, detection, salient-object detection and crowd counting.

TL;DR

  • EAEF is a fusion block placed between an RGB encoder and a thermal encoder at several scales §III.A. The segmentation code uses five EAEFNet.py.
  • The attention-interaction branch (AIB) gates both streams with one cross-modal gate. The complement branch (ACB) gates them with one minus that gate. A per-pixel softmax then weights the two streams §III.B.
  • Author-reported headline results: 58.9 mIoU on MFNet with a ResNet-152 backbone, 85.56 mIoU on PST900, 0.801 mAP@0.5 on M3FD, and MAE 0.031 on VT5000 tab. I tab. II tab. III tab. IV.
  • The official code departs from the equations in several places (learned scale, L2-normalised descriptors, two decoder heads with one joint loss, two-stream output in segmentation). Read "Paper vs code" before implementing.
  • The repository has no LICENSE file, so its code licence is not stated.

Problem

RGB-only perception degrades under poor lighting, noise and bad weather §I. Earlier RGB-T fusion methods either add the two modalities at the input, concatenate features, or apply attention to each modality. The authors call these strategies implicit and insufficient, because it is unclear how the two modalities complement each other §I fig. 1. They also argue that attention-based methods cannot guarantee strong features, because feature generation and attention extraction both happen implicitly inside the CNN §II.

The method assumes that RGB and thermal images of the same scene are spatially registered. The VT datasets are described as registered pairs §IV.C.1.

Method

Framework. The network has an RGB encoder, a thermal encoder and a decoder. EAEF fuses the two encoders' features at several scales §III.A fig. 2. All tasks share these components, but backbones and heads differ by task §III.A.

Descriptors. Each stream is reduced by global average pooling and an MLP to a per-channel vector, one for RGB and one for thermal §III.B eq. 1.

Conventional baseline. Adding sigmoid channel weights from each stream works only when one of the weights is already high §III.B.

Attention interaction branch (AIB). The element-wise product of the two descriptors is scaled by c and passed through a sigmoid. The same gate multiplies both feature maps channel-wise. The paper states that c is the number of channels and acts as a scaling factor §III.B eq. 2.

Attention complement branch (ACB). The complement of the same gate multiplies both feature maps. It is used when only one modality is informative, which the paper writes as the two descriptors having opposite signs §III.B eq. 3.

Interaction module. The AIB outputs are concatenated, passed through a depth-wise convolution, global max pooling, an MLP and a sigmoid, and the result multiplies the concatenation channel-wise. It is applied to the AIB only, because the authors found it adds little to the ACB §III.B eq. 4.

Merge. For each stream, the AIB output and the ACB output are summed §III.B eq. 5.

Spatial merge. A 1×1 convolution followed by a softmax weights the two merged streams. The paper writes the sum of the two weighted streams as the output §III.B eq. 6 eq. 7.

Key equations, transcribed in the paper's notation (⊛ is the paper's channel-wise product, written \oast in its source):

R = f_{MLP}\bigl(f_{GAP}(F_{rgb})\bigr), \quad T = f_{MLP}\bigl(f_{GAP}(F_{t})\bigr) eq. 1

F'_{rgb} = \sigma\bigl(c\,(R \otimes T)\bigr) \circledast F_{rgb}, \quad F'_{t} = \sigma\bigl(c\,(R \otimes T)\bigr) \circledast F_{t} eq. 2

\hat F_{rgb} = \bigl(1-\sigma(c\,(R \otimes T))\bigr) \circledast F_{rgb}, \quad \hat F_{t} = \bigl(1-\sigma(c\,(R \otimes T))\bigr) \circledast F_{t} eq. 3

\tilde F_{rgb,t} = F'_{rgb,t} \circledast \sigma\Bigl(f_{MLP}\bigl(f_{GMP}(f_{dw}(F'_{rgb,t}))\bigr)\Bigr) eq. 4

\bar F_{rgb} = \tilde F_{rgb} + \hat F_{rgb}, \quad \bar F_{t} = \tilde F_{t} + \hat F_{t} eq. 5

F^{*}_{rgb,t} = \bar F_{rgb,t} \otimes \mathrm{SoftMax}\bigl(\mathrm{Conv}_{1\times1}(\bar F_{rgb,t})\bigr) eq. 6

F_{final} = F^{*}_{rgb} + F^{*}_{t} eq. 7

Pseudocode for one EAEF block, following the paper. Each deviation the code makes is listed under "Paper vs code".

Pseudocode
# EAEF on one encoder scale  [§III.B]
# Input:  rgb_feat, thermal_feat  (B, C, H, W)
# Output: fused features          (B, C, H, W); see Paper vs code

def eaef(rgb_feat, thermal_feat):
    # Channel descriptor per stream: global average pool, then MLP  [eq. 1]
    rgb_vec = mlp(global_avg_pool(rgb_feat))          # (B, C)
    thermal_vec = mlp(global_avg_pool(thermal_feat))  # (B, C)

    # Shared cross-modal gate; c is the channel count  [eq. 2] [§III.B]
    cross_gate = sigmoid(c * (rgb_vec * thermal_vec))   # (B, C)
    complement_gate = 1 - cross_gate                    # (B, C)

    # AIB: both streams scaled by the same gate  [eq. 2]
    aib_rgb = channel_scale(cross_gate, rgb_feat)
    aib_thermal = channel_scale(cross_gate, thermal_feat)

    # ACB: both streams scaled by the complement gate  [eq. 3]
    acb_rgb = channel_scale(complement_gate, rgb_feat)
    acb_thermal = channel_scale(complement_gate, thermal_feat)

    # Interaction module on the AIB outputs only  [eq. 4]
    joint = concat_channels(aib_rgb, aib_thermal)       # (B, 2C, H, W)
    joint_gate = sigmoid(mlp(global_max_pool(depthwise_conv(joint))))
    aib_rgb, aib_thermal = split_channels(joint * joint_gate)

    # Sum AIB and ACB outputs per stream  [eq. 5]
    rgb_bar = aib_rgb + acb_rgb
    thermal_bar = aib_thermal + acb_thermal

    # Spatial weights: one 1x1 conv per stream, softmax across the two
    # maps. The paper gives no softmax axis; this follows the code.  [eq. 6]
    rgb_score = conv1x1_rgb(rgb_bar)                    # (B, 1, H, W)
    thermal_score = conv1x1_thermal(thermal_bar)        # (B, 1, H, W)
    rgb_weight, thermal_weight = softmax_pair(rgb_score, thermal_score)
    rgb_star = rgb_bar * rgb_weight
    thermal_star = thermal_bar * thermal_weight

    # Paper output: one sum of the two weighted streams  [eq. 7]
    return rgb_star + thermal_star

Using the block in a network. The paper says EAEF fuses features at multiple scales §III.A. The segmentation code places five EAEF blocks between the two ResNet encoders, one after each stage. The paper's own text giving five is commented out in the source EAEFNet.py EAEFNet.py. The segmentation decoder is a cascaded decoder based on BBSNet with three changes §IV.A.2. Object detection inserts the block into a YOLOv5 CSP-Darknet backbone §IV.B.2 Bi_csp_darknet.py. Crowd counting feeds the output of the last block to a small regression head §IV.D.2.

Hyperparameters

Semantic segmentation (MFNet and PST900)

NameValueWhereNotes
OptimizerSGD§IV.A.2
Learning rate0.02§IV.A.2Code default is also 0.02 train.py
Momentum0.9§IV.A.2train.py
Weight decay0.0005§IV.A.2train.py
LR scheduleExponentialLR, gamma not stated§IV.A.2Code gamma is 0.95, stepped once per epoch train.py train.py
Batch size5§IV.A.2Code default is 3 train.py
Epochs100 (MFNet), 60 (PST900)§IV.A.2Code defaults match: 100 train.py, 60 train.py
Loss0.5 Dice + 0.5 soft cross-entropy§IV.A.2Code uses smoothing 0.1 on soft CE, not in the paper train.py
Input size480 × 640 for MFNet§IV.A.1Loader reads full images without resizing MF_dataset.py
BackboneResNet-152 and ResNet-50§IV.A.3Code hard-codes 152 EAEFNet.py
Pretrainingnot statedCode loads ImageNet torchvision weights EAEFNet.py
Augmentationnot statedCode uses random horizontal flip (p = 0.5) and random crop (rate 0.1, p = 1) train.py
MFNet split784 train / 392 val / 420 test§IV.A.1Sums to 1596, not the 1569 pairs the paper states
PST900 split597 train / 297 test§IV.A.1No validation split stated

Object detection (M3FD, YOLOv5)

NameValueWhereNotes
DetectorYOLOv5§IV.B.2Code: CSP-Darknet with depth 0.67 and width 0.75 bi_yolov5.py
OptimizerSGD, learning rate 0.01, momentum 0.9, weight decay 0.0005§IV.B.2Code: momentum 0.937 with Nesterov bi_yolov5.py
Batch size32§IV.B.2Code: 4 per GPU bi_yolov5.py
LR scheduleExponentialLR§IV.B.2Code: linear schedule, factor 0.01 bi_yolov5.py
Epochsnot statedCode sets 300 in the scheduler and 400 in the train loop bi_yolov5.py bi_yolov5.py
LossIoU loss 0.3, cross-entropy 0.7§IV.B.2Code: class CE 0.3, CIoU box 0.05, objectness CE 0.7 bi_yolov5.py
Initialisationnot statedCode starts from COCO-pretrained YOLOv5-m bi_yolov5.py
MetricmAP@0.5§IV.B.2Follows TarDAL's protocol

Salient object detection (VT821, VT1000, VT5000)

NameValueWhereNotes
Input size224 × 224§IV.C.2Code default 224 config.py
OptimizerAdam§IV.C.2train.py
Learning rate0.001§IV.C.2Code default 1e-4 config.py
Batch size8§IV.C.2Code default 16 config.py
Epochs20§IV.C.2Code default 50 config.py
LR decaynot statedCode: ×0.1 every 40 epochs config.py
Augmentationrandom flips, rotations, clipping, as in LSNet§IV.C.2
Training data2,500 pairs from VT5000§IV.C.1Remaining VT5000 pairs plus VT821 and VT1000 are for testing
Lossnot statedCode adds a boundary loss to the BCE + IoU terms train.py
Pretrainingnot statedCode loads ImageNet torchvision weights EAEFNet_50.py

Crowd counting (RGBT-CC)

NameValueWhereNotes
BackboneBL network on VGG16§IV.D.2Code uses a five-block VGG-style network. Its ImageNet loading was not found in the shipped code EAEFNet.py
OptimizerAdam§IV.D.2regression_trainer.py
Learning rate0.00001§IV.D.2train.sh
Epochs300, evaluated every 10§IV.D.2Code defaults 300 train.py
Weight decaynot statedCode default 1e-4 train.py
Crop sizenot statedCode default 256 train.py
Batch sizenot statedCode default 1 train.py
DecoderMLP of two 1×1 convolutions§IV.D.2Code: 3×3, 3×3, then 1×1, with 2× upsampling and absolute value EAEFNet.py
Split1,545 train / 300 val / 1,200 test§IV.D.1Sums to 3,045, not the 2,030 pairs the paper states
MetricsRMSE and GAME(0–3)§IV.D.2

Results

Author-reported throughout. Each row uses the paper's setup and baseline. No run-to-run variance is reported. Cross-task numbers are not comparable because each task uses its own dataset and metric.

Semantic segmentation, MFNet tab. I (mAcc and mIoU, in %)

SettingMetricBaselineThis paperSource
ResNet-152 backbonemAcc / mIoUMFTNet: 74.7 / 57.375.1 / 58.9tab. I
ResNet-50 backbonemAcc / mIoUGMNet (ResNet-50): 74.1 / 57.373.2 / 55.9tab. I

Semantic segmentation, PST900 tab. II (mAcc and mIoU, in %)

SettingMetricBaselineThis paperSource
Single backbone (not stated)mAcc / mIoUGMNet: 89.61 / 84.1291.42 / 85.56tab. II

Object detection, M3FD tab. III

SettingMetricBaselineThis paperSource
YOLOv5 with EAEFmAP@0.5TarDAL: 0.7780.801tab. III

Salient object detection tab. IV (S-measure, adpE, adpF, MAE)

SettingMetricBaselineThis paperSource
VT821S / adpE / adpF / MAECMDBIF-Net: 0.882 / 0.923 / 0.855 / 0.0320.885 / 0.927 / 0.846 / 0.031tab. IV
VT1000S / adpE / adpF / MAECMDBIF-Net: 0.927 / 0.952 / 0.914 / 0.0190.926 / 0.964 / 0.905 / 0.017tab. IV
VT5000S / adpE / adpF / MAECMDBIF-Net: 0.886 / 0.933 / 0.868 / 0.0320.885 / 0.934 / 0.853 / 0.031tab. IV

Crowd counting, RGBT-CC tab. V (RMSE, lower is better)

SettingMetricBaselineThis paperSource
No pretrainingRMSETAFNet: 22.4521.85tab. V
ImageNet-pretrained VGG16 (marked *)RMSEDEFNet*: 21.0919.39tab. V

Ablation, MFNet tab. VI (mAcc / mIoU, in %)

SettingMetricBaselineThis paperSource
Neither branch (baseline)mAcc / mIoU—71.7 / 56.5tab. VI
AIB onlymAcc / mIoU71.7 / 56.572.5 / 57.1tab. VI
ACB onlymAcc / mIoU71.7 / 56.574.3 / 57.7tab. VI
AIB and ACB (EAEF)mAcc / mIoU71.7 / 56.575.1 / 58.9tab. VI

Computational cost is reported in tab. I as FLOPs and parameters for each method. The units are not stated, and the input size used for the counts is not stated. In that table the ResNet-152 variant shows 147.3 and 200.4, and MFTNet shows 330.6 and 360.9.

Paper vs code

The official repository is at commit a8306549c83d716cea6422fe94e64489353e8deb. It has four task folders: segmentation (MFNet and PST900), detection, salient object detection and crowd counting. Code comparisons are for the segmentation folder unless noted.

  • Output of the block. Paper eq. 7 returns a single sum of the two weighted streams eq. 7. The segmentation code returns a concatenated pair. Its RGB output is a weighted sum of both streams and its thermal output is the thermal stream scaled by its own weight EAEFNet.py. The crowd-counting code follows the paper: it scales each stream separately and then sums them EAEFNet.py.
  • Scale c. The paper states c is the channel count, with no statement about learning it §III.B eq. 2. The segmentation code makes it a learnable parameter initialised to the channel count EAEFNet.py. The crowd-counting code uses the channel count as a constant EAEFNet.py.
  • Descriptor normalisation. The paper does not mention normalisation. Both codes L2-normalise the pooled descriptors before the cross product EAEFNet.py EAEFNet.py.
  • Interaction module. The paper leaves the kernel size, reduction ratio and activation unstated eq. 4. The segmentation code uses a 7×7 depth-wise convolution EAEFNet.py, a channel-attention reduction ratio of 16 EAEFNet.py and GELU EAEFNet.py.
  • Spatial softmax. Eq. 6 applies one 1×1 convolution to the concatenated features and gives no softmax axis eq. 6. The code uses one 1×1 convolution per stream and a softmax across the two maps EAEFNet.py. The crowd code does the same EAEFNet.py.
  • Two outputs. The paper describes one loss on one output §IV.A.2. The segmentation model returns two heads EAEFNet.py. Training sums the same joint loss over both heads train.py.
  • Which head is reported. The paper does not say. The test loop reports the sum of the two heads train.py. Separate functions evaluate each head alone train.py train.py.
  • ResNet-50 variant. The paper reports a ResNet-50 model tab. I. The shipped segmentation code hard-codes ResNet-152 EAEFNet.py. The 50-layer branch is unreachable as shipped.
  • Dropout. The code defines a Monte Carlo dropout module at rate 0.09 but never calls it in the forward pass EAEFNet.py EAEFNet.py EAEFNet.py.
  • Batch size and model selection. The paper uses batch size 5 and selects the best model on the validation set §IV.A.2. The code's default batch size is 3 train.py. It saves a checkpoint and reports test metrics every epoch, and it does not select a best checkpoint train.py.
  • Augmentation and mixed precision. The paper says nothing about augmentation or mixed precision. The code uses flips, random cropping train.py and automatic mixed precision train.py train.py.
  • Detection loss. The paper weights an IoU loss 0.3 and cross-entropy 0.7 §IV.B.2. The code gives cross-entropy on the class term weight 0.3, CIoU on the box term weight 0.05 and cross-entropy on objectness weight 0.7 bi_yolov5.py.
  • Salient detection loss. The paper states no loss beyond the training settings §IV.C.2. The code adds a boundary term to BCE and IoU train.py. The paper also uses 20 epochs with batch 8 and learning rate 0.001 §IV.C.2. The code defaults are 50 epochs, batch 16 and learning rate 1e-4 config.py.
  • Crowd decoder. The paper says the decoder is an MLP of two 1×1 convolutions §IV.D.2. The code uses two 3×3 convolutions and a 1×1 convolution, then a 2× upsample and an absolute value EAEFNet.py.
  • Licence. The README badge says MIT README.md. The repository contains no LICENSE file, so code_license is recorded as not stated.

Pitfalls

  • Dataset counts do not add up. MFNet's stated 1,569 pairs do not match its 784 / 392 / 420 split, which sums to 1,596 §IV.A.1. RGBT-CC's stated 2,030 pairs do not match its 1,545 / 300 / 1,200 split, which sums to 3,045 §IV.D.1. The shipped code does not include the split lists, so the discrepancy cannot be resolved from the repository.
  • Claims that the tables do not support. The text says the method is best on all salient-detection metrics §IV.C.3. Table IV shows CMDBIF-Net higher on S-measure for VT5000 (0.886 vs 0.885) and on adpF for VT821 (0.855 vs 0.846) and VT1000 (0.914 vs 0.905) tab. IV. On RGBT-CC without pretraining, the method's GAME(0–3) values are worse than TAFNet's tab. V. The text does not mention this.
  • ResNet-50 segmentation trails GMNet. The ResNet-50 variant scores 73.2 mAcc and 55.9 mIoU. GMNet with ResNet-50 scores 74.1 and 57.3 tab. I. The text reports only the ResNet-152 comparison.
  • Abstract and table disagree on crowd counting. The abstract gives an 8.1% gain for crowd counting and names MAE as the metric. The table has no MAE column. The 8% figure in the text is the relative RMSE reduction against DEFNet with pretraining §IV.D.3 tab. V.
  • Units of reported gains. The abstract's 1.6% (mIoU), 2.3% (mAP) and 3.1% (MAE on VT5000) are absolute points for the first two. The 3.1% is relative, computed from 0.032 to 0.031. The 2.7% and 8% in the crowd text are relative RMSE reductions. Check each figure's basis before comparing it with other papers.
  • Equation 4 typo. The text says the thermal output is split off as "F̃_rgb" §III.B, which is a typo for F̃_t.
  • Softmax axis not stated. The paper gives no axis for the softmax in eq. 6. The code applies it across the two modality maps per pixel EAEFNet.py. Other readings produce different weights.
  • Pretraining is not stated for segmentation. The paper does not say whether the encoders start from ImageNet. The code uses torchvision ImageNet weights, and the thermal first convolution is initialised from the mean of the RGB first-layer weights EAEFNet.py.
  • Which head produces the reported numbers. Two heads are trained. Test code reports their sum, and separate functions report each one train.py. Pick the head that matches the reported figures before comparing.
  • Model selection. The paper says it uses the best validation model. The code does not implement that choice. The reported test numbers therefore depend on which checkpoint you pick.
  • Single runs. No variance is reported. The code seeds training with 3407 by default, but cuDNN determinism is turned off train.py train.py.
  • Datasets come from third-party hosts. The README links MFNet, PST900, M3FD and RGBT-CC through external file hosts README.md. The paper's CC licence covers the paper, not these datasets. Check each dataset's terms before redistributing data.
  • Repository licence. The repository has no LICENSE file. Treat its code as unlicensed for reuse until the authors confirm terms. This card paraphrases the method and quotes no code.
  • Hidden costs. The block adds a depth-wise convolution, a global max pool and an MLP at every scale, and the ACB is computed alongside the AIB. The cost is not separated from the backbone in the reported FLOPs tab. I.
  • Hardware line is commented out. The paper's GPU description is in a LaTeX comment and is not published. Do not cite it.

Lineage

  • Builds on: BBSNet's cascaded decoder for segmentation, with an SE-layer and ASPP change §IV.A.2. The data split follows RTFNet §IV.A.1. The crowd setup follows IADM §IV.D.2. The SOD augmentation follows LSNet §IV.C.2. The detector is YOLOv5 §IV.B.2. Earlier attention-based RGB-T methods are reviewed in §II.
  • Compared against: segmentation on MFNet with MFNet, FuseNet, PSTNet, RTFNet, FuseSeg, ABMDRNet, GMNet, FEANet, MFTNet and EGFNet tab. I; PST900 with Efficient FCN, CCNet, ACNet, SA-Gate, RTFNet, PSTNet and GMNet tab. II; detection with YOLOv5, U2F and TarDAL tab. III; salient detection with MTMR, M3S-NIR, SGDL, FMCF, ADF, MIDD, ECFFNet, LSNet and CMDBIF-Net tab. IV; crowd counting with UcNet, HDFNet, BBSNet, MVMS, IADM, CSCA, TAFNet and DEFNet tab. V.
  • Superseded by: not known from wiki content.

Source and license

Adapted from “Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks” by Mingjian Liang, Junjie Hu, Chenyu Bao, Hua Feng, Fuqin Deng, Tin Lun Lam (arXiv:2303.15710), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/FreeformRobotics/EAEFNet at commit a8306549c83d (not stated (no LICENSE file in repository)). Any quoted code is shown under that license.