Paper · 2020-11-14 · cs.CV
Duality-Gated Mutual Condition Network for RGBT Tracking
DMCNet tracks one target in paired RGB and thermal video. Each modality's features scale and shift the other's, and sigmoid gates suppress noisy conditions. When the tracker is unsure, optical flow checks for sudden camera motion and re-samples candidates. Author-reported: 83.9% PR and 59.3% SR on RGBT234; EAO 0.3796 on VOT-RGBT2019, below mfDiMP's 0.3879.
Visual object trackingMultimodal fusionOptical flowrgbt-trackingmultimodalfeature-modulationoptical-flow
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Tracking means following one object through a video, starting from its box in the first frame. This paper tracks in video from two cameras at once: an ordinary colour camera and a thermal (heat) camera. Each camera can help or mislead depending on conditions, and one of them can be unreliable at any moment.
The method lets the two streams guide each other. Features from one camera are used to scale and shift features from the other. Small learned gates decide how much of each signal gets through, so noisy parts are pushed down.
Cameras also sometimes jerk suddenly, and the target can slide out of the search window. The tracker then checks for camera motion with optical flow, but only when it is unsure of its own answer. If it finds a sudden motion, it samples new candidate boxes in the direction opposite to that motion.
TL;DR
- DMCNet is a single-object RGB-thermal (RGBT) tracker. Its two streams condition each other through gated scale-and-shift operations.
- A re-sampling step uses DIS optical flow to recover from sudden camera motion.
- Author-reported: 0.839 PR and 0.593 SR on RGBT234 (trained on GTOT); 0.909 PR and 0.733 SR on GTOT (trained on RGBT234); EAO 0.3796 on VOT-RGBT2019.
- The only repository found has no model or training code (README, LICENSE and a results archive), so nothing can be checked against code. See Paper vs code.
- Many details are unstated or ambiguous (learning-rate groups, gate inputs, sampling layout, flow units). See Pitfalls.
Problem
RGBT trackers must handle cases where one modality is unreliable, such as total darkness, bad weather or thermal crossover §I. The authors argue that existing RGBT methods either weight modalities or fuse features in fixed ways, and so do not use the useful parts of low-quality modalities well §I §II.A.
Abrupt camera motion is common in RGBT footage. Then the search window often misses the target. Widening the search or searching globally adds background and computation §I §III.C.
The method assumes aligned RGB and thermal frame pairs, and an initial ground-truth box in the first frame §I.
Method
Notation. Section numbers follow the paper's IEEE roman numerals, written
with a dot: [§III.B] is Section III-B. Equation, table, figure and algorithm
numbers are those of the compiled arXiv version, v3: equations eq. 1–eq. 7,
tables tab. I–tab. VI, figures fig. 1–fig. 9 and alg. 1.
Backbone. The network takes candidate patches cropped from aligned RGB and thermal frames, resized to 107×107 §III.A. A shared trunk is the first three convolutional layers of VGG-M, with 7×7×96, 5×5×256 and 3×3×512 kernels. Its first two layers use ReLU, local response normalisation and max pooling; the third uses ReLU only §III.A. The binary classifier is three fully connected layers with 512, 512 and 2 outputs, trained with multi-domain learning §III.A. Each modality also has its own adapter, built from convolutions with 3×3×96, 1×1×256 and 1×1×512 kernels across three levels, with ReLU, batch normalisation, dropout and max pooling §III.A. The trunk plus adapters form a modality-shared and modality-specific design, as in MANet §III.A.
Mutual condition module. The module runs in both directions at once: RGB to thermal and thermal to RGB §III.B. For the scaling condition, the multi-scale branch takes the adapter output of the source modality eq. 1. Its first level uses four parallel convolutions (1×1, 3×3, a 3×3 dilated convolution with rate 2, and 5×5); the middle and high levels use a 1×1, 3×3 and 1×1 stack §III.B fig. 4.
The thermal stream is modulated by the RGB multi-scale feature eq. 2:
f^{out}_{T} = f_T \odot f^{ms}_{R} + f^{scaled}_{T2R} eq. 2
where the shifting term comes from the other direction eq. 3:
f^{scaled}_{T2R} = f_R \odot G_3(f^{ms}_{T}) eq. 3
The multi-scale branch is defined for RGB only eq. 1:
f^{ms}_{R} = W^{f} \ast [\, f_R \ast W^{ms} \,] eq. 1
Each gate has the same form, a 1×1 convolution followed by a sigmoid eq. 4:
G = \sigma(\mathrm{Conv}(f)) eq. 4
With gates in place, the RGB-to-thermal output is eq. 5:
f^{out}_{T} = f_T \odot G_1(f^{ms}_{R}) + G_2(f^{scaled}_{T2R}) eq. 5
and the thermal-to-RGB output is the mirror image eq. 6:
f^{out}_{R} = f_R \odot G_3(f^{ms}_{T}) + G_4(f^{scaled}_{R2T}) eq. 6
The paper defines the cross term f^scaled_T2R (eq. 3) but does not write out f^scaled_R2T. It says the construction is similar; see Pitfalls for the interpretation used here. The authors state that the gates suppress noise in the conditions and in the propagated features, and that this adds no extra memory §I §III.B fig. 3.
# Mutual condition module, one adapter level [eq. 1-6] [§III.B]
# Input: rgb_feats, thermal_feats (same shape, (B, C, H, W))
# Output: rgb_out, thermal_out (same shape)
def mutual_condition(rgb_feats, thermal_feats):
# Multi-scale branches, one per modality [eq. 1]
rgb_ms = multi_scale_rgb(rgb_feats)
thermal_ms = multi_scale_thermal(thermal_feats)
# Gate G3 on the thermal multi-scale feature [eq. 3] [eq. 4]
gated_thermal_ms = sigmoid(conv_g3(thermal_ms))
# Shifting condition from thermal to RGB [eq. 3]
t2r = rgb_feats * gated_thermal_ms
# Gate G1 on the RGB multi-scale feature [eq. 4] [eq. 5]
gated_rgb_ms = sigmoid(conv_g1(rgb_ms))
# RGB-to-thermal scaling term; inferred by symmetry [§III.B]
r2t = thermal_feats * gated_rgb_ms
# Second gates on the shifting terms [eq. 5] [eq. 6]
thermal_out = r2t + sigmoid(conv_g2(t2r))
rgb_out = t2r + sigmoid(conv_g4(r2t))
return rgb_out, thermal_outRe-sampling module. It runs only when the tracker is unreliable §III.C. (1) The tracker's top score is below 0; the authors take this, following MDNet, as the sign of an unreliable state §III.C. (2) DIS optical flow gives the mean displacement [dx, dy] over a region centred on the previous target and three times the size of its box §III.C. (3) Camera motion is flagged if |dx| or |dy| exceeds the threshold u §III.C. (4) The target is taken to move in the direction opposite to the camera §III.C. (5) Sixteen candidates are drawn along that direction, with a step of one quarter of the box width horizontally or one quarter of its height vertically §III.C. (6) They are scored, and the result is combined with the Gaussian candidates §III.C fig. 5.
Training. The backbone starts from MDNet's pretrained parameters; the modality-specific adapters and mutual-condition sub-networks start from random initialisation §III.D. The optimiser is SGD, for 200 epochs, with softmax cross-entropy over the two classes §III.D. Each iteration uses 8 frames chosen at random from the training set. For each frame, anchors are drawn uniformly around the ground truth; those with IoU above 0.7 are positives and those with IoU below 0.5 are negatives. Per frame, 32 positives and 96 negatives are kept §III.D. For evaluation on GTOT the model is trained on RGBT234; for RGBT234 and RGBT210 it is trained on GTOT, because those two sets overlap §III.D.
Online tracking. alg. 1 §IV
- Initialisation: the last fully connected layer is replaced by a new, sequence-specific layer. Convolutions are frozen. The three fully connected layers are fine-tuned for 50 iterations on 500 positives and 5000 negatives from the first frame §IV.
- A bounding-box regressor is trained on the first frame only §IV.
- Each frame: draw Gaussian candidates around the previous state, with the covariance diag(0.09 r², 0.09 r², 0.25), where r is the mean of the target's width and height §IV. Score the candidates; the estimate is the mean of the five highest positive scores eq. 7.
- If the mean of those five scores is above 0, refine the estimate with the regressor and update the sample sets alg. 1.
- Otherwise, compute optical flow. If |dx| or |dy| exceeds u, re-sample and keep the re-sampled result if its top score is higher alg. 1 §IV.
- If no sudden motion is found, do a short-term update only alg. 1.
- Long-term update every 10 frames alg. 1.
X^{*}_{m} = \arg\max_{i=1,\dots,256} f^{+}(X^{i}_{t}), \qquad X^{*}_{t} = \mathrm{mean}(X^{*}_{m}) eq. 7
# Online tracking loop [alg. 1] [§IV] [eq. 7]
# Input: frames, initial_box, threshold u (= 5)
# Output: one box per frame
def track(frames, initial_box, u):
# Fine-tune the last three FC layers on the first frame [§IV]
net = fine_tune_fc_layers(frames[0], initial_box)
# Bounding-box regressor, trained once on the first frame [§IV]
regressor = train_regressor(frames[0], initial_box)
short_term, long_term = init_sample_sets(net, frames[0])
prev_box = initial_box
boxes = [initial_box]
for t in range(1, len(frames)):
# Gaussian candidates around the previous estimate [§IV]
candidates = sample_gaussian(prev_box)
pos_scores = net.positive_scores(frames[t], candidates)
# Mean of the five best candidates; F(X_t) [eq. 7]
top = top_k_indices(pos_scores, k=5)
box = mean(candidates[top])
score = mean(pos_scores[top])
if score > 0:
# Reliable: refine with regressor, update sample sets [alg. 1]
box = regressor(box)
update(short_term, long_term)
else:
# Unreliable: estimate camera motion with DIS flow [§III.C]
dx, dy = mean_flow(frames[t - 1], frames[t], prev_box)
if abs(dx) > u or abs(dy) > u:
# Re-sample 16 boxes against the camera motion [§III.C]
resampled = resample_opposite(prev_box, dx, dy, n=16)
rscore, rbox = best_score(net, frames[t], resampled)
if rscore > score:
box = rbox
else:
# No camera motion: short-term update only [alg. 1]
update(short_term)
if t % 10 == 0:
# Long-term update every 10 frames [alg. 1]
update(long_term)
boxes.append(box)
prev_box = box
return boxesHyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Input patch size | 107×107 | §III.A | candidate patches, RGB and thermal |
| Trunk convolutions | 7×7×96, 5×5×256, 3×3×512 | §III.A | first three conv layers of VGG-M |
| Trunk pooling and LRN | not stated | §III.A | LRN and max pooling after the first two layers; window and stride not stated |
| Classifier FC widths | 512, 512, 2 | §III.A | three FC layers, binary output |
| Adapter convolutions | 3×3×96, 1×1×256, 1×1×512 | §III.A | per modality; ReLU, batch norm, dropout, max pooling |
| Adapter dropout rate | not stated | §III.A | |
| MSConv, first level | 1×1, 3×3, 3×3 dilated (rate 2), 5×5 | §III.B | parallel branches |
| MSConv, middle and high levels | 1×1, 3×3, 1×1 | §III.B | in sequence |
| Gate | 1×1 conv + sigmoid | §III.B eq. 4 | output channels and gate input not stated |
| Training epochs | 200 | §III.D | |
| Optimiser | SGD | §III.D | momentum, weight decay, schedule not stated |
| Loss | softmax cross-entropy | §III.D | two classes |
| LR, backbone, first two FC, binary layer | 0.001 | §III.D | |
| LR, modality-specific and mutual-condition sub-networks | 0.002 | §III.D | |
| Frames per iteration | 8 | §III.D | chosen at random |
| Samples per frame | 32 positive, 96 negative | §III.D | 1024 samples per iteration (derived: 8 × 128) |
| Training IoU thresholds | positive > 0.7; negative < 0.5 | §III.D | anchors drawn uniformly around ground truth |
| Pairing of training and test sets | GTOT test: train on RGBT234; RGBT234 and RGBT210 test: train on GTOT | §III.D | |
| First-frame fine-tuning | 50 iterations; 500 positive, 5000 negative | §IV | |
| LR, first-frame fine-tuning | 0.005 (last layer); 0.0005 (other two FC) | §IV | |
| Positive and negative IoU, first frame | > 0.7 and < 0.5 | §IV | |
| Per-frame update samples | 50 positive (IoU > 0.7); 200 negative (IoU < 0.3) | §IV | short- and long-term |
| LR, per-frame update | 0.01 (last layer); 0.001 (other two FC) | §IV | |
| Long-term update interval | every 10 frames | alg. 1 | |
| Gaussian covariance | diag(0.09 r², 0.09 r², 0.25) | §IV | r = mean of target width and height |
| Candidates scored per frame | 256 | eq. 7 | from the index range of eq. 7; not stated in text |
| Estimate | mean of top 5 candidates | §IV eq. 7 | |
| Unreliable test | top score < 0 | §III.C alg. 1 | |
| Camera-motion threshold u | 5 | §IV §V.F | units not stated |
| Flow region | centred on previous target, 3× box size | §III.C | |
| Re-sampled candidates | 16 | §III.C | |
| Re-sampling step | ¼ of box width (horizontal); ¼ of box height (vertical) | §III.C | |
| Bounding-box regression | trained on first frame only | §IV | |
| PR threshold | 5 px (GTOT); 20 px (RGBT234) | §V.A | |
| Software and hardware | PyTorch 1.0+; 8 NVIDIA GTX 2080Ti | §V |
Results
Author-reported throughout. Setup: four benchmarks, PR and SR with the threshold above, and VOT metrics (EAO, A, R) for VOT-RGBT2019 §V.A. GTOT has 50 pairs (about 15K frames), RGBT210 has 210 pairs (210K frames), RGBT234 has 234 pairs (about 200K frames), and VOT-RGBT2019 has 60 sequences (over 20K frames) §V.A. The text says whether baseline numbers were rerun or copied only for VOT-RGBT2019, where reported numbers are used §V.E.
| Dataset | Metric | Baseline (author-reported) | DMCNet | Source |
|---|---|---|---|---|
| RGBT234 | PR / SR (%) | CMPP 82.3 / 57.5; CAT 80.4 / 56.1; JMMAC 79.0 / 57.3 | 83.9 / 59.3 | tab. I §V.D |
| GTOT | PR / SR | CMPP reported higher performance (values not in the text); DMCNet "comparable" in SR | 0.909 / 0.733 | tab. V tab. VI §V.B |
| RGBT210 | PR / SR | +1.1 PR over mfDiMP; +2.2 SR over CAT (values in the figure) | not stated in text | §V.C fig. 6 |
| VOT-RGBT2019 | EAO / A / R | mfDiMP 0.3879 / 0.6019 / 0.8036; FSRPN 0.3553 / 0.6362 / 0.7069 | 0.3796 / 0.6009 / 0.7088 | tab. II §V.E |
Per-attribute results on RGBT234 (PR / SR, %). Abbreviations: NO no occlusion, PO partial occlusion, HO heavy occlusion, LI low illumination, LR low resolution, TC thermal crossover, DEF deformation, FM fast motion, SV scale variation, MB motion blur, CM camera motion, BC background clutter tab. I. The table header reads "JAMMC"; the text and bibliography use JMMAC.
| Tracker | NO | PO | HO | LI | LR | TC | DEF | FM | SV | MB | CM | BC | ALL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MDNet+RGBT | 86.2 / 61.1 | 76.1 / 51.8 | 61.9 / 42.1 | 67.0 / 45.5 | 67.0 / 45.5 | 75.6 / 51.7 | 66.9 / 47.3 | 58.6 / 36.3 | 73.5 / 50.5 | 65.4 / 46.3 | 64.0 / 45.4 | 64.4 / 43.2 | 72.2 / 49.5 |
| SGT | 87.7 / 55.5 | 77.9 / 51.3 | 59.2 / 39.4 | 70.5 / 46.2 | 72.5 / 46.2 | 76.0 / 47.0 | 68.5 / 47.4 | 67.7 / 40.2 | 69.0 / 43.4 | 64.7 / 43.6 | 66.7 / 45.2 | 65.8 / 41.8 | 72.0 / 47.2 |
| CMR | 89.5 / 61.6 | 77.7 / 53.6 | 56.3 / 37.7 | 74.2 / 49.8 | 72.0 / 47.6 | 67.5 / 44.3 | 66.7 / 47.3 | 61.3 / 38.4 | 71.0 / 49.3 | 60.0 / 42.7 | 62.9 / 44.7 | 63.1 / 39.8 | 71.1 / 48.6 |
| DAPNet | 90.0 / 64.4 | 82.1 / 57.4 | 66.0 / 45.7 | 77.5 / 53.0 | 75.9 / 51.5 | 76.8 / 54.3 | 71.7 / 57.8 | 67.0 / 44.3 | 78.0 / 54.2 | 65.3 / 46.7 | 66.8 / 47.4 | 71.7 / 48.4 | 76.6 / 53.7 |
| MANet | 88.7 / 64.6 | 81.6 / 56.6 | 68.9 / 46.5 | 76.9 / 51.3 | 70.8 / 48.7 | 75.4 / 54.3 | 72.0 / 52.4 | 69.4 / 44.9 | 77.7 / 54.2 | 72.6 / 51.6 | 71.9 / 50.8 | 73.9 / 48.6 | 77.7 / 53.9 |
| MaCNet | 92.7 / 66.5 | 81.1 / 57.2 | 70.9 / 48.8 | 77.7 / 52.7 | 75.1 / 47.6 | 77.0 / 56.3 | 73.1 / 51.4 | 69.4 / 44.9 | 78.7 / 56.1 | 71.6 / 52.5 | 71.7 / 51.7 | 77.8 / 50.1 | 79.0 / 55.4 |
| FANet | 88.2 / 65.7 | 86.6 / 60.2 | 66.5 / 45.8 | 80.3 / 54.8 | 75.0 / 51.0 | 76.6 / 54.9 | 72.2 / 52.6 | 68.1 / 43.6 | 78.5 / 56.3 | 70.0 / 50.3 | 72.4 / 52.3 | 75.7 / 50.2 | 78.7 / 55.3 |
| CMPP | 95.6 / 67.8 | 85.5 / 60.1 | 73.2 / 50.3 | 86.2 / 58.4 | 86.5 / 57.1 | 83.5 / 58.3 | 75.0 / 54.1 | 78.6 / 50.8 | 81.5 / 57.2 | 75.4 / 54.1 | 75.6 / 54.1 | 83.2 / 53.8 | 82.3 / 57.5 |
| JAMMC | 93.2 / 69.4 | 84.1 / 61.1 | 67.7 / 48.3 | 84.0 / 58.8 | 77.1 / 51.7 | 74.9 / 52.6 | 70.6 / 52.9 | 61.0 / 41.7 | 83.7 / 61.6 | 75.1 / 54.9 | 76.2 / 55.6 | 68.7 / 48.5 | 79.0 / 57.3 |
| CAT | 93.2 / 66.8 | 85.1 / 59.3 | 70.0 / 48.0 | 81.0 / 54.7 | 82.0 / 53.9 | 82.0 / 53.9 | 76.2 / 54.1 | 73.1 / 47.0 | 79.7 / 56.6 | 68.3 / 49.0 | 75.2 / 52.7 | 81.1 / 51.9 | 80.4 / 56.1 |
| DMCNet | 92.3 / 67.1 | 89.5 / 63.1 | 74.5 / 52.1 | 85.3 / 58.7 | 85.4 / 57.9 | 87.2 / 61.2 | 77.9 / 56.5 | 80.0 / 52.4 | 84.6 / 59.8 | 77.3 / 55.9 | 80.1 / 57.6 | 83.8 / 55.9 | 83.9 / 59.3 |
The paper's colour ranking marks (best, second, third) are omitted here tab. I.
VOT-RGBT2019 comparison tab. II:
| Metric | GESBTT | CISRDCF | MPAT | FSRPN | mfDiMP | MaCNet | MANet | DMCNet |
|---|---|---|---|---|---|---|---|---|
| A (↑) | 0.6163 | 0.5215 | 0.5723 | 0.6362 | 0.6019 | 0.5451 | 0.5823 | 0.6009 |
| R (↑) | 0.6350 | 0.6904 | 0.7242 | 0.7069 | 0.8036 | 0.5914 | 0.7010 | 0.7088 |
| EAO | 0.2896 | 0.2923 | 0.3180 | 0.3553 | 0.3879 | 0.3052 | 0.3463 | 0.3796 |
Low-quality data experiment: a modality is removed where it is degraded (thermal crossover for thermal, low illumination for RGB); single-modality sequences then feed the same modality §V.F tab. III:
| Dataset | PR | SR |
|---|---|---|
| GTOT | 0.909 | 0.733 |
| GTOT-V1 | 0.782 | 0.692 |
| RGBT234 | 0.839 | 0.593 |
| RGBT234-V1 | 0.820 | 0.584 |
Input-modality variants §V.F tab. IV:
| Variant | RGBT234 PR | RGBT234 SR | GTOT PR | GTOT SR |
|---|---|---|---|---|
| DMCNet | 0.839 | 0.593 | 0.909 | 0.733 |
| DMCNet-AVG-RGBT (average image) | 0.761 | 0.538 | 0.779 | 0.636 |
| DMCNet-RGB | 0.740 | 0.526 | 0.800 | 0.664 |
| DMCNet-T | 0.703 | 0.482 | 0.760 | 0.624 |
Component ablation §V.F tab. V:
| Variant | DMC | RS | RGBT234 PR | RGBT234 SR | GTOT PR | GTOT SR | FPS |
|---|---|---|---|---|---|---|---|
| DMCNet-v1 | no | no | 0.802 | 0.565 | 0.866 | 0.693 | 3.17 |
| DMCNet-v2 | yes | no | 0.820 | 0.584 | 0.909 | 0.733 | 2.38 |
| DMCNet-v3 | no | yes | 0.809 | 0.569 | 0.866 | 0.693 | 3.16 |
| DMCNet | yes | yes | 0.839 | 0.593 | 0.909 | 0.733 | 2.38 |
DMC module variants §V.F tab. VI:
| Variant | RGBT234 PR | RGBT234 SR | GTOT PR | GTOT SR | FPS |
|---|---|---|---|---|---|
| DMC | 0.820 | 0.584 | 0.909 | 0.733 | 2.38 |
| DMC-w/o-msconv | 0.811 | 0.575 | 0.901 | 0.725 | 2.85 |
| DMC-w/o-gate | 0.807 | 0.572 | 0.897 | 0.722 | 2.28 |
| DMC-w/-one-gate | 0.813 | 0.577 | 0.902 | 0.724 | 2.43 |
| DMC-w/o-shifting | 0.790 | 0.563 | 89.9 (as printed) | 73.0 (as printed) | 2.32 |
| FiLM (replaces DMC) | 0.812 | 0.565 | 0.874 | 0.707 | 2.64 |
Paper vs code
The arXiv source links no code. A search of the repositories found
github.com/Alexadlu/DMCNet (Apache-2.0). The owner handle is consistent with
the first author's name (Andong Lu), and the repository README lists the five
authors and the IEEE T-NNLS version of the paper. We take it as the official
repository, though the owner's link to the authors is inferred, not stated.
A shallow clone at 62e6a0e (2022-05-12, after arXiv v3 of 2022-04-29)
contains only README.md, LICENSE and DMCNet_result.zip. The archive was not
extracted; its name suggests result files. The repository has no model,
training or inference code and no configuration files. No pretrained weights
are linked. Every hyperparameter, equation and algorithm step is therefore
compared against the paper alone. No discrepancies can be checked against
code.
Pitfalls
- Learning-rate groups are named differently in two places. Training §III.D names the backbone, the first two FC layers and the binary layer (0.001). Online tracking §IV names the last layer and the other two FC layers. Keep the two sets separate.
- Training negatives are sampled two ways. The training section draws anchors uniformly §III.D. The sampling paragraph says uniform and global sampling supply negatives in both training and inference §IV. The paper does not resolve this.
- Candidate count and re-sampling layout are not stated. The 256 candidates appear only as the index range of eq. 7. The 16 re-sampled boxes are "along the direction" at quarter-box steps §III.C. The text does not say whether they form a line, a grid, or how far from the previous box they start.
- Units and signs for u and the flow. The threshold u = 5 is given without units; pixels are likely but not stated §IV. The flow parameters for DIS are not stated. The direction sign convention beyond "the sign of |dx| and |dy|" is not spelled out §III.C.
- u was swept on a reported benchmark. The sweep over u = {0, …, 30} was run on RGBT234 §V.F fig. 9. RGBT234 is also an evaluation set in the main results, and the paper describes no separate validation split.
- Sign test on the mean score. The paper tests the mean of the five highest scores against 0 §IV. Eq. 7 uses f⁺ only, but the text also computes f⁻, and the text does not say which score is used in the sign test.
- Algorithm 1 cites the wrong equation. The scoring line refers to Eq. 5, which is the gated modulation. The scoring formula is eq. 7. Treat the reference as stale.
- Gate inputs and the cross term. The paper gives one gate formula eq. 4 but does not name the input of each gate, nor whether G1–G4 have separate weights. It does not write out f^scaled_R2T. This card reads it as f_T times G1 applied to the RGB multi-scale feature, which is the first term of eq. 5 §III.B. That reading is inferred from symmetry, not stated, and the shifting-gate inputs (G2, G4) are likewise inferred. Test this against the reported ablation before relying on it.
- Multi-scale operators are underdefined. Eq. 1 uses W^f and W^ms without kernel sizes or activations for the outer W^f convolution. The scaling term in eq. 2 is not bounded, unlike the sigmoid gates. The paper does not say whether this is intended.
- Module placement is implied, not stated. The multi-scale branch differs by level §III.B, which suggests the module sits at all three adapter levels. The paper does not say so directly.
- Missing standard details. Dropout rate, pooling window, LRN parameters, SGD momentum and weight decay, batch-norm settings, and any learning-rate schedule are not stated §III.A §III.D.
- Which MDNet checkpoint. The paper says the backbone starts from MDNet pretrained parameters §III.D. It does not say which checkpoint or release.
- Cost of the parts. The DMC module has the largest effect on speed, and RS adds little §V.F. FPS is in tables V and VI with no hardware or measurement protocol per row. Table V shows DMCNet at 2.38 FPS with DMC, against 3.17 without DMC and RS. The optical flow runs only when the tracker is unreliable §III.C.
- Table VI formatting. The DMC-w/o-shifting GTOT row is printed as 89.9 / 73.0, while every other value is on the 0–1 scale. The paper does not explain this. Reading it as 0.899 / 0.730 is an inference. Keep it as printed until confirmed.
- VOT direction and wording. The text describes R as a failure count §V.E, but the table marks R with ↑, which suggests higher is better. DMCNet's EAO (0.3796) is below mfDiMP's (0.3879), though the text calls the two "comparable" §V.E.
- Cross-dataset training. GTOT results use RGBT234 for training, and RGBT234 and RGBT210 results use GTOT §III.D. The VOT results use GTOT too §V.E. Comparisons across datasets therefore reflect different training data.
- Commented-out numbers in the source. The TeX source contains a commented row for GTOT-V1 and RGBT234-V1 with other values. It is not in the compiled table. This card uses only the compiled values tab. III.
- RGBT210 values are only in a figure. The text gives only the margins over mfDiMP and CAT §V.C fig. 6. Read the figure legend for absolute values.
Lineage
- Builds on: MDNet, for the backbone, training and online recipe §III.A §III.D §IV; MANet, for the modality-specific adapters and the training recipe §III.A §III.D; FiLM, which the mutual condition module extends (with an ablation that swaps it in) §III.B tab. VI; DIS optical flow, for re-sampling §III.C.
- Compared against: on RGBT234, MDNet+RGBT, SGT, CMR, DAPNet, MANet, MaCNet, FANet, CMPP, JMMAC and CAT tab. I; on GTOT and RGBT210, a larger set of RGBT and RGB trackers §V.B §V.C; on VOT-RGBT2019, GESBTT, CISRDCF, MPAT, FSRPN, mfDiMP, MaCNet and MANet tab. II.
- Superseded by: not known from the content of this wiki.
Source and license
Adapted from “Duality-Gated Mutual Condition Network for RGBT Tracking” by Andong Lu, Cun Qian, Chenglong Li, Jin Tang, Liang Wang (arXiv:2011.07188), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Alexadlu/DMCNet at commit 62e6a0e3f6d4 (Apache-2.0). Any quoted code is shown under that license.