Paper · 2024-04-30 · cs.CV
Revisiting RGBT Tracking Benchmarks from the Perspective of Modality Validity: A New Benchmark, Problem, and Solution
MV-RGBT is an RGB-thermal tracking benchmark of 122 video pairs recorded only where one sensor is unreliable. MoETrack runs RGB, TIR and fused experts on a shared ViT and outputs the box of the expert with the highest classification-map score. Author-reported: 93.6/78.4 PR/SR on GTOT, 65.1 SR on RGBT234, 72.1/57.8 on LasHeR, 83.6/69.5 on VTUAV-ST.
Visual object trackingMultimodal fusionMixture of experts (MoE)rgbt-trackingbenchmarkmodality-validitymixture-of-experts
arXivPDFCodeMarkdown11 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Some cameras are blind in some conditions. A regular camera struggles at night or in thick fog, and a thermal camera can lose the target behind glass or next to a mirror. This paper asks what a tracker should do when one of its two sensors is useless. The usual answer is to always combine both sensors. The authors show that this can hurt in exactly those hard moments.
They build a benchmark of video where at least one sensor is unreliable, so that tests reflect those moments. They also propose a tracker with three "experts": one that looks only at the visible-light camera, one that looks only at the thermal camera, and one that combines both. Each expert proposes a box and reports how confident it is, and the tracker keeps the box of the most confident expert. The question the paper frames is when to fuse, and the answer depends on which sensor is working.
TL;DR
MV-RGBT is a 122-pair RGB-thermal tracking benchmark recorded only in scenes where one modality is unreliable §III-A tab. I. MoETrack uses a shared ViT backbone and three tracking heads, one per expert §IV-B fig. 3. The final box comes from the expert whose classification map has the highest peak §IV-B eq. 3. Author-reported, it ties GMMT on GTOT precision and trails it by 0.1 on success, and it has the best reported RGBT234 SR, LasHeR and VTUAV-ST results tab. II.
Problem
Existing RGBT benchmarks are mostly recorded in common conditions where both sensors work. The authors sampled 20% of the videos of several benchmarks and judged that almost all were common scenes §I. Methods fused the two modalities in every frame, and the paper argues this is sub-optimal when one modality is non-informative §I §II-B. A per-frame classifier of data validity was rejected because training data for it is scarce §I.
Method
Benchmark: MV-RGBT
The benchmark has 122 RGB-thermal video pairs, 89.9k frame pairs at 640x480, objects from 36 classes, and 19 scenes §III-A tab. I. Frames come from a FLIR Boson+ 640 thermal camera and an Intel RealSense D456 camera on one platform §III-A. Videos are divided by five RGB-specific or TIR-specific challenges: bad weather, extreme illumination (RGB), TIR truncation, TIR reflection and TIR background clutter §III-A. Frames are aligned with LoFTR key points, and manual key points replace LoFTR when it fails §III-B fig. 2. Annotations mark only the visible part of an object, and all values are set to 0 when the object is unseen or occluded §III-B.
Two subsets are defined from the challenges: MV-RGBT-RGB (the RGB modality is affected) and MV-RGBT-TIR (the TIR modality is affected) §V-F §III-A. The paper does not state how many videos each subset has.
Evaluation
Success rate (sr) counts frames whose IoU with the ground truth exceeds a threshold, and precision rate (pr) counts frames whose centre distance is compared with a second threshold. Both are averaged over frames within each sequence, then over sequences §III-C eq. 1. Area under the curve over a threshold sweep is the final score §III-C. For absent frames, a predicted box gives IoU and distance 0, while a predicted absence gives 1 §III-C. Thresholds per benchmark are in §V-B: the GTOT precision threshold is 5, the LasHeR precision threshold is 20, and only PR and SR are used on LasHeR §V-B.
\mathrm{sr} = \frac{1}{m}\sum_{j=1}^{m}\left(\frac{1}{t}\sum_{i=1}^{t} \mathbf{1}\!\left[\mathrm{IoU}(g_{j,i}, p_{j,i}) > th_s\right]\right)
\mathrm{pr} = \frac{1}{m}\sum_{j=1}^{m}\left(\frac{1}{t}\sum_{i=1}^{t} \mathbf{1}\!\left[\mathrm{Dis}(g_{j,i,c}, p_{j,i,c}) > th_p\right]\right)
Eq. 1 as printed, with the indicator written here in place of the paper's bracket notation eq. 1. Here m is the number of videos, t the frames in a sequence, g and p ground-truth and predicted boxes, and c the box centre.
Solution: MoETrack
The tracker input is an RGB frame and a TIR frame. Both are split into patches, turned into tokens, and given a learnable positional embedding §IV-B. A ViT-base backbone from OSTrack, with 12 transformer encoders, processes each modality, and the backbone weights are shared between the RGB and TIR branches §IV-B.
Three tracking heads form three experts §IV-B fig. 3:
- RGB expert: head on the RGB backbone output only.
- TIR expert: head on the TIR backbone output only.
- RGBT expert: head on the element-wise sum of the RGB and TIR outputs.
The paper states the RGB and TIR heads are CNN-based and the RGBT head's type is not stated §V-G. Each head produces a classification map. Its maximum is the expert's confidence §IV-B. The final prediction is the box of the expert with the highest confidence eq. 3 §IV-B.
p_i = \begin{cases} f(X_{i,rgb};\theta) & \text{if } mc = cs_{rgb} \\ f(X_{i,rgb};X_{i,tir};\theta;\phi) & \text{if } mc = cs_{rgbt} \\ f(X_{i,tir};\theta) & \text{if } mc = cs_{tir} \end{cases}
mc = \max(cs_{rgb}, cs_{tir}, cs_{rgbt})
Eq. 3, as printed eq. 3. cs is the confidence of each expert, \theta the tracker parameters and \phi the fusion weights.
Training gives each expert its own tracking loss, computed as in ViPT §IV-B. The total loss is the mean of the three expert losses eq. 4 §IV-B. Only the backbone is fine-tuned; the other parameters are trained from scratch §IV-B.
loss = \frac{l_{rgb} + l_{tir} + l_{rgbt}}{3}
Eq. 4, as printed eq. 4.
# Per-frame inference [§IV-B] [eq. 3]
# Input: rgb_frame, tir_frame
# Output: box of the most confident expert
def track_frame(rgb_frame, tir_frame):
# Patches to tokens, plus learnable positional embedding [§IV-B]
rgb_tokens = embed(rgb_frame) + pos_rgb # (k, d)
tir_tokens = embed(tir_frame) + pos_tir # (k, d)
# Shared ViT-base backbone, applied to each modality [§IV-B]
rgb_feat = backbone(rgb_tokens) # (k, d)
tir_feat = backbone(tir_tokens) # (k, d)
# Fused feature by element-wise addition [§IV-B]
rgbt_feat = rgb_feat + tir_feat # (k, d)
# One tracking head per expert [§IV-B] [§V-G]
boxes = {}
scores = {}
for expert, feat in experts(rgb_feat, tir_feat, rgbt_feat):
# Confidence = max of the classification map [§IV-B]
boxes[expert], scores[expert] = head[expert](feat)
# Keep the box of the expert with the highest confidence [eq. 3]
best = argmax(scores)
return boxes[best]# Training loss [§IV-B] [eq. 4]
# Input: per-expert losses from ViPT-style tracking losses
# Output: scalar loss for one training step
def training_loss(l_rgb, l_tir, l_rgbt):
# Mean of the three expert losses [eq. 4]
return (l_rgb + l_tir + l_rgbt) / 3Hyperparameters
Only values stated in the paper are listed. The LaTeX gives no total epoch count, weight decay, warm-up, input resolution, token size or head architecture.
| Name | Value | Where | Notes |
|---|---|---|---|
| Hardware | NVIDIA RTX 3090 Ti | §V-A | |
| Backbone | ViT-base from OSTrack, 12 encoders | §IV-B | Section V-A calls it "ViT-256"; meaning not stated |
| Optimizer | AdamW | §V-A | Weight decay not stated |
| Initial learning rate | 7.5e-5 | §V-A | |
| LR schedule | divided by 10 every 10 epochs | §V-A | Total epochs not stated |
| Batch size | 32 | §V-A | |
| Training data | LasHeR training split | §V-A | Whether MV-RGBT is used for training is not stated |
| Fusion | element-wise addition of RGB and TIR features | §IV-B | |
| Expert confidence | max of the classification map | §IV-B | |
| Loss | mean of three expert losses | eq. 4 | Per-expert loss from ViPT §IV-B |
| RGB/TIR head type | CNN-based | §V-G | RGBT head type not stated |
| Positional embedding | learnable | §IV-B | Size not stated |
Results
Author-reported. Setup: each tracker follows the common protocol of its benchmark. The paper compares MoETrack with 22 trackers and its own variants tab. II. Values are as printed; the paper marks best values in red bold, which this card does not reproduce. "-" means not reported.
Table II (GTOT, RGBT234, LasHeR, VTUAV-ST; PR and SR in %, FPS) tab. II:
| Method | Venue | GTOT PR | GTOT SR | RGBT234 PR | RGBT234 SR | LasHeR PR | LasHeR SR | VTUAV-ST PR | VTUAV-ST SR | FPS |
|---|---|---|---|---|---|---|---|---|---|---|
| FSRPN | ICCVW2019 | 89.0 | 69.5 | 71.9 | 52.5 | - | - | 65.3 | 54.4 | - |
| mfDiMP | ICCVW2019 | 83.6 | 69.7 | 84.6 | 59.1 | 44.7 | 34.3 | 67.3 | 55.4 | 10.0 |
| DAFNet | ICCVW2019 | 89.1 | 71.6 | 79.6 | 54.4 | 44.8 | 31.1 | 62.0 | 45.8 | 20.0 |
| CAT | ECCV2020 | 88.9 | 71.7 | 80.4 | 56.1 | 45.0 | 31.4 | - | - | 20.0 |
| CMPP | CVPR2020 | 92.6 | 73.8 | 82.3 | 57.5 | - | - | - | - | 1.3 |
| MANet++ | TIP2021 | 88.2 | 70.7 | 80.0 | 55.4 | 46.7 | 31.4 | - | - | 25.0 |
| JMMAC | TIP2021 | 90.2 | 73.2 | 79.0 | 57.3 | 46.7 | 31.4 | - | - | 4.0 |
| ADRNet | IJCV2021 | 90.4 | 73.9 | 80.7 | 57.0 | - | - | 62.2 | 46.6 | 25.0 |
| MFGNet | TMM2022 | 88.9 | 70.7 | 78.3 | 53.5 | - | - | - | - | - |
| DMCNet | TNNLS2022 | 90.9 | 73.3 | 83.9 | 59.3 | 49.0 | 35.5 | - | - | 2.3 |
| APFNet | AAAI2022 | 90.5 | 73.7 | 82.7 | 57.9 | 50.0 | 36.2 | - | - | 1.3 |
| ProTrack | ACMMM2022 | - | - | 78.6 | 58.7 | 50.9 | 42.1 | - | - | 30.0 |
| MIRNet | ICME2022 | 90.9 | 74.4 | 81.6 | 58.9 | - | - | - | - | 30.0 |
| HMFT | CVPR2022 | 91.2 | 74.9 | 78.8 | 56.8 | - | - | 75.8 | 62.7 | 30.2 |
| LANet | TMM2023 | 91.3 | 75.1 | 79.5 | 58.4 | 53.8 | 43.1 | - | - | 21.7 |
| ECMD | CVPR2023 | 90.7 | 73.5 | 84.4 | 60.1 | 59.7 | 46.7 | - | - | 30.0 |
| QAT | ACMMM2023 | 91.5 | 75.5 | 88.4 | 64.3 | 64.2 | 50.1 | 80.1 | 66.7 | 22.0 |
| ViPT | CVPR2023 | 91.4 | 76.3 | 83.5 | 61.7 | 65.1 | 52.5 | - | - | 39.0 |
| TBSI | CVPR2023 | 91.5 | 75.9 | 87.1 | 63.8 | 69.2 | 55.6 | - | - | 36.0 |
| CAT++ | TIP2024 | 91.5 | 73.3 | 84.0 | 59.2 | 50.9 | 35.6 | - | - | 14.0 |
| BAT | AAAI2024 | 90.9 | 76.3 | 86.8 | 64.1 | 70.2 | 56.3 | 81.8 | 67.4 | 15.0 |
| GMMT | AAAI2024 | 93.6 | 78.5 | 87.9 | 64.7 | 70.7 | 56.6 | 82.8 | 68.5 | 20.0 |
| SETrack | - | 91.7 | 76.6 | 87.1 | 64.4 | 71.7 | 57.2 | 82.7 | 68.7 | 25.0 |
| MoETrack-TIR | - | 64.3 | 56.3 | 76.5 | 54.0 | 59.8 | 47.4 | 51.7 | 41.2 | 25.0 |
| MoETrack-RGB | - | 84.9 | 68.9 | 81.6 | 60.7 | 62.4 | 50.2 | 76.1 | 65.7 | 25.0 |
| MoETrack-RGBT | - | 92.9 | 77.7 | 87.5 | 64.8 | 71.7 | 57.5 | 82.9 | 69.1 | 25.0 |
| MoETrack | - | 93.6 | 78.4 | 88.1 | 65.1 | 72.1 | 57.8 | 83.6 | 69.5 | 23.0 |
MV-RGBT fig. 5 §V-C: the text reports PR 51.4% and SR 67.6% for MoETrack, 2.3 and 2.2 points above GMMT. Those numbers are only in the text and figure; see Pitfalls.
Variant results on MV-RGBT subsets (PR) tab. III:
| PR % | GTOT | RGBT234 | LasHeR | VTUAV-ST | MV-RGBT |
|---|---|---|---|---|---|
| MoETrack-RGBT | 92.9 | 87.5 | 71.7 | 82.9 | 65.3 |
| MoETrack-RGB | 84.9 | 81.6 | 62.4 | 76.1 | 44.0 |
| MoETrack-TIR | 64.3 | 76.5 | 59.8 | 51.7 | 39.7 |
| (1-TIR/RGBT) % up | 30.8 (3) | 12.6 (5) | 16.6 (4) | 37.4 (2) | 39.3 (1) |
| (1-TIR/RGB) % down | 24.3 (4) | 6.2 (2) | 4.2 (1) | 32.1 (5) | 9.8 (3) |
| mRank down | 3.5 | 3.5 | 2.5 | 3.5 | 2 |
Variant results on MV-RGBT subsets (SR) tab. IV:
| SR % | GTOT | RGBT234 | LasHeR | VTUAV-ST | MV-RGBT |
|---|---|---|---|---|---|
| MoETrack-RGBT | 77.7 | 64.8 | 57.5 | 69.1 | 49.1 |
| MoETrack-RGB | 68.9 | 60.7 | 50.2 | 65.7 | 34.8 |
| MoETrack-TIR | 56.3 | 54.0 | 47.4 | 41.2 | 29.5 |
| (1-TIR/RGBT) % up | 27.6 | 16.7 | 17.6 | 40.4 | 40.0 |
| (1-TIR/RGB) % down | 18.3 | 11.1 | 5.6 | 37.4 | 15.3 |
| mRank down | 3.5 | 3.5 | 2.5 | 3.5 | 2.5 |
Benchmark statistics tab. I:
| Benchmark | Num. seq. | Avg. frame | Max. frame | Total frame | Modality validity | Resolution | Category | Scene |
|---|---|---|---|---|---|---|---|---|
| GTOT | 50 | 157 | 376 | 7.8k | no | 384x288 | 9 | 6 |
| RGBT210 | 210 | 498 | 4140 | 104.7k | no | 630x460 | 22 | 8 |
| RGBT234 | 234 | 498 | 4140 | 116.7k | no | 630x460 | 22 | 8 |
| VOT-RGBT2019 | 60 | 334 | 1335 | 40.2k | no | 630x460 | 13 | 5 |
| VOT-RGBT2020 | 60 | 334 | 1335 | 40.2k | no | 630x460 | 13 | 5 |
| LasHeR-test | 234 | 900 | 12862 | 22.0k | no | 630x460 | 19 | 15 |
| VTUAV-test-st | 176 | 3588 | 25295 | 63.1k | no | 1920x1080 (super-resolved) | 13 | 10 |
| MV-RGBT (this paper) | 122 | 737 | 2113 | 89.9k | yes | 640x480 | 36 | 19 |
Ablation. SETrack (fused branch only) is 25.0 FPS. MoETrack-RGBT, the fused expert inside MoETrack, beats SETrack on GTOT, RGBT234 and VTUAV-ST, ties on LasHeR PR (71.7) and is ahead on LasHeR SR (57.5 vs 57.2) tab. II §V-D. On MV-RGBT the text says MoETrack-RGBT is slightly lower on SR and higher on PR than SETrack §V-D; SETrack's MV-RGBT values are only in the figure.
Paper vs code
The official repository is Zhangyong-Tang/MoETrack, owned by the first author. The arXiv metadata abstract names a second repository, Zhangyong-Tang/MVRGBT, which holds the benchmark and one test script; it has no model code.
Checked: test_Mydataset.py in the MoETrack repository. It loads MV-RGBT sequences, reads a per-sequence flag.txt to choose which modality pair to use, and runs a tracker class named InerTrack with config name iner and a default checkpoint epoch of 25 test_Mydataset.py test_Mydataset.py. The paper's name, MoETrack, does not appear in the test script, so the mapping between them is not verified here.
Not checked: the model, expert heads, selection rule and training schedule are in lib.zip in the same repository lib.zip. This run could not extract or list the archive, so the paper-vs-code comparison of the eq. 3 selection rule, the head architectures, the loss, the optimizer and the learning-rate schedule is still open. The README says the method is built on the MPLT baseline and that lib.zip replaces that baseline's lib directory README.md. The repository has no LICENSE file, so no code is quoted here.
Pitfalls
- Version mismatch. The arXiv metadata (v3, updated 2026-03-17) differs from the LaTeX this card reads. The LaTeX title ends in "Method", its abstract names RGBT234, LasHeR and VTUAV-ST, and its author list has 11 names. The metadata title ends in "Solution", its abstract names GTOT and LasHeR and gives the MVRGBT repository, and its author list has 7 names. The card uses the metadata for title and authors, and the LaTeX for everything else. Check numbers against the v3 PDF before implementing.
- Eq. 1 direction. The text says precision counts frames with a centre distance below the threshold, but the printed pr uses ">" eq. 1 §V-B. Read the printed form as a likely typo and implement "below" unless the code shows otherwise.
- Eq. 1 averaging. The text says IoU and distance are averaged within each sequence §III-C; eq. 1 instead averages thresholded indicators. The threshold sweep range and step size are not stated.
- Absent frames. The distance is set to 1 when absence is predicted §III-C. The unit is not given, and it is not clear whether 1 is pixels.
- LasHeR size. Table I says 234 test sequences tab. I; the text says 245 §II-A §V-B.
- MV-RGBT numbers. The MoETrack result on MV-RGBT (PR 51.4, SR 67.6) appears only in the text and in Fig. 5 §V-C. The variant tables give the fused expert alone as PR 65.3 and SR 49.1 tab. III tab. IV. A single expert scoring higher than the full selector on PR, and SR above PR for the full selector, both need checking against the figure and the v3 PDF.
- Input size. Section V-A says "ViT-256" and section IV-B says ViT-base. The input resolution, patch size and token count are not stated.
- Schedule gaps. Total epochs, weight decay, warm-up, search and template sizes, and the RGBT head architecture are not stated. The only schedule hint is the test script's default checkpoint epoch of 25 test_Mydataset.py.
- Training data. The paper trains on the LasHeR training split §V-A. Whether MV-RGBT, GTOT or RGBT234 are used for training is not stated, and MV-RGBT has no train/test split described.
- Selection ties and calibration. The rule for ties is not stated. The three heads' maxima are compared directly; the paper does not discuss their calibration eq. 3.
- Overstated claims. "Outperforms ViPT across all the benchmarks" §V-G: ViPT has no VTUAV-ST value tab. II. "MoETrack-RGBT exceeds SETrack on all benchmarks" §V-D: LasHeR PR is tied at 71.7.
- Speed. MoETrack runs at 23 FPS, below ViPT's 39.0 tab. II. The efficiency advantage is claimed only against GMMT and BAT §V-G.
- Missing counts. Video counts per MV-RGBT subset, and the number of manual key-point corrections, are not stated §III-B §V-F.
- Dataset licence. The MVRGBT repository says the data is hosted on Baidu Pan README.md. The dataset's own licence is not stated. This card reproduces no dataset content.
Lineage
- Builds on: ViT-base backbone from OSTrack §IV-B README.md; per-expert tracking loss from ViPT §IV-B; code built on the MPLT baseline README.md.
- Compared against: ViPT, TBSI, BAT, GMMT, QAT, HMFT and the other RGBT trackers in Table II tab. II.
- Superseded by: not known from content on this wiki.
Source and license
Adapted from “Revisiting RGBT Tracking Benchmarks from the Perspective of Modality Validity: A New Benchmark, Problem, and Solution” by Zhangyong Tang, Tianyang Xu, Zhenhua Feng, Xuefeng Zhu, Chunyang Cheng, Xiao-Jun Wu, Josef Kittler (arXiv:2405.00168), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Zhangyong-Tang/MoETrack at commit 2be4b703bf7d (No LICENSE file in the repository (not licensed; no code quoted)). Any quoted code is shown under that license.