RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

In plain words

A tracker follows one object through a video, given its position in the first frame. An RGBT tracker gets two video streams at once: a normal color camera and a thermal infrared camera. The two see different things, and each helps where the other fails, so combining them can help.

The obvious designs have problems. One option gives each camera its own full network, which wastes parameters and never lets the two streams learn from each other. The other shares one network for both cameras, which blurs what only one camera can see. This paper uses three parts. A shared network handles what both cameras agree on. A small side branch per camera carries what is specific to that camera. A final stage weighs the two cameras for each candidate position and combines them. A training loss also rewards the shared features for looking alike across the two cameras and rewards the specific features for looking different.

TL;DR

• Multi-adapter RGBT tracker: generality (shared), modality (per-camera, small) and instance (per-target, fused) adapters, trained end to end. §3.1
• Hierarchical divergence loss: at each of three layers, a multi-kernel MMD minimises divergence in the shared branch and maximises it in the modality branch. §3.2 eq. 11
• Author-reported best on RGBT234: 80.0% PR and 55.4% SR, against 77.7% and 53.9% for the earlier MANet. tab. 1 tab. 4
• Speed claim: 25.4 FPS on RGBT234 versus 3.1 FPS for MANet, from RoIAlign feature extraction. tab. 4 §5.7
• Code: author repository holds only a README and a results archive; no model or training code, no license. README.md

Problem

RGBT trackers that run one network per modality ignore shared structure and add redundant parameters. Single-stream trackers that share all weights keep the shared structure but suppress useful single-modality cues §1. Cross-modal interaction layers can miss the shared information §1. The paper argues that both shared and modality-specific cues must be modelled, and that a per-target fusion step is needed on top §1 §2. The method targets an online tracking setting, where the instance part is updated while the sequence runs §4.

Method

Overview. Both frames of an RGBT pair pass through the shared generality adapter (GA) and the modality adapter (MA). Their outputs are combined per modality, features for each candidate box are pooled with RoIAlign, and the instance adapter (IA) scores each candidate §3.1 fig. 2.

1. Generality adapter (GA). The first three layers of a modified VGG-M, shared by both modalities. Kernels are 7×7×96, 5×5×256 and 3×3×512; the third layer uses dilation 3 §3.1. Layers one and two are convolution, ReLU and local response normalisation §3.1.

2. Modality adapter (MA). A parallel branch on each GA layer, one per modality, with a small kernel. Kernels are 3×3 (for the 7×7 GA layer), 1×1 (for the 5×5 layer) and 1×1 (for the 3×3 layer). Channel counts match GA so the two can be added §3.1. The MA branch uses an Independent Component layer after convolution and ReLU, then max pooling §3.1 [Chen et al. 2019, arXiv:1905.05928, cited in §3.1]. The text says MA has about 20% of GA's parameters §3.1.

3. Decomposition. The full layer kernel W is written as the shared kernel W^GA plus a zero-padded, centred embedding of the modality kernel W^MA_m. This yields eq. 4: each MA layer applies its own convolution to the same input as the shared layer eq. 1 eq. 2 eq. 3 eq. 4.

4. Combining modalities. For each modality, the GA and MA feature outputs are added element-wise §3.1. The network input is two whole frames, one per modality §3.1.

5. RoIAlign. An adaptive RoIAlign layer extracts a 7×7 map per candidate from the combined maps. Max pooling reduces it to 3×3 §3.1. The flattened size fed to the FC layers is not stated; 3×3×512 is implied by the layer widths §3.1.

6. Instance adapter (IA). Eight FC layers: FC_R, FC_R1, FC_T, FC_T1, WP_R, WP_T, FC_fusion and FC_instance, with output widths 512, 512, 512, 512, 2, 2, 512 and 2 §3.1. FC_R and FC_R1 share parameters, and FC_T and FC_T1 share parameters fig. 2. Dropout follows every FC layer except FC_R and FC_T. WP_R and WP_T and FC_instance end in a softmax over two classes. Other FC layers use ReLU §3.1.

7. Quality-aware fusion. Each modality gets a weight from its candidate scores. For modality m, the weight is a sigmoid of the absolute mean of the difference between positive and negative scores over the candidates eq. 5. The candidate features of each modality are multiplied by its weight, then re-encoded by FC_R (FC_T). The two are concatenated and passed through FC_fusion §3.1. Fusion sits at the first FC layer, which the paper picks over earlier and later positions §3.1 §5.6 tab. 3.

8. Losses. The overall offline loss is L_off = L_cls + ν1·L_inst + ν2·L_hd, with ν1 = 0.1 eq. 15.

   • Binary classification loss L_cls = L_fusion + λ1·L_R + λ2·L_T, with λ1 = λ2 = 0.5 offline and 1 online eq. 13 eq. 12 §3.2.
   • Instance embedding loss L_inst, applied to positive samples only, to separate targets across training sequences eq. 14 §3.2.
   • Hierarchical divergence loss L_hd = Σ_j ψ^j(GA_rgb, GA_t) − Σ_j ψ^j(MA_rgb, MA_t), summed over j = 1..3 eq. 11 §3.2. ψ is the unbiased MK-MMD estimate of eq. 10, over a kernel built from d = 11 Gaussians eq. 8 eq. 9 eq. 10 §3.3.

9. Offline training. One-stage, end to end, with SGD §3.3. Each iteration draws one sequence: 8 random frames, each with 32 positive and 96 negative candidates §3.3. The paper's wording is that GA is initialised from VGG-M and the rest is random §3.3. The GTOT model is fine-tuned on RGBT234 and the reverse §3.3. The ν2 schedule is 1 for the first 200 iterations, 0.1 until 500, then 0.01 §3.3.

10. Online initialisation on the first frame. A new FC_instance branch is built for the sequence. Training uses 500 positive and 5000 negative candidates from the first frame, for 50 iterations. Learning rate is 0.001 for FC_instance and 0.0001 for the other IA layers §3.3. A bounding-box regressor (from MDNet) is trained on the first frame only §3.3 §4.

11. Online tracking §4 eq. 16.

    • Draw 256 candidates by Gaussian sampling around the previous box §4.
    • Score each candidate and take the highest positive score as the box eq. 16.
    • If the positive score exceeds 0.5, refine the box with the regressor §4.
    • If the score drops below the threshold (set to zero), run a short-term update. Run a long-term update every 10 frames §3.3 §4.
    • Only the classification loss L_cls is used during online updates §3.3.
    • GA and MA stay frozen while tracking §4.

# Offline training step  [§3.3] [eq. 15]
# frames: 8 frames per iteration; 32 pos and 96 neg candidates each
def training_step(frames, candidates, labels, iteration):
    # Shared and modality-specific features  [§3.1] [eq. 4]
    feat_rgb, feat_thermal, ga_maps, ma_maps = extract_features(frames)
    # Classification loss with lambda = 0.5 offline  [eq. 13] [eq. 12]
    loss_cls = scoring_loss(candidates, labels, lam_rgb=0.5, lam_t=0.5)
    # Instance embedding loss on positives only  [eq. 14]
    loss_inst = instance_loss(candidates, labels)
    # Hierarchical divergence over layers j = 1..3  [eq. 11]
    loss_hd = 0
    for j in range(3):
        loss_hd += mk_mmd(ga_maps[j].rgb, ga_maps[j].thermal)
        loss_hd -= mk_mmd(ma_maps[j].rgb, ma_maps[j].thermal)
    # nu2 schedule: 1, then 0.1 after 200, then 0.01 after 500  [§3.3]
    nu2 = nu2_schedule(iteration)
    # Overall loss with nu1 = 0.1  [eq. 15]
    loss = loss_cls + 0.1 * loss_inst + nu2 * loss_hd
    # SGD, learning rate 1e-4, weight decay 5e-4  [§3.3]
    sgd_step(loss, lr=1e-4, weight_decay=5e-4)

# Unbiased MK-MMD between the two modality maps of one layer  [eq. 10]
# feats_a, feats_b: (b, C, H, W) layer outputs; b is the batch size
def mk_mmd(feats_a, feats_b):
    # Flatten each sample to one vector; the paper does not say how  [eq. 10]
    xs = flatten_per_sample(feats_a)    # (b, D)
    ys = flatten_per_sample(feats_b)    # (b, D)
    total = 0
    # Pairs (2i-1, 2i) in 1-based indexing, i = 1..b/2  [eq. 10]
    for i in range(b // 2):
        a1, a2 = xs[2 * i], xs[2 * i + 1]
        t1, t2 = ys[2 * i], ys[2 * i + 1]
        # H_k term: same-modality terms minus cross-modality terms  [eq. 10]
        total += (multi_kernel(a1, a2) + multi_kernel(t1, t2)
                  - multi_kernel(a1, t2) - multi_kernel(t1, a2))
    return 2 * total / b

# Multi-kernel Gaussian sum with d = 11 kernels  [eq. 8] [§3.3]
# Written as in the paper, not the standard form; see Pitfalls
def multi_kernel(p, q):
    total = 0
    for u in range(1, 12):
        # sigma_u = 2^(u - 6); beta_u values taken from Gretton et al.  [§3.3]
        total += beta[u] * exp(-(norm(p - q) ** (2 / sigma[u])))
    return total

# Online tracking loop for one sequence  [§4] [§3.3] [eq. 16]
def track_sequence(frames, first_box):
    # Build the sequence-specific FC_instance and fine-tune IA  [§3.3]
    initial_train(frames[0], first_box, iterations=50)
    # Box regressor on the first frame only  [§3.3]
    train_regressor(frames[0], first_box)
    box = first_box
    for t, frame in enumerate(frames[1:], start=1):
        # 256 Gaussian samples around the previous result  [§4]
        candidates = gaussian_sample(box, count=256)
        # Positive score per candidate, GA and MA frozen  [§4]
        scores = positive_scores(frame, candidates)
        best = argmax(scores)  # [eq. 16]
        box = candidates[best]
        if scores[best] > 0.5:
            # Refine scale and position with the regressor  [§4]
            box = regress(frame, box)
        if scores[best] < threshold:  # threshold = 0  [§3.3] [§4]
            # Short-term update from IoU > 0.7 and IoU < 0.3 samples  [§3.3]
            short_term_update(frame, box)
        if t % 10 == 0:
            # Long-term update every 10 frames  [§3.3] [§4]
            long_term_update()
    return boxes

Hyperparameters

Results

Author-reported. Setup: RGBT234 (234 sequences), GTOT (50 sequences) and VOT-RGBT2019 (60 sequences) §5.1. Baselines are RGB trackers, RGB trackers extended with the thermal channel (+RGBT), and RGBT trackers. Metrics are maximum precision rate (MPR) and maximum success rate (MSR) on RGBT234 §5.1. Values below are copied from the paper's tables tab. 1 tab. 2 tab. 3 tab. 4; the text's GTOT values are in §5.2.

RGBT234, attribute-based PR/SR (%) tab. 1

The source marks best and second-best values in colour; those marks are not reproduced here. tab. 1

VOT-RGBT2019 tab. 2 (EAO: expected average overlap; A: accuracy; R: robustness; higher is better for all three)

GTOT and RGBT234 with ablations tab. 4 (PR, SR, FPS)

Fusion position on RGBT234 tab. 3

Other reported numbers, from text only

• GTOT, relative to RGB trackers: +8.9/+9.0 PR/SR over MDNet, +11.6/+15.7 over DSST, +13.0/+10.5 over DAT §5.2. The text does not say whether these are percentage points or relative gains. Their source figures are fig. 3.
• GTOT, relative to RGBT trackers: PR +0.7 over MANet, +1.0 over FANet, +1.9 over DAPNet §5.2. SR is slightly lower than MANet and FANet §5.2. Their source figure is fig. 4.
• RGBT234: +6.9 PR over DAT and +4.0 SR over C-COT §5.3. These two trackers are not in tab. 1; their source figures are fig. 5.
• RGBT234: +2.3 PR and +1.5 SR over MANet §5.3 (matches tab. 1).
• Attribute gains on LR (about 6 points PR over the second-best) §5.3. The text says PO, LI, TC, DEF, CM and BC gain about 3 points §5.3.
• Speed: MANet++ is about 1.34 times faster than FANet and about 31.75 times faster than MaCNet on VOT-RGBT2019 §5.7 (matches tab. 2). The text says "8 times faster than MANet" on GTOT §5.2 (27.3/3.5 ≈ 7.8 in tab. 4).
• Visual comparisons on four sequences are in fig. 8 §5.3.

Paper vs code

The repository is https://github.com/Alexadlu/MANet_pp. The owner handle matches the first author's name (an inference; the owner is not stated anywhere). The README names this paper, the IEEE Transactions on Image Processing venue and a contact address README.md. The paper's own footnote points to chenglongli.cn/code-dataset/ §1. That page links only to the GitHub organisation mmic-lcl, and a repository search there for MANet returned nothing. A repository search for "MANet RGBT" returned two repositories: this one and Alexadlu/MANet, the earlier conference version (not this paper's method). Neither reports a license in the GitHub API.

What the repository holds, at commit 04016491928a57c7b23c6dce5fac7dd6b729d985: README.md and MANet++_res.zip. There is no model code, training script, configuration file or LICENSE file. I did not list the zip's contents, so it may contain results only, and I did not check that. The network, losses and training loop could not be compared with the paper.

No discrepancies can be checked. Every value on this card comes from the paper. The code_license field records that the repository has no LICENSE file, so the code cannot be reused.

Pitfalls

• Adapter combination is ambiguous. Eq. 4 adds the GA and MA convolutions layer by layer on the same input eq. 4. §3.1 instead says the two feature types are added element-wise per modality, which reads as a single addition at the output §3.1. The IC layer and max pooling are described only for the MA branch §3.1. Whether GA also pools or normalises is not stated, apart from the LRN in its first two layers §3.1.
• Eq. 3 is not implementable as written. The index bounds are strict (0<i<a, 0<j<b), which would exclude the first and last kernel entries. S is not given, and the offset (S−a)/2 needs S−a to be even eq. 3. The text describes a small kernel, not a padded one §3.1, so the equation is a conceptual aid rather than an implementation recipe.
• Gaussian kernel form. The paper writes k_u(p, q) = exp(−‖p−q‖^(2/σ_u)) §3.3. That is not the usual Gaussian (which has a squared distance over σ or 2σ²). σ_u = 2^(u−6) spans 0.03 to 32 §3.3. The kernel weights β_u are "the same as in" Gretton et al. and their values are not given §3.3. Choose the form and the β_u explicitly.
• MK-MMD estimator is underspecified. Eq. 10 sums over i = 1..b/2 with a "1<i<b" bound, which is inconsistent. The batch b is not defined. It is not stated which samples enter the estimate (positives, negatives or both), or whether the feature maps are per candidate or per frame eq. 10 §3.2. Feature flattening is not stated, and neither is the value of D in eq. 8.
• Probability versus logit thresholds. The text says WP and FC_instance use softmax §3.1. Thresholds on f+ are then "f+ > 0.5" for regression and "f+ < 0" for the short-term update §4. A softmax probability can never be negative, so the second threshold can't trigger. The paper does not say whether f+ is a logit difference or a probability. §3.3 gives the threshold as zero "on the target score". Decide which score is meant.
• Loss labels. §3.2 calls the classification loss binary cross entropy but writes softmax cross entropy §3.2 eq. 12. The subscript d on L_fusion (the d-th FC_instance branch) is missing from L_R and L_T, so the branch used there is not stated eq. 12. L_inst is written for positives only eq. 14.
• Modality weights. Eq. 5 writes a mean over n = 0..255 and an absolute value over the sum. The index range conflates the candidate count with the summation index. The weight is a sigmoid, so the two weights are not normalised to sum to one eq. 5 §3.1.
• ν2 schedule. The text says "200 iterations" and then "500 rounds" §3.3. A round may not be an iteration. Whether the schedule is per iteration or per epoch is not stated.
• Training length. "Until the model converges" §3.3 gives no count, no stopping rule and no learning rate schedule.
• Input size and dropout. Input frame size, RoIAlign sampling ratio, the dropout rate and the bounding-box regressor architecture are not stated. Only "a bounding-box regressor from MDNet" is given §3.3 §4.
• Dataset counts disagree. The abstract and conclusion say "two benchmark datasets" [abstract] §6, the introduction says "three" §1, and §5.1 lists three (GTOT, RGBT234, VOT-RGBT2019). The RGBT234 text says "10 state-of-the-art RGBT trackers" and "11 RGB trackers", but its lists name 12 and 10 §5.3.
• VOT result is not uniformly better. On VOT-RGBT2019, MANet++ (EAO 0.2716) is below MANet (0.3463) and the RoIAlign-free variant (0.3635) tab. 2. The paper attributes this to RoIAlign weakening low-resolution features §5.4. On RGBT234 the RoIAlign-free variant has higher PR (80.6 vs 80.0) and lower SR (55.2 vs 55.4) tab. 4. RoIAlign is the main speed source (25.4 vs 3.1 FPS).
• Cross-dataset training. GTOT results come from a model fine-tuned on RGBT234, and RGBT234 results from one fine-tuned on GTOT §3.3. Compare like with like when quoting these numbers.
• Speed numbers depend on hardware. FPS is measured on PyTorch 0.4.1 with an RTX 2080Ti §5.7. The text mixes fractions (0.901 PR) and percents (90.1 PR) for the same GTOT numbers §5.2 tab. 4.
• Unverifiable parameter claim. The "about 20% of GA's parameters" claim for MA §3.1 has no count. The total model size in tab. 2 is 28.163 Mb for MANet++ and 27.802 Mb for MANet, which is larger.
• No implementation released. Nothing in the public repository implements the method. Reproduction must follow the paper alone.
• Version. The source is arXiv v3 (2021-06-04). The README cites the IEEE TIP 2021 version (vol. 30, pp. 5613–5625). Differences between the two are not checked.

Lineage

• Builds on: MANet, the conference version by the same group (Li et al., ICCV workshops 2019), which this paper extends with the one-stage training, the MK-MMD divergence loss, the IC layer, RoIAlign and fusion §1 §3.1. RT-MDNet (Jung et al., ECCV 2018) for the adaptive RoIAlign and pooling §3.1 §3.3. MDNet (Nam and Han, CVPR 2016) for the instance-adapter structure and bounding-box regression §3.1 §4. VGG-M (Simonyan and Zisserman, ICLR 2015) for the generality adapter §3.1. MK-MMD (Gretton et al., NeurIPS 2012) for the divergence §3.2. The independent component layer (Chen et al., arXiv:1905.05928) for the MA normalisation §3.1.
• Compared against: MaCNet, FANet, DAPNet, SGT, CMR, SOWP+RGBT, CFNet+RGBT, RT-MDNet+RGBT, MDNet+RGBT, mfDiMP and others §5.2 §5.3 §5.4.
• Superseded by: not known from content in this wiki.