wikillm

Paper · 2023-11-27 · cs.CV

Single-Model and Any-Modality for Video Object Tracking

Un-Track: one OSTrack-based ViT tracker for RGB plus any one of depth, thermal or event input. It learns a shared embedding from edge maps and low-rank factors, prompts RGB tokens with it, and adds LoRA. One parameter set reaches 0.610 F-score on DepthTrack versus 0.594 for the depth-specific ViPT (author-reported).

Visual object trackingMultimodal fusionLow-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningvideo-trackingmultimodalloraprompting

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Video object tracking means following one object from frame to frame. Extra sensors such as depth, thermal and event cameras help in hard cases. Earlier methods need a separate set of weights for each sensor, so a system ends up storing several sets.

Un-Track keeps one set of weights for all of them. It learns a common representation of the extra sensor's signal, a bit like one translator that turns several languages into one shared language. Training uses only RGB plus one extra sensor at a time, so no sample needs every sensor. At run time the shared representation can stand in for a sensor that is missing.

The shared features are then used as hints that fix weak image patches, and the big pretrained RGB tracker is adjusted with small add-on matrices. This keeps training cheap.

TL;DR

  • Un-Track is a single-parameter-set tracker for RGB plus one of depth, thermal or event input, built on the OSTrack ViT tracker §3.1.
  • A shared embedding is learned from the auxiliary input through edge maps, per-modality low-rank factors and a reconstruction step §3.2.
  • RGB tokens are prompted with that embedding by classing tokens as positive, negative or uncertain and fusing them in low-rank space §3.3; LoRA adapts the attention layers §3.4.
  • Author-reported: on DepthTrack the uni-model reaches F-score 0.610 against 0.594 for the depth-specific ViPT tab. 1. Code is official, but the repository has no LICENSE file.

Problem

RGB trackers fail in hard cases, and extra modalities help §1. Existing unified multimodal trackers still need fine-tuning per modality, which leaves several parameter sets §1, §2. The authors also name two constraints: tracking datasets contain one modality each, and not every paired combination exists, so joint training over all combinations is not possible §1, §2. The method must therefore handle a missing modality at training and inference time §1.

Method

The training set is a union of RGB-X pairs, with X one of depth (DepthTrack), thermal (LasHeR) or event (VisEvent) §4.1. Each sample carries exactly one auxiliary modality. A baseline tracker with frozen parameters is adapted; a pretrained OSTrack model supplies the initial weights §3.1, fig. 2. Trainable parts are the cross-modal binding, the outer prompting, and the inner LoRA parameters §3.1. The paper says the pipeline fits on a single 24G GPU §3.1.

Shared embedding (§3.2, fig. 3). Gradient maps are computed from the auxiliary image by differences between neighbouring pixels in x and y. The paper also states that RGB gradient maps are computed, and all gradient maps are merged with the visual feature into a gradient feature G §3.2, fig. 3. The auxiliary features are split by modality (depth, thermal, event). Each subset gets its own low-rank factor of rank k, and the three factors are joined and combined with the low-rank form of G §3.2, eq. 1–2, alg. 1. A reconstruction step maps the joint factor back to the feature width and adds G §3.2, eq. 3.

Pseudocode
# Shared embedding from the auxiliary input  [alg. 1] [§3.2]
# Input:  aux_tokens (B, N, C); modality (B,); grad_feat G (B, N, C)
# Output: shared embedding F (B, N, C)

def shared_embedding(aux_tokens, modality, grad_feat):
    # Low-rank form of the gradient feature  [§3.2] [eq. 2]
    grad_low = low_rank_g(grad_feat)              # (B, N, k)
    # Keep the tokens of each modality, zeros for the others  [§3.2]
    depth_tokens = aux_tokens * (modality == DEPTH)
    thermal_tokens = aux_tokens * (modality == THERMAL)
    event_tokens = aux_tokens * (modality == EVENT)
    # One low-rank factor of rank k per modality  [eq. 1]
    depth_low = sigma_d(depth_tokens)
    thermal_low = sigma_t(thermal_tokens)
    event_low = sigma_e(event_tokens)
    # Join the three factors, add the gradient factor  [eq. 2]
    joint_low = phi_r1(concat(depth_low, thermal_low, event_low))
    joint_low = joint_low + phi_r2(grad_low)
    # Map back to the feature width, add the gradient feature  [eq. 3]
    return Phi_r(joint_low) + grad_feat

Outer modal prompting (§3.3, fig. 4). Each RGB token is put in one of three groups by a learnable score: negative, uncertain or positive. Negative tokens are replaced by the matching tokens of the shared embedding, uncertain tokens are completed with them, and positive tokens are kept. The two parts are fused in low-rank space. The same steps are run with the shared embedding as the token source, and the two low-rank matrices are added and projected back §3.3, eq. 4–7.

Pseudocode
# One branch of outer modal prompting  [§3.3] [eq. 4-6]
# Input:  own (B, N, C); other (B, N, C) from the other stream
# Output: low-rank matrix (B, N, k) for the own tokens

def prompt_branch(own, other, score):
    # Mask each token as negative, uncertain or positive  [§3.3]
    neg, unc, pos = split_by_score(score(own))
    # Keep positive tokens, replace negative ones  [eq. 4]
    low_pos = sigma_c(neg * other + pos * own)
    # Complete uncertain tokens from the other stream  [eq. 5]
    low_unc = sigma_n(unc * other + unc * own)
    # Fuse both parts in low-rank space  [eq. 6]
    return phi_p(concat(low_pos, low_unc))
Pseudocode
# Outer modal prompting of RGB tokens  [§3.3] [eq. 7]
# Input:  rgb_tokens I (B, N, C); shared embedding F (B, N, C)
# Output: fused tokens O (B, N, C)

def modal_prompt(rgb_tokens, shared):
    # Low-rank matrix of the RGB tokens  [§3.3]
    rgb_low = prompt_branch(rgb_tokens, shared, score_rgb)
    # Same steps with F as the token source  [§3.3]
    shared_low = prompt_branch(shared, rgb_tokens, score_shared)
    # Add both low-rank matrices, project back  [eq. 7]
    return Phi_p(rgb_low + shared_low)

Inner finetuning with LoRA (§3.4). Each attention module keeps its frozen weight W0 and gets a trainable low-rank path B·A §3.4, eq. 8.

Pseudocode
# Low-rank adaptation of one attention weight  [§3.4] [eq. 8]
# Input: x (N, k); frozen W0 (d, k); B (d, r); A (r, k)
# Output: h (N, d)

def lora_linear(x, W0, B, A):
    # Frozen path, as in the pretrained module  [§3.4]
    frozen = W0 @ x
    # Trainable path of rank r  [eq. 8]
    return frozen + B @ (A @ x)

Loss (§3.4). The paper says the loss is the same as the baseline tracker's §3.4. It does not name the terms or weights. The official code uses a GIoU term, an L1 term and a focal term on the centre map, with weights 2.0, 5.0 and 1.0 train_script.py. The weights below are from code only.

Pseudocode
# Training loss, same family as the OSTrack baseline  [§3.4]
# Weights 2.0, 5.0, 1.0 come from code, not the paper
# [code: lib/train/train_script.py#L71-L72]
# Input: pred_box, gt_box (B, 4); score_map, gt_map (B, 1, H, W)

def training_loss(pred_box, gt_box, score_map, gt_map):
    # Generalised IoU on the predicted box  [§3.4]
    giou = giou_loss(pred_box, gt_box)
    # L1 on box coordinates  [§3.4]
    l1 = l1_loss(pred_box, gt_box)
    # Focal loss on the centre heat map  [§3.4]
    focal = focal_loss(score_map, gt_map)
    return 2.0 * giou + 5.0 * l1 + 1.0 * focal

Hyperparameters

The paper states few training values. The values below in the first group are from the paper's ablations; the rest come from the official experiment config and are not stated in the paper.

NameValueWhereNotes
shared-embedding rank k4 (default); 2 and 8 tested§5 tab. 8(a) vit_ce_prompt.pypaper: lower ranks give sparse features, higher ones keep modality detail
modal-prompting rank8 (default); 4 and 16 tested§5 tab. 8(b) vit_ce_prompt.py
LoRA rank r4 (default); 2 and 8 tested§5 tab. 8(c) vit_ce_prompt.pyno scaling factor in code
positive and negative token share1/4 each (default); 1/8 and 1/3 tested§5 tab. 8(d) vit_ce_prompt.pycode fixes the share as N//4
LoRA placementevery attention block§3.4 vit_ce_prompt.py
initialisationOSTrack checkpoint, epoch 300fig. 2 §3.1 deep_rgbx.yamlcheckpoint is on Google Drive per README; not fetched
search crop256 px, area factor 4.0deep_rgbx.yamlnot stated in paper
template crop128 px, area factor 2.0deep_rgbx.yamlnot stated in paper
optimiserAdamW, LR 4e-4, weight decay 1e-4deep_rgbx.yaml base_functions.pynot stated in paper
batch size48deep_rgbx.yamlthe YAML comment reads 32
epochs and LR step80 epochs; step decay ×0.1 at epoch 48deep_rgbx.yamlnot stated in paper
samples per epoch60000 training, 10000 validationdeep_rgbx.yaml
training data mixDepthTrack, LasHeR, VisEvent, ratio 1:1:1deep_rgbx.yamlpaper: 150, 979 and 500 sequences §4.1
gradient clip0.1deep_rgbx.yamlconfigured; use not traced
drop path0.1deep_rgbx.yaml
loss weightsGIoU 2.0, L1 5.0, focal 1.0deep_rgbx.yaml train_script.pypaper: same as baseline §3.4
candidate eliminationkeep 0.7 at blocks 3, 6, 9; start epoch 4; warm-up 16deep_rgbx.yamlnot mentioned in paper
prompt typedeepdeep_rgbx.yaml
test sizessearch 256 / 4.0; template 128 / 2.0; test epoch 60deep_rgbx.yamldefaults in lib/config/untrack/config.py differ (search 320, factor 5.0)
hardwaresingle 24G GPU§3.1paper statement; batch and epochs not given

Results

Author-reported. Setup: five benchmarks §4. Within distribution: DepthTrack, LasHeR, VisEvent §4.2. Generalisation: VOT-RGBD2022 and RGBT234, with the RGB-only setting §4.3. The "uni-model" is the single parameter set trained on the union of all pairs §4.1. Baselines and metrics follow the cited original papers §4.2. Table numbers follow LaTeX source order; check them against the compiled PDF.

DepthTrack test set tab. 1 (F-score, recall, precision; depth-specific left, uni-model right)

TrackerSettingF-scoreRePr
ATCAISdepth-specific0.4760.4550.500
DDiMPdepth-specific0.4850.4690.503
DeTdepth-specific0.5320.5060.560
SPTdepth-specific0.5380.5490.527
ProTrackdepth-specific0.5780.5730.583
ViPTdepth-specific0.5940.5960.592
Un-Trackdepth-specific0.6120.6100.613
Starkuni-model0.3970.4060.388
AiATrackuni-model0.5150.5260.505
OSTrackuni-model0.5690.5820.557
UniNextuni-model0.4220.4320.413
SeqTrackuni-model0.5900.6000.580
ViPTuni-model0.5610.5620.560
Un-Trackuni-model0.6100.6100.610

LasHeR thermal test set tab. 2 (precision PR, success SR)

TrackerSettingPRSR
SGTthermal-specific0.3270.232
FANetthermal-specific0.4410.309
mfDiMPthermal-specific0.4470.343
DAFNetthermal-specific0.4480.311
MaCNetthermal-specific0.4820.350
ProTrackthermal-specific0.5380.420
ViPTthermal-specific0.6510.525
Un-Trackthermal-specific0.6670.536
Starkuni-model0.4180.333
AiATrackuni-model0.4630.365
OSTrackuni-model0.5300.422
SeqTrackuni-model0.5820.441
ViPTuni-model0.6080.490
Un-Trackuni-model0.6460.513

VisEvent test set tab. 3 (precision, success)

TrackerSettingPrecisionSuccess
SiamCarevent-specific0.5990.420
Stark_Eevent-specific0.6120.446
VITAL_Eevent-specific0.6490.415
PrDiMP_Eevent-specific0.6440.453
TransT_Eevent-specific0.6500.474
ProTrackevent-specific0.6320.471
ViPTevent-specific0.7580.592
Un-Trackevent-specific0.7630.597
Starkuni-model0.6160.448
AiATrackuni-model0.6260.444
OSTrackuni-model0.6910.525
SeqTrackuni-model0.6650.504
ViPTuni-model0.7400.579
Un-Trackuni-model0.7550.589

VOT-RGBD2022, generalisation tab. 4 (EAO, accuracy, robustness)

TrackerSettingEAOAccuracyRobustness
DRefinedepth-specific0.5920.7750.760
Stark_Ddepth-specific0.6470.8030.798
DMTrackerdepth-specific0.6580.7580.851
DeTdepth-specific0.6570.7600.845
SBT_Ddepth-specific0.7080.8090.864
SPTdepth-specific0.6510.7980.851
ProTrackdepth-specific0.6510.8010.802
ViPTdepth-specific0.7210.8150.871
Starkuni-model0.4450.7140.598
AiATrackuni-model0.6410.7690.832
OSTrackuni-model0.6660.8080.814
SeqTrackuni-model0.6790.8020.846
Un-Trackuni-model0.7180.8200.864

RGBT234, generalisation tab. 5 (MPR, MSR)

TrackerSettingMPRMSR
mfDiMPthermal-specific0.6460.428
SGTthermal-specific0.7200.472
DAFNetthermal-specific0.7960.544
FANetthermal-specific0.7870.553
MaCNetthermal-specific0.7900.554
CMPPthermal-specific0.8230.575
APFNetthermal-specific0.8270.579
ProTrackthermal-specific0.7950.599
ViPTthermal-specific0.8350.617
Un-Trackthermal-specific0.8370.618
Starkuni-model0.6770.496
AiATrackuni-model0.7110.508
OSTrackuni-model0.7550.569
SeqTrackuni-model0.8060.599
Un-Trackuni-model0.8420.625

Cost and RGB-only setting, DepthTrack with dummy depth input tab. 6

ModelGFLOPsParams (M)F-scoreRePr
RGB baseline (OSTrack)21.5092.080.5290.5220.536
ViPT21.8092.960.5420.5380.546
Un-Track23.6498.730.5580.5570.560

Ablations on DepthTrack, single parameter set §5

Key components tab. 7:

VariantF-scoreRePr
without shared embedding0.5990.6020.597
ViPT's prompt block in place of ours0.5790.5750.584
without LoRA finetuning0.5940.5980.596

Rank and share tab. 8; the shaded default is marked "default":

SettingValueF-scoreRePr
(a) shared-embedding rank k20.6070.6060.608
4 (default)0.6100.6080.611
80.6020.6010.604
(b) modal-prompting rank40.5960.5930.599
8 (default)0.6100.6080.611
160.6060.6090.604
(c) LoRA rank r20.6010.5990.602
4 (default)0.6100.6080.611
80.6000.5980.602
(d) positive/negative share1/80.6040.6060.602
1/4 (default)0.6100.6080.611
1/30.5950.5930.596

Shared embedding tab. 9:

VariantF-scoreRePr
without explicit edge guidance0.6000.6020.598
without implicit learning0.6040.6090.599
without in-domain approximation0.5810.5830.579

Headline claims. On DepthTrack the depth-specific model is 2.1 points above ViPT in precision §4.2, tab. 1. The uni-model is 3.8 points above ViPT on LasHeR precision §4.2, tab. 2, 1.1 points above on VisEvent precision as stated in the text (see Pitfalls) §4.2, tab. 3, and 0.5 points above ViPT on VOT-RGBD2022 accuracy §4.3, tab. 4. On RGBT234 it is 0.7 points above the thermal ViPT in precision §4.3, tab. 5. The "+8.1 absolute F-score" in the abstract is stated against the RGB baseline §1, abstract and matches tab. 1 uni-model (0.610) minus tab. 6 RGB baseline (0.529).

Paper vs code

Code: repository Zongwei97/UnTrack at commit 8eec76ec912c19e326e2b0020444f8f20c7d4355, linked from the abstract. The owner name matches the first author, Zongwei Wu, and the README names the paper as the source [abstract] [code README.md]. The repository has no LICENSE file, so code_license is recorded as not stated.

Differences found:

  • RGB gradient maps are not computed. The paper says RGB gradient maps are generated alongside the auxiliary ones §3.2. The code computes both gradient terms from the auxiliary image only; the variables named for RGB are fed gradient(z_dte) and gradient(x_dte) vit_ce_prompt.py.
  • Gradient definition. The paper describes differences between neighbouring pixels along x and y §3.2. The code takes a 7×7 max filter and subtracts copies shifted in four directions (up, down, left, right), setting zero where the values are equal vit_ce_prompt.py.
  • MLP versus linear. The paper calls σ_x and the fusion and reconstruction maps MLPs §3.2, §3.3. The code uses single linear layers for them vit_ce_prompt.py.
  • Token selection is not Gumbel-based. The paper describes a learnable score function that sorts tokens into three groups §3.3. The code uses a linear score head and a top-k selection. Its function is called gumbel_softmax, but the noise term is commented out, so the selection is deterministic with a straight-through estimator vit_ce_prompt.py.
  • Fixed quarter. The paper tests 1/8, 1/4 and 1/3 §5. The code always sets the number of positive and negative tokens to N//4 vit_ce_prompt.py.
  • Extra edge term. The code adds a token built from the maximum edge magnitude to the prompt path. The paper does not describe this vit_ce_prompt.py.
  • Prompt injection points. The paper says prompting happens "at each scale" fig. 2 caption. The code injects prompts before the first block and, with the deep prompt type, before every later block, using one prompt block per depth level vit_ce_prompt.py.
  • LoRA form. The paper writes h = W0 x + BA x on the attention module §3.4. The code adds the rank-4 path to the output of the whole attention block, with no scaling factor, B initialised to zero and A with a Kaiming-uniform start vit_ce_prompt.py.
  • Candidate elimination. The paper does not mention it. The experiment config enables OSTrack-style candidate elimination with keep ratio 0.7 at blocks 3, 6 and 9 deep_rgbx.yaml.
  • Loss. The paper says the loss matches the baseline §3.4. The code's weights are 2.0, 5.0 and 1.0 train_script.py. The GIoU term falls back silently to zero if the call raises an exception untrack.py.
  • Unused file. lib/models/layers/MoE_prompt.py is not imported by the UnTrack model. The README calls the mixture-of-experts variant a separate preprint README.md MoE_prompt.py.

Pitfalls

  • Trainable set. With the deep prompt type, only parameters whose names contain "prompt" receive gradients. The backbone, the box head and every OSTrack weight are frozen. Adapter names also count as prompt names, because the LoRA matrices are called prompt_w_a and prompt_w_b base_functions.py vit_ce_prompt.py. Implementers should copy the head from OSTrack rather than train it.
  • Gradient shift size. The shift in the gradient code is 7/H in normalised coordinates. With the default align_corners=False, this works out to about 3.5 pixels. The paper does not state the shift vit_ce_prompt.py. Verify this against a run before relying on it.
  • Zero-masked modalities. A sample whose modality code is not 1, 2 or 3 gets zero features from all three low-rank branches. This is the code's implicit RGB-only path. The paper describes dummy values §4.3 vit_ce_prompt.py. The RGB-only path was not run.
  • Token counts for the quarter rule. With the experiment sizes, the search branch has 16×16 = 256 tokens and the template branch 8×8 = 64 tokens (stride 16). The quarter rule then selects 64 and 16 tokens deep_rgbx.yaml. Check the grid size if the config changes.
  • Hard-coded GPU calls. The actor and the scatter helper call .cuda() directly, so CPU-only runs fail untrack.py vit_ce_prompt.py.
  • Silent loss fallback. A failing GIoU call gives a zero loss without an error. This can hide a broken training run untrack.py.
  • Parameter counts disagree. The abstract says "over 93M"; the introduction says 92M; the cost table gives 92.08M for the baseline and 98.73M for Un-Track; the abstract gives +6.6M, the text +6.65M [abstract] §1 tab. 6.
  • The +8.1 claim. The baseline in tab. 6 is 0.529. Un-Track uni-model at 0.610 gives +8.1; the depth-specific model at 0.612 would give +8.3. The paper does not say which one it means §1 tab. 1 tab. 6.
  • VisEvent precision claim. The text says the uni-model is 1.1 points above the current SOTA in precision §4.2. Tab. 3 gives 0.755 against ViPT's 0.740, a 1.5-point gap. The stated figure matches neither column tab. 3.
  • Default row differs. The default setting appears as 0.610 / 0.608 / 0.611 in tab. 8 and as 0.610 / 0.610 / 0.610 in tab. 1, both for the uni-model. Keep the two in mind when comparing ablations with the main table tab. 1 tab. 8.
  • Table numbers. Table numbers were counted in LaTeX source order, not the compiled PDF. Verify them before citing.
  • Licence of the code. The repository has no LICENSE file. Its code is not licensed for reuse by default. We quote no code and record code_license as not stated.
  • Checkpoint. The pretrained model is on Google Drive, linked from the README. It was not fetched, and its licence is not stated.

Lineage

  • Builds on: OSTrack, which supplies the baseline, the initial weights and the loss family §3.1, §3.4 (no card yet).
  • Builds on: LoRA (Hu et al., 2021) for the inner finetuning §3.4 (no card yet).
  • Contrasted with: MaPLe and visual prompt tuning, from which the paper says its prompting differs §1 (no card yet).
  • Compared against: ProTrack and ViPT, the earlier unified or multimodal trackers §1, §2; Stark, AiATrack, UniNext and SeqTrack in the uni-model tables §4.2 (no cards yet).
  • Superseded by: not known from content on this wiki.

Source and license

Adapted from “Single-Model and Any-Modality for Video Object Tracking” by Zongwei Wu, Jilai Zheng, Xiangxuan Ren, Florin-Alexandru Vasluianu, Chao Ma, Danda Pani Paudel, Luc Van Gool, Radu Timofte (arXiv:2311.15851), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Zongwei97/UnTrack at commit 8eec76ec912c (not stated (no LICENSE file in repository)). Any quoted code is shown under that license.